2025-08-26T20:07:13.8602418Z Current runner version: '2.328.0' 2025-08-26T20:07:13.8607414Z Runner name: 'i-04c468ba96b53884f' 2025-08-26T20:07:13.8608181Z Runner group name: 'default' 2025-08-26T20:07:13.8608986Z Machine name: 'ip-10-0-58-230' 2025-08-26T20:07:13.8611227Z ##[group]GITHUB_TOKEN Permissions 2025-08-26T20:07:13.8613339Z Contents: read 2025-08-26T20:07:13.8613752Z Metadata: read 2025-08-26T20:07:13.8614149Z ##[endgroup] 2025-08-26T20:07:13.8615877Z Secret source: Actions 2025-08-26T20:07:13.8616489Z Prepare workflow directory 2025-08-26T20:07:13.9024418Z Prepare all required actions 2025-08-26T20:07:13.9057029Z Getting action download info 2025-08-26T20:07:14.2186648Z Download action repository 'pytorch/test-infra@main' (SHA:0192d56cb596bb73b125bd368553908cc5c513f0) 2025-08-26T20:07:16.1346537Z Download action repository 'pytorch/pytorch@main' (SHA:9f6e1b8730d6a7a7d012be90ae08674294aa4933) 2025-08-26T20:07:31.4688250Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-08-26T20:07:31.8767877Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-08-26T20:07:32.0800757Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-08-26T20:07:32.2641595Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-08-26T20:07:32.5543028Z Getting action download info 2025-08-26T20:07:32.6615003Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-08-26T20:07:32.9114324Z Getting action download info 2025-08-26T20:07:33.0287754Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-08-26T20:07:33.2155862Z Getting action download info 2025-08-26T20:07:33.3112735Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-08-26T20:07:33.4876021Z Getting action download info 2025-08-26T20:07:33.6232336Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (262640fd220236042fbf4443cc163c8838c84c3d) 2025-08-26T20:07:33.6235822Z ##[group] Inputs 2025-08-26T20:07:33.6236197Z build-environment: linux-jammy-py3.9-gcc11-build 2025-08-26T20:07:33.6238117Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-08-26T20:07:33.6240434Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:07:33.6241029Z sync-tag: 2025-08-26T20:07:33.6241753Z timeout-minutes: 240 2025-08-26T20:07:33.6241936Z use-gha: 2025-08-26T20:07:33.6242108Z dashboard-tag: 2025-08-26T20:07:33.6242301Z s3-bucket: gha-artifacts 2025-08-26T20:07:33.6242503Z aws-role-to-assume: 2025-08-26T20:07:33.6242888Z disable-monitor: false 2025-08-26T20:07:33.6243099Z monitor-log-interval: 5 2025-08-26T20:07:33.6243615Z monitor-data-collect-interval: 1 2025-08-26T20:07:33.6243944Z ##[endgroup] 2025-08-26T20:07:33.6244322Z Complete job name: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:07:33.6731186Z A job started hook has been configured by the self-hosted runner administrator 2025-08-26T20:07:33.6819094Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-08-26T20:07:33.6826723Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:07:33.6827197Z ##[endgroup] 2025-08-26T20:07:34.7081817Z Runner Type: linux.8xlarge.amx 2025-08-26T20:07:34.7082351Z Instance Type: m7i-flex.8xlarge 2025-08-26T20:07:34.7082743Z AMI Name: unknown 2025-08-26T20:07:34.7125510Z AMI ID: ami-05ffe3c48a9991133 2025-08-26T20:07:39.2336281Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-08-26T20:07:39.2336697Z with: 2025-08-26T20:07:39.2337445Z github-secret: *** 2025-08-26T20:07:39.2338020Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-08-26T20:07:39.2338620Z activate-with-label: false 2025-08-26T20:07:39.2338920Z label: with-ssh 2025-08-26T20:07:39.2339169Z remove-existing-keys: true 2025-08-26T20:07:39.2339434Z fail-silently: true 2025-08-26T20:07:39.2339725Z env: 2025-08-26T20:07:39.2339954Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:07:39.2340226Z ##[endgroup] 2025-08-26T20:07:39.3577972Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-08-26T20:07:39.3579900Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-08-26T20:07:39.3731713Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-08-26T20:07:39.3732138Z with: 2025-08-26T20:07:39.3732358Z no-sudo: true 2025-08-26T20:07:39.3732657Z submodules: recursive 2025-08-26T20:07:39.3732985Z fetch-depth: 0 2025-08-26T20:07:39.3733217Z env: 2025-08-26T20:07:39.3733467Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:07:39.3733753Z ##[endgroup] 2025-08-26T20:07:39.3900017Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-26T20:07:39.3900677Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-26T20:07:39.3909171Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:07:39.3909448Z env: 2025-08-26T20:07:39.3909654Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:07:39.3909848Z ##[endgroup] 2025-08-26T20:07:39.3996876Z ##[group]Run # Use all available CPUs for fetching 2025-08-26T20:07:39.3997261Z # Use all available CPUs for fetching 2025-08-26T20:07:39.3997551Z cd "${GITHUB_WORKSPACE}" 2025-08-26T20:07:39.3997834Z git config --global fetch.parallel 0 2025-08-26T20:07:39.3998117Z git config --global submodule.fetchJobs 0 2025-08-26T20:07:39.3998386Z  2025-08-26T20:07:39.3998754Z # Clean workspace. The default checkout action should also do this, but 2025-08-26T20:07:39.3999343Z # do it here as well just in case 2025-08-26T20:07:39.3999583Z if [[ -d .git ]]; then 2025-08-26T20:07:39.3999812Z  if [ -z "${NO_SUDO}" ]; then 2025-08-26T20:07:39.4000049Z  sudo git clean -ffdx 2025-08-26T20:07:39.4000286Z  else 2025-08-26T20:07:39.4000469Z  git clean -ffdx 2025-08-26T20:07:39.4000669Z  fi 2025-08-26T20:07:39.4000840Z fi 2025-08-26T20:07:39.4006079Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:07:39.4006346Z env: 2025-08-26T20:07:39.4006528Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:07:39.4006727Z NO_SUDO: true 2025-08-26T20:07:39.4006906Z ##[endgroup] 2025-08-26T20:07:39.4131455Z ##[group]Run actions/checkout@v4 2025-08-26T20:07:39.4131736Z with: 2025-08-26T20:07:39.4131957Z ref: 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:07:39.4132406Z fetch-depth: 0 2025-08-26T20:07:39.4132615Z submodules: recursive 2025-08-26T20:07:39.4132835Z show-progress: false 2025-08-26T20:07:39.4133057Z repository: pytorch/pytorch 2025-08-26T20:07:39.4133387Z token: *** 2025-08-26T20:07:39.4133583Z ssh-strict: true 2025-08-26T20:07:39.4133778Z ssh-user: git 2025-08-26T20:07:39.4133985Z persist-credentials: true 2025-08-26T20:07:39.4134202Z clean: true 2025-08-26T20:07:39.4134411Z sparse-checkout-cone-mode: true 2025-08-26T20:07:39.4134643Z fetch-tags: false 2025-08-26T20:07:39.4134814Z lfs: false 2025-08-26T20:07:39.4134983Z set-safe-directory: true 2025-08-26T20:07:39.4135193Z env: 2025-08-26T20:07:39.4135358Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:07:39.4135545Z ##[endgroup] 2025-08-26T20:07:39.5074354Z Syncing repository: pytorch/pytorch 2025-08-26T20:07:39.5075711Z ##[group]Getting Git version info 2025-08-26T20:07:39.5076079Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-08-26T20:07:39.5076621Z [command]/usr/bin/git version 2025-08-26T20:07:39.5265146Z git version 2.47.1 2025-08-26T20:07:39.5286615Z ##[endgroup] 2025-08-26T20:07:39.5294947Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/d115a785-d260-40fb-ae8e-7874c0696889/.gitconfig' 2025-08-26T20:07:39.5319273Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d115a785-d260-40fb-ae8e-7874c0696889' before making global git config changes 2025-08-26T20:07:39.5320060Z Adding repository directory to the temporary git global config as a safe directory 2025-08-26T20:07:39.5326718Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-26T20:07:39.5369930Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-08-26T20:07:39.5373113Z ##[group]Initializing the repository 2025-08-26T20:07:39.5376833Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-26T20:07:39.5431123Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-08-26T20:07:39.5433604Z hint: is subject to change. To configure the initial branch name to use in all 2025-08-26T20:07:39.5434073Z hint: of your new repositories, which will suppress this warning, call: 2025-08-26T20:07:39.5434374Z hint: 2025-08-26T20:07:39.5434623Z hint: git config --global init.defaultBranch 2025-08-26T20:07:39.5434880Z hint: 2025-08-26T20:07:39.5435136Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-08-26T20:07:39.5435617Z hint: 'development'. The just-created branch can be renamed via this command: 2025-08-26T20:07:39.5435936Z hint: 2025-08-26T20:07:39.5436116Z hint: git branch -m 2025-08-26T20:07:39.5448213Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-08-26T20:07:39.5463043Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-08-26T20:07:39.5497010Z ##[endgroup] 2025-08-26T20:07:39.5497408Z ##[group]Disabling automatic garbage collection 2025-08-26T20:07:39.5501425Z [command]/usr/bin/git config --local gc.auto 0 2025-08-26T20:07:39.5536776Z ##[endgroup] 2025-08-26T20:07:39.5541207Z ##[group]Setting up auth 2025-08-26T20:07:39.5545903Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-26T20:07:39.5557106Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-26T20:07:39.5916445Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-26T20:07:39.5953573Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-26T20:07:39.6277419Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-26T20:07:39.6346868Z ##[endgroup] 2025-08-26T20:07:39.6347448Z ##[group]Fetching the repository 2025-08-26T20:07:39.6353333Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-08-26T20:08:25.8152620Z From https://github.com/pytorch/pytorch 2025-08-26T20:08:25.8154987Z * [new branch] 160583 -> origin/160583 2025-08-26T20:08:25.8166295Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-08-26T20:08:25.8167638Z * [new branch] 5addvllmbuild -> origin/5addvllmbuild 2025-08-26T20:08:25.8168161Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-08-26T20:08:25.8172387Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-08-26T20:08:25.8172906Z * [new branch] ISSUE-154849 -> origin/ISSUE-154849 2025-08-26T20:08:25.8173339Z * [new branch] JackCaoG/dynamo_make_fx_non_core_aten_ops -> origin/JackCaoG/dynamo_make_fx_non_core_aten_ops 2025-08-26T20:08:25.8173789Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-08-26T20:08:25.8174171Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-08-26T20:08:25.8174539Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-08-26T20:08:25.8175193Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-08-26T20:08:25.8178386Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-08-26T20:08:25.8178855Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-08-26T20:08:25.8179238Z * [new branch] VLA_exp -> origin/VLA_exp 2025-08-26T20:08:25.8179666Z * [new branch] add-missing-args-normalization -> origin/add-missing-args-normalization 2025-08-26T20:08:25.8180089Z * [new branch] add-user-guide-structure -> origin/add-user-guide-structure 2025-08-26T20:08:25.8180439Z * [new branch] addVllmPin -> origin/addVllmPin 2025-08-26T20:08:25.8180818Z * [new branch] add_compile_benchmarking -> origin/add_compile_benchmarking 2025-08-26T20:08:25.8181172Z * [new branch] add_windows_testing_back -> origin/add_windows_testing_back 2025-08-26T20:08:25.8181522Z * [new branch] addbuildvllm -> origin/addbuildvllm 2025-08-26T20:08:25.8181834Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-08-26T20:08:25.8182148Z * [new branch] addsimde -> origin/addsimde 2025-08-26T20:08:25.8182474Z * [new branch] adi/acl_upgrade -> origin/adi/acl_upgrade 2025-08-26T20:08:25.8182819Z * [new branch] adi/test -> origin/adi/test 2025-08-26T20:08:25.8183150Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-08-26T20:08:25.8183502Z * [new branch] adi/test_fusions -> origin/adi/test_fusions 2025-08-26T20:08:25.8183890Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-08-26T20:08:25.8184248Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-08-26T20:08:25.8184653Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-08-26T20:08:25.8185045Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-08-26T20:08:25.8185441Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-08-26T20:08:25.8185842Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-08-26T20:08:25.8186485Z * [new branch] alt-disable -> origin/alt-disable 2025-08-26T20:08:25.8186873Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-08-26T20:08:25.8187278Z * [new branch] angelayi/aoti_inductor_fx -> origin/angelayi/aoti_inductor_fx 2025-08-26T20:08:25.8187738Z * [new branch] angelayi/assert_tensor_metadata_device -> origin/angelayi/assert_tensor_metadata_device 2025-08-26T20:08:25.8188176Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-08-26T20:08:25.8188539Z * [new branch] angelayi/benchmark2 -> origin/angelayi/benchmark2 2025-08-26T20:08:25.8188962Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-08-26T20:08:25.8189379Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-08-26T20:08:25.8189752Z * [new branch] angelayi/custom_op_subgraph -> origin/angelayi/custom_op_subgraph 2025-08-26T20:08:25.8190137Z * [new branch] angelayi/customop -> origin/angelayi/customop 2025-08-26T20:08:25.8190513Z * [new branch] angelayi/is_symbolic_tracing -> origin/angelayi/is_symbolic_tracing 2025-08-26T20:08:25.8190915Z * [new branch] angelayi/logging.bak -> origin/angelayi/logging.bak 2025-08-26T20:08:25.8191265Z * [new branch] angelayi/logging2 -> origin/angelayi/logging2 2025-08-26T20:08:25.8191706Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-08-26T20:08:25.8192074Z * [new branch] angelayi/opoverload -> origin/angelayi/opoverload 2025-08-26T20:08:25.8192420Z * [new branch] angelayi/pytree -> origin/angelayi/pytree 2025-08-26T20:08:25.8192760Z * [new branch] angelayi/save_error -> origin/angelayi/save_error 2025-08-26T20:08:25.8193101Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-08-26T20:08:25.8193454Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-08-26T20:08:25.8193843Z * [new branch] angelayi/tensor_nn_module_meta -> origin/angelayi/tensor_nn_module_meta 2025-08-26T20:08:25.8194242Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-08-26T20:08:25.8194590Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-08-26T20:08:25.8194932Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-08-26T20:08:25.8195272Z * [new branch] aoti_weight_sharing -> origin/aoti_weight_sharing 2025-08-26T20:08:25.8195632Z * [new branch] arsh/symint_mm_ind_decomp -> origin/arsh/symint_mm_ind_decomp 2025-08-26T20:08:25.8196039Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-08-26T20:08:25.8196689Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-08-26T20:08:25.8197092Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2025-08-26T20:08:25.8197433Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-08-26T20:08:25.8197781Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-08-26T20:08:25.8198130Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-08-26T20:08:25.8198485Z * [new branch] atalman_inductor_2.3.0 -> origin/atalman_inductor_2.3.0 2025-08-26T20:08:25.8198894Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-08-26T20:08:25.8199539Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-08-26T20:08:25.8200041Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-08-26T20:08:25.8200491Z * [new branch] autoupdate-transformers-pin-via-pr -> origin/autoupdate-transformers-pin-via-pr 2025-08-26T20:08:25.8200914Z * [new branch] backupvllm -> origin/backupvllm 2025-08-26T20:08:25.8201233Z * [new branch] bahuang/test -> origin/bahuang/test 2025-08-26T20:08:25.8201537Z * [new branch] base/1.5 -> origin/base/1.5 2025-08-26T20:08:25.8201917Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-08-26T20:08:25.8207435Z * [new branch] bc-lint-config -> origin/bc-lint-config 2025-08-26T20:08:25.8212511Z * [new branch] bc-lint-test-new-config -> origin/bc-lint-test-new-config 2025-08-26T20:08:25.8217598Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-08-26T20:08:25.8222156Z * [new branch] benchmarker_compat_with_do_bench -> origin/benchmarker_compat_with_do_bench 2025-08-26T20:08:25.8226610Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-08-26T20:08:25.8231126Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-08-26T20:08:25.8233001Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-08-26T20:08:25.8233379Z * [new branch] bf/cg-log -> origin/bf/cg-log 2025-08-26T20:08:25.8234023Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-08-26T20:08:25.8234409Z * [new branch] bf/cg-skip-1-kernel -> origin/bf/cg-skip-1-kernel 2025-08-26T20:08:25.8234761Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-08-26T20:08:25.8235203Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-08-26T20:08:25.8235861Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-08-26T20:08:25.8236425Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-08-26T20:08:25.8236827Z * [new branch] bf/default-recompile-reason -> origin/bf/default-recompile-reason 2025-08-26T20:08:25.8237234Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-08-26T20:08:25.8237614Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-08-26T20:08:25.8237964Z * [new branch] bf/partition-doc -> origin/bf/partition-doc 2025-08-26T20:08:25.8238325Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-08-26T20:08:25.8238696Z * [new branch] bf/partition-turn-on -> origin/bf/partition-turn-on 2025-08-26T20:08:25.8239081Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-08-26T20:08:25.8239748Z * [new branch] bf/rope -> origin/bf/rope 2025-08-26T20:08:25.8240081Z * [new branch] bf/skip-asserts -> origin/bf/skip-asserts 2025-08-26T20:08:25.8240424Z * [new branch] bf16adamw -> origin/bf16adamw 2025-08-26T20:08:25.8240776Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-08-26T20:08:25.8241176Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-08-26T20:08:25.8241557Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-08-26T20:08:25.8241933Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-08-26T20:08:25.8242374Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-08-26T20:08:25.8242745Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-08-26T20:08:25.8243129Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-08-26T20:08:25.8243506Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-08-26T20:08:25.8243885Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-08-26T20:08:25.8244268Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-08-26T20:08:25.8244634Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-08-26T20:08:25.8245048Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-08-26T20:08:25.8245448Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-08-26T20:08:25.8245839Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-08-26T20:08:25.8246221Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-08-26T20:08:25.8246589Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-08-26T20:08:25.8246974Z * [new branch] bowbao/bench_updates_stage -> origin/bowbao/bench_updates_stage 2025-08-26T20:08:25.8247389Z * [new branch] bowbao/dort_rewriter -> origin/bowbao/dort_rewriter 2025-08-26T20:08:25.8247741Z * [new branch] bowbao/wip_prs -> origin/bowbao/wip_prs 2025-08-26T20:08:25.8248126Z * [new branch] bowenbao/partial_min_max_reduce -> origin/bowenbao/partial_min_max_reduce 2025-08-26T20:08:25.8248535Z * [new branch] brister/always_wrapper_ir -> origin/brister/always_wrapper_ir 2025-08-26T20:08:25.8248930Z * [new branch] brister/break_tensorbox -> origin/brister/break_tensorbox 2025-08-26T20:08:25.8249331Z * [new branch] brister/flatten_contig -> origin/brister/flatten_contig 2025-08-26T20:08:25.8249698Z * [new branch] brister/fx_custom_triton -> origin/brister/fx_custom_triton 2025-08-26T20:08:25.8250068Z * [new branch] brister/tensor_box_output -> origin/brister/tensor_box_output 2025-08-26T20:08:25.8250448Z * [new branch] brister/test_block_ptr_same -> origin/brister/test_block_ptr_same 2025-08-26T20:08:25.8250883Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-08-26T20:08:25.8251276Z * [new branch] c57382a49 -> origin/c57382a49 2025-08-26T20:08:25.8251584Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-08-26T20:08:25.8251910Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-08-26T20:08:25.8252532Z * [new branch] camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 -> origin/camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 2025-08-26T20:08:25.8253184Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-08-26T20:08:25.8253656Z * [new branch] cherry-pick-149654-by-pytorch_bot_bot_ -> origin/cherry-pick-149654-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8254163Z * [new branch] cherry-pick-151939-by-pytorch_bot_bot_ -> origin/cherry-pick-151939-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8254655Z * [new branch] cherry-pick-154174-by-pytorch_bot_bot_ -> origin/cherry-pick-154174-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8255131Z * [new branch] cherry-pick-155896-by-pytorch_bot_bot_ -> origin/cherry-pick-155896-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8255663Z * [new branch] cherry-pick-156260-by-pytorch_bot_bot_ -> origin/cherry-pick-156260-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8256147Z * [new branch] cherry-pick-156719-by-pytorch_bot_bot_ -> origin/cherry-pick-156719-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8256637Z * [new branch] cherry-pick-156888-by-pytorch_bot_bot_ -> origin/cherry-pick-156888-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8257118Z * [new branch] cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8257606Z * [new branch] cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8258079Z * [new branch] cherry-pick-157558-by-pytorch_bot_bot_ -> origin/cherry-pick-157558-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8258561Z * [new branch] cherry-pick-157598-by-pytorch_bot_bot_ -> origin/cherry-pick-157598-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8259045Z * [new branch] cherry-pick-157630-by-pytorch_bot_bot_ -> origin/cherry-pick-157630-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8259526Z * [new branch] cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8260006Z * [new branch] cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8260496Z * [new branch] cherry-pick-157733-by-pytorch_bot_bot_ -> origin/cherry-pick-157733-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8261011Z * [new branch] cherry-pick-157985-by-pytorch_bot_bot_ -> origin/cherry-pick-157985-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8261499Z * [new branch] cherry-pick-157993-by-pytorch_bot_bot_ -> origin/cherry-pick-157993-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8261984Z * [new branch] cherry-pick-158064-by-pytorch_bot_bot_ -> origin/cherry-pick-158064-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8262471Z * [new branch] cherry-pick-158152-by-pytorch_bot_bot_ -> origin/cherry-pick-158152-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8262957Z * [new branch] cherry-pick-158301-by-pytorch_bot_bot_ -> origin/cherry-pick-158301-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8263963Z * [new branch] cherry-pick-158537-by-pytorch_bot_bot_ -> origin/cherry-pick-158537-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8264533Z * [new branch] cherry-pick-159181-by-pytorch_bot_bot_ -> origin/cherry-pick-159181-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8265096Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8265655Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-08-26T20:08:25.8266284Z * [new branch] cherrypick-e4e2701429c17078c3c475382a8b1fa4c8a8cefc -> origin/cherrypick-e4e2701429c17078c3c475382a8b1fa4c8a8cefc 2025-08-26T20:08:25.8266861Z * [new branch] chilli/flex_vllm -> origin/chilli/flex_vllm 2025-08-26T20:08:25.8268371Z * [new branch] cleantest1 -> origin/cleantest1 2025-08-26T20:08:25.8269326Z * [new branch] cleanup-inductor-benchmark-images -> origin/cleanup-inductor-benchmark-images 2025-08-26T20:08:25.8272453Z * [new branch] codex-testing -> origin/codex-testing 2025-08-26T20:08:25.8272978Z * [new branch] codex/add-metadata-field-for-file-path -> origin/codex/add-metadata-field-for-file-path 2025-08-26T20:08:25.8273621Z * [new branch] codex/add-test-for-inductor-local-cache-behavior -> origin/codex/add-test-for-inductor-local-cache-behavior 2025-08-26T20:08:25.8274342Z * [new branch] codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph 2025-08-26T20:08:25.8275140Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-08-26T20:08:25.8275631Z * [new branch] codex/fix-issue-160415-in-pytorch -> origin/codex/fix-issue-160415-in-pytorch 2025-08-26T20:08:25.8276246Z * [new branch] codex/fix-noqengine-quantized-engine-support -> origin/codex/fix-noqengine-quantized-engine-support 2025-08-26T20:08:25.8276943Z * [new branch] codex/fix-pin_memory-error-handling -> origin/codex/fix-pin_memory-error-handling 2025-08-26T20:08:25.8277439Z * [new branch] codex/propose-fix-for-issue-160332 -> origin/codex/propose-fix-for-issue-160332 2025-08-26T20:08:25.8278123Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-08-26T20:08:25.8278843Z * [new branch] codex/remove-allow-untyped-defs-and-fix-type-errors -> origin/codex/remove-allow-untyped-defs-and-fix-type-errors 2025-08-26T20:08:25.8279909Z * [new branch] codex/verify-torch-output-and-log-results -> origin/codex/verify-torch-output-and-log-results 2025-08-26T20:08:25.8280467Z * [new branch] compile_fsdp2_disable_stream_and_event -> origin/compile_fsdp2_disable_stream_and_event 2025-08-26T20:08:25.8280896Z * [new branch] context_test -> origin/context_test 2025-08-26T20:08:25.8281243Z * [new branch] copilot/fix-157446 -> origin/copilot/fix-157446 2025-08-26T20:08:25.8281606Z * [new branch] copilot/fix-159257 -> origin/copilot/fix-159257 2025-08-26T20:08:25.8282067Z * [new branch] copy_graph -> origin/copy_graph 2025-08-26T20:08:25.8282422Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-08-26T20:08:25.8282798Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-08-26T20:08:25.8283175Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-08-26T20:08:25.8284412Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-08-26T20:08:25.8284832Z * [new branch] csl/disable_flaky_cpp_test -> origin/csl/disable_flaky_cpp_test 2025-08-26T20:08:25.8285247Z * [new branch] csl/disable_periodic_test -> origin/csl/disable_periodic_test 2025-08-26T20:08:25.8285674Z * [new branch] csl/executorch_docker_fail -> origin/csl/executorch_docker_fail 2025-08-26T20:08:25.8286070Z * [new branch] csl/fix_check_alerts -> origin/csl/fix_check_alerts 2025-08-26T20:08:25.8286422Z * [new branch] csl/katex -> origin/csl/katex 2025-08-26T20:08:25.8286763Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-08-26T20:08:25.8287123Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-08-26T20:08:25.8289102Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-08-26T20:08:25.8289983Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-08-26T20:08:25.8293884Z * [new branch] csl/name_link_check_job -> origin/csl/name_link_check_job 2025-08-26T20:08:25.8300087Z * [new branch] csl/no_keep_goin_rocm -> origin/csl/no_keep_goin_rocm 2025-08-26T20:08:25.8304554Z * [new branch] csl/not_600_timeout -> origin/csl/not_600_timeout 2025-08-26T20:08:25.8309676Z * [new branch] csl/remove_unused_docker_images -> origin/csl/remove_unused_docker_images 2025-08-26T20:08:25.8314915Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-08-26T20:08:25.8315363Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-08-26T20:08:25.8315888Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-08-26T20:08:25.8316964Z * [new branch] csl/unused_docker -> origin/csl/unused_docker 2025-08-26T20:08:25.8317346Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-08-26T20:08:25.8317695Z * [new branch] cublasltrelax2 -> origin/cublasltrelax2 2025-08-26T20:08:25.8318038Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-08-26T20:08:25.8318384Z * [new branch] cudnnsdparefactor -> origin/cudnnsdparefactor 2025-08-26T20:08:25.8318766Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-08-26T20:08:25.8319389Z * [new branch] czhuge_muon_dev -> origin/czhuge_muon_dev 2025-08-26T20:08:25.8319765Z * [new branch] d4l3k/delete_hook -> origin/d4l3k/delete_hook 2025-08-26T20:08:25.8320108Z * [new branch] dcp_zoc -> origin/dcp_zoc 2025-08-26T20:08:25.8320458Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-08-26T20:08:25.8321083Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.2 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.2 2025-08-26T20:08:25.8321942Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.3 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.3 2025-08-26T20:08:25.8322861Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.4 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.4 2025-08-26T20:08:25.8323577Z * [new branch] dependabot/pip/dot-ci/docker/protobuf-5.29.5 -> origin/dependabot/pip/dot-ci/docker/protobuf-5.29.5 2025-08-26T20:08:25.8324217Z * [new branch] dependabot/pip/dot-github/requirements/protobuf-5.29.5 -> origin/dependabot/pip/dot-github/requirements/protobuf-5.29.5 2025-08-26T20:08:25.8324777Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-08-26T20:08:25.8325240Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-08-26T20:08:25.8325705Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-08-26T20:08:25.8326096Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-08-26T20:08:25.8326448Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-08-26T20:08:25.8326807Z * [new branch] dev/joona/cat_remove_graph -> origin/dev/joona/cat_remove_graph 2025-08-26T20:08:25.8327202Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-08-26T20:08:25.8327604Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-08-26T20:08:25.8328087Z * [new branch] dev/joona/maxpool2dwithindices_errmsg -> origin/dev/joona/maxpool2dwithindices_errmsg 2025-08-26T20:08:25.8328559Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-08-26T20:08:25.8328926Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-08-26T20:08:25.8329313Z * [new branch] dev/joona/synchronize_benchmark -> origin/dev/joona/synchronize_benchmark 2025-08-26T20:08:25.8329695Z * [new branch] dev/joona/topk_newapi -> origin/dev/joona/topk_newapi 2025-08-26T20:08:25.8330047Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-08-26T20:08:25.8330395Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-08-26T20:08:25.8330739Z * [new branch] disable -> origin/disable 2025-08-26T20:08:25.8331109Z * [new branch] e2e-baseline -> origin/e2e-baseline 2025-08-26T20:08:25.8331521Z * [new branch] eigen_for_sparse_addmm_v2 -> origin/eigen_for_sparse_addmm_v2 2025-08-26T20:08:25.8331899Z * [new branch] embg/test_inductor_ci_128B -> origin/embg/test_inductor_ci_128B 2025-08-26T20:08:25.8332253Z * [new branch] embg/test_inductor_ci_base -> origin/embg/test_inductor_ci_base 2025-08-26T20:08:25.8332615Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-08-26T20:08:25.8333002Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-08-26T20:08:25.8333371Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-08-26T20:08:25.8333743Z * [new branch] enable-b200-benchmark -> origin/enable-b200-benchmark 2025-08-26T20:08:25.8334098Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-08-26T20:08:25.8334412Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-08-26T20:08:25.8334727Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-08-26T20:08:25.8335035Z * [new branch] eqy-patch-4 -> origin/eqy-patch-4 2025-08-26T20:08:25.8335396Z * [new branch] example-convert-torch.nn -> origin/example-convert-torch.nn 2025-08-26T20:08:25.8335788Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-08-26T20:08:25.8336300Z * [new branch] exclamaforte/bump-transformer-version -> origin/exclamaforte/bump-transformer-version 2025-08-26T20:08:25.8336810Z * [new branch] exclamaforte/clear-feedback-savers -> origin/exclamaforte/clear-feedback-savers 2025-08-26T20:08:25.8337305Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-08-26T20:08:25.8337792Z * [new branch] exclamaforte/debug-autotuner-profile -> origin/exclamaforte/debug-autotuner-profile 2025-08-26T20:08:25.8338233Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-08-26T20:08:25.8338651Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-08-26T20:08:25.8339124Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-08-26T20:08:25.8339619Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-08-26T20:08:25.8340130Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-08-26T20:08:25.8340585Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-08-26T20:08:25.8340984Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-08-26T20:08:25.8341426Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-08-26T20:08:25.8341839Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-08-26T20:08:25.8342333Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-08-26T20:08:25.8342829Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-08-26T20:08:25.8343228Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-08-26T20:08:25.8343696Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-08-26T20:08:25.8344148Z * [new branch] exclamaforte/memory-counter -> origin/exclamaforte/memory-counter 2025-08-26T20:08:25.8344683Z * [new branch] exclamaforte/profiler-combo -> origin/exclamaforte/profiler-combo 2025-08-26T20:08:25.8345119Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-08-26T20:08:25.8345580Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-08-26T20:08:25.8346223Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-08-26T20:08:25.8349339Z * [new branch] exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final 2025-08-26T20:08:25.8349823Z * [new branch] exec -> origin/exec 2025-08-26T20:08:25.8354932Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-08-26T20:08:25.8355412Z * [new branch] export-D58091437 -> origin/export-D58091437 2025-08-26T20:08:25.8355765Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-08-26T20:08:25.8356126Z * [new branch] export-D70112642 -> origin/export-D70112642 2025-08-26T20:08:25.8356465Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-08-26T20:08:25.8356799Z * [new branch] export-D72483950 -> origin/export-D72483950 2025-08-26T20:08:25.8357140Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-08-26T20:08:25.8357474Z * [new branch] export-D75183591 -> origin/export-D75183591 2025-08-26T20:08:25.8358007Z * [new branch] export-D75605373 -> origin/export-D75605373 2025-08-26T20:08:25.8358363Z * [new branch] export-D75617432 -> origin/export-D75617432 2025-08-26T20:08:25.8358697Z * [new branch] export-D75659965 -> origin/export-D75659965 2025-08-26T20:08:25.8359038Z * [new branch] export-D76080931 -> origin/export-D76080931 2025-08-26T20:08:25.8359751Z * [new branch] export-D76463347 -> origin/export-D76463347 2025-08-26T20:08:25.8360102Z * [new branch] export-D76797250 -> origin/export-D76797250 2025-08-26T20:08:25.8360443Z * [new branch] export-D76885271 -> origin/export-D76885271 2025-08-26T20:08:25.8360805Z * [new branch] export-D76885620 -> origin/export-D76885620 2025-08-26T20:08:25.8361139Z * [new branch] export-D76936623 -> origin/export-D76936623 2025-08-26T20:08:25.8361483Z * [new branch] export-D76958268 -> origin/export-D76958268 2025-08-26T20:08:25.8361842Z * [new branch] export-D78308105 -> origin/export-D78308105 2025-08-26T20:08:25.8362185Z * [new branch] export-D78375400 -> origin/export-D78375400 2025-08-26T20:08:25.8362523Z * [new branch] export-D78431305 -> origin/export-D78431305 2025-08-26T20:08:25.8362866Z * [new branch] export-D78580107 -> origin/export-D78580107 2025-08-26T20:08:25.8363215Z * [new branch] export-D78822171 -> origin/export-D78822171 2025-08-26T20:08:25.8363556Z * [new branch] export-D78822351 -> origin/export-D78822351 2025-08-26T20:08:25.8363889Z * [new branch] export-D78822507 -> origin/export-D78822507 2025-08-26T20:08:25.8364218Z * [new branch] export-D78826994 -> origin/export-D78826994 2025-08-26T20:08:25.8364568Z * [new branch] export-D78894142 -> origin/export-D78894142 2025-08-26T20:08:25.8364901Z * [new branch] export-D78894324 -> origin/export-D78894324 2025-08-26T20:08:25.8365366Z * [new branch] export-D78929245 -> origin/export-D78929245 2025-08-26T20:08:25.8366151Z * [new branch] export-D78934925 -> origin/export-D78934925 2025-08-26T20:08:25.8366766Z * [new branch] export-D78953203 -> origin/export-D78953203 2025-08-26T20:08:25.8367457Z * [new branch] export-D78953229 -> origin/export-D78953229 2025-08-26T20:08:25.8368071Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-08-26T20:08:25.8368745Z * [new branch] export-D78957389 -> origin/export-D78957389 2025-08-26T20:08:25.8369514Z * [new branch] export-D78979812 -> origin/export-D78979812 2025-08-26T20:08:25.8370194Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-08-26T20:08:25.8371518Z * [new branch] export-D79026433 -> origin/export-D79026433 2025-08-26T20:08:25.8372791Z * [new branch] export-D79230339 -> origin/export-D79230339 2025-08-26T20:08:25.8373157Z * [new branch] export-D79319835 -> origin/export-D79319835 2025-08-26T20:08:25.8373522Z * [new branch] export-D79328456 -> origin/export-D79328456 2025-08-26T20:08:25.8374384Z * [new branch] export-D79534608 -> origin/export-D79534608 2025-08-26T20:08:25.8374859Z * [new branch] export-D79647167 -> origin/export-D79647167 2025-08-26T20:08:25.8375216Z * [new branch] export-D79751098 -> origin/export-D79751098 2025-08-26T20:08:25.8376166Z * [new branch] export-D79785974 -> origin/export-D79785974 2025-08-26T20:08:25.8377040Z * [new branch] export-D80025417 -> origin/export-D80025417 2025-08-26T20:08:25.8377428Z * [new branch] export-D80120333 -> origin/export-D80120333 2025-08-26T20:08:25.8378517Z * [new branch] export-D80214882 -> origin/export-D80214882 2025-08-26T20:08:25.8379566Z * [new branch] export-D80319069 -> origin/export-D80319069 2025-08-26T20:08:25.8380015Z * [new branch] export-D80321215 -> origin/export-D80321215 2025-08-26T20:08:25.8381051Z * [new branch] export-D80503451 -> origin/export-D80503451 2025-08-26T20:08:25.8381420Z * [new branch] export-D80771648 -> origin/export-D80771648 2025-08-26T20:08:25.8383111Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-08-26T20:08:25.8383762Z * [new branch] export-D80948073 -> origin/export-D80948073 2025-08-26T20:08:25.8384178Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-08-26T20:08:25.8384547Z * [new branch] export-D80970483 -> origin/export-D80970483 2025-08-26T20:08:25.8384892Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-08-26T20:08:25.8385991Z * [new branch] export-D81060182 -> origin/export-D81060182 2025-08-26T20:08:25.8387563Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-08-26T20:08:25.8388026Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-08-26T20:08:25.8388411Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-08-26T20:08:25.8389122Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-08-26T20:08:25.8390418Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-08-26T20:08:25.8391118Z * [new branch] fca -> origin/fca 2025-08-26T20:08:25.8399109Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-08-26T20:08:25.8399932Z * [new branch] fca5 -> origin/fca5 2025-08-26T20:08:25.8400399Z * [new branch] feature/function-numa-binding -> origin/feature/function-numa-binding 2025-08-26T20:08:25.8401122Z * [new branch] feature/function-numa-binding-take2 -> origin/feature/function-numa-binding-take2 2025-08-26T20:08:25.8401571Z * [new branch] feature/numa-nproc-fix -> origin/feature/numa-nproc-fix 2025-08-26T20:08:25.8401993Z * [new branch] feature/numa-signpost-serialize -> origin/feature/numa-signpost-serialize 2025-08-26T20:08:25.8402625Z * [new branch] fengyuan/external-proj -> origin/fengyuan/external-proj 2025-08-26T20:08:25.8403609Z * [new branch] fengyuan/out-of-tree-xpu-ops-improve-test -> origin/fengyuan/out-of-tree-xpu-ops-improve-test 2025-08-26T20:08:25.8404242Z * [new branch] fengyuan/out-of-tree-xpu-ops-remove-dtype -> origin/fengyuan/out-of-tree-xpu-ops-remove-dtype 2025-08-26T20:08:25.8404719Z * [new branch] fengyuan/test-xpu -> origin/fengyuan/test-xpu 2025-08-26T20:08:25.8405093Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-08-26T20:08:25.8405441Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-08-26T20:08:25.8405789Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-08-26T20:08:25.8406130Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-08-26T20:08:25.8406490Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-08-26T20:08:25.8406837Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-08-26T20:08:25.8407377Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-08-26T20:08:25.8408498Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-08-26T20:08:25.8409011Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-08-26T20:08:25.8409505Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-08-26T20:08:25.8409957Z * [new branch] fix -> origin/fix 2025-08-26T20:08:25.8410430Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-08-26T20:08:25.8410802Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-08-26T20:08:25.8411408Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-08-26T20:08:25.8411840Z * [new branch] fix-distributed-warning -> origin/fix-distributed-warning 2025-08-26T20:08:25.8413743Z * [new branch] fix-inductor-periodic-0528 -> origin/fix-inductor-periodic-0528 2025-08-26T20:08:25.8414269Z * [new branch] fix-mps-benchmark -> origin/fix-mps-benchmark 2025-08-26T20:08:25.8414807Z * [new branch] fix-rlease-feature-template -> origin/fix-rlease-feature-template 2025-08-26T20:08:25.8415707Z * [new branch] fix-run-condition-upload-results -> origin/fix-run-condition-upload-results 2025-08-26T20:08:25.8416173Z * [new branch] fix-torchbench -> origin/fix-torchbench 2025-08-26T20:08:25.8416495Z * [new branch] fix_153389 -> origin/fix_153389 2025-08-26T20:08:25.8416807Z * [new branch] fix_fsdp_rs_bucket2 -> origin/fix_fsdp_rs_bucket2 2025-08-26T20:08:25.8417167Z * [new branch] fix_inductor_peridic_tests -> origin/fix_inductor_peridic_tests 2025-08-26T20:08:25.8417537Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-08-26T20:08:25.8422413Z * [new branch] fixes-triage -> origin/fixes-triage 2025-08-26T20:08:25.8422963Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-08-26T20:08:25.8423418Z * [new branch] flex-flash -> origin/flex-flash 2025-08-26T20:08:25.8424046Z * [new branch] flex-lowering -> origin/flex-lowering 2025-08-26T20:08:25.8424499Z * [new branch] flex-warning -> origin/flex-warning 2025-08-26T20:08:25.8424933Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-08-26T20:08:25.8425407Z * [new branch] flex_flash -> origin/flex_flash 2025-08-26T20:08:25.8426201Z * [new branch] flexdecode-gqa-groups -> origin/flexdecode-gqa-groups 2025-08-26T20:08:25.8426684Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-08-26T20:08:25.8427104Z * [new branch] fmassa/try_fix_ac_tag_propagation -> origin/fmassa/try_fix_ac_tag_propagation 2025-08-26T20:08:25.8427486Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-08-26T20:08:25.8427801Z * [new branch] fsdpv2_3d -> origin/fsdpv2_3d 2025-08-26T20:08:25.8428136Z * [new branch] fsdpv2_3d_m1 -> origin/fsdpv2_3d_m1 2025-08-26T20:08:25.8428455Z * [new branch] fx_cpp -> origin/fx_cpp 2025-08-26T20:08:25.8428771Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-08-26T20:08:25.8429487Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-08-26T20:08:25.8429995Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-08-26T20:08:25.8430627Z * [new branch] gh/CaoE/2/base -> origin/gh/CaoE/2/base 2025-08-26T20:08:25.8431074Z * [new branch] gh/CaoE/2/head -> origin/gh/CaoE/2/head 2025-08-26T20:08:25.8431411Z * [new branch] gh/CaoE/2/orig -> origin/gh/CaoE/2/orig 2025-08-26T20:08:25.8435032Z * [new branch] gh/ColinPeppler/78/base -> origin/gh/ColinPeppler/78/base 2025-08-26T20:08:25.8435446Z * [new branch] gh/ColinPeppler/78/head -> origin/gh/ColinPeppler/78/head 2025-08-26T20:08:25.8435841Z * [new branch] gh/ColinPeppler/78/orig -> origin/gh/ColinPeppler/78/orig 2025-08-26T20:08:25.8436232Z * [new branch] gh/ColinPeppler/79/base -> origin/gh/ColinPeppler/79/base 2025-08-26T20:08:25.8436634Z * [new branch] gh/ColinPeppler/79/head -> origin/gh/ColinPeppler/79/head 2025-08-26T20:08:25.8437024Z * [new branch] gh/ColinPeppler/79/orig -> origin/gh/ColinPeppler/79/orig 2025-08-26T20:08:25.8437410Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-08-26T20:08:25.8437768Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-08-26T20:08:25.8438113Z * [new branch] gh/EikanWang/80/base -> origin/gh/EikanWang/80/base 2025-08-26T20:08:25.8438462Z * [new branch] gh/EikanWang/80/head -> origin/gh/EikanWang/80/head 2025-08-26T20:08:25.8438804Z * [new branch] gh/EikanWang/80/orig -> origin/gh/EikanWang/80/orig 2025-08-26T20:08:25.8440382Z * [new branch] gh/EikanWang/81/base -> origin/gh/EikanWang/81/base 2025-08-26T20:08:25.8450584Z * [new branch] gh/EikanWang/81/head -> origin/gh/EikanWang/81/head 2025-08-26T20:08:25.8456086Z * [new branch] gh/EikanWang/81/orig -> origin/gh/EikanWang/81/orig 2025-08-26T20:08:25.8458096Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-08-26T20:08:25.8458586Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-08-26T20:08:25.8464452Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-08-26T20:08:25.8466477Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-08-26T20:08:25.8467072Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-08-26T20:08:25.8467760Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-08-26T20:08:25.8468258Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-08-26T20:08:25.8468739Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-08-26T20:08:25.8469227Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-08-26T20:08:25.8470102Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-08-26T20:08:25.8470572Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-08-26T20:08:25.8470924Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-08-26T20:08:25.8471275Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-08-26T20:08:25.8471628Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-08-26T20:08:25.8472009Z * [new branch] gh/H-Huang/187/base -> origin/gh/H-Huang/187/base 2025-08-26T20:08:25.8472365Z * [new branch] gh/H-Huang/187/head -> origin/gh/H-Huang/187/head 2025-08-26T20:08:25.8472768Z * [new branch] gh/H-Huang/187/orig -> origin/gh/H-Huang/187/orig 2025-08-26T20:08:25.8473116Z * [new branch] gh/H-Huang/196/base -> origin/gh/H-Huang/196/base 2025-08-26T20:08:25.8473464Z * [new branch] gh/H-Huang/196/head -> origin/gh/H-Huang/196/head 2025-08-26T20:08:25.8473974Z * [new branch] gh/H-Huang/196/orig -> origin/gh/H-Huang/196/orig 2025-08-26T20:08:25.8474345Z * [new branch] gh/H-Huang/197/base -> origin/gh/H-Huang/197/base 2025-08-26T20:08:25.8474694Z * [new branch] gh/H-Huang/197/head -> origin/gh/H-Huang/197/head 2025-08-26T20:08:25.8475056Z * [new branch] gh/H-Huang/197/orig -> origin/gh/H-Huang/197/orig 2025-08-26T20:08:25.8475411Z * [new branch] gh/H-Huang/198/base -> origin/gh/H-Huang/198/base 2025-08-26T20:08:25.8475758Z * [new branch] gh/H-Huang/198/head -> origin/gh/H-Huang/198/head 2025-08-26T20:08:25.8476116Z * [new branch] gh/H-Huang/198/orig -> origin/gh/H-Huang/198/orig 2025-08-26T20:08:25.8476470Z * [new branch] gh/H-Huang/200/base -> origin/gh/H-Huang/200/base 2025-08-26T20:08:25.8476831Z * [new branch] gh/H-Huang/200/head -> origin/gh/H-Huang/200/head 2025-08-26T20:08:25.8477186Z * [new branch] gh/H-Huang/200/orig -> origin/gh/H-Huang/200/orig 2025-08-26T20:08:25.8477541Z * [new branch] gh/H-Huang/201/base -> origin/gh/H-Huang/201/base 2025-08-26T20:08:25.8477910Z * [new branch] gh/H-Huang/201/head -> origin/gh/H-Huang/201/head 2025-08-26T20:08:25.8478271Z * [new branch] gh/H-Huang/201/orig -> origin/gh/H-Huang/201/orig 2025-08-26T20:08:25.8478630Z * [new branch] gh/H-Huang/202/base -> origin/gh/H-Huang/202/base 2025-08-26T20:08:25.8478984Z * [new branch] gh/H-Huang/202/head -> origin/gh/H-Huang/202/head 2025-08-26T20:08:25.8479618Z * [new branch] gh/H-Huang/202/orig -> origin/gh/H-Huang/202/orig 2025-08-26T20:08:25.8479981Z * [new branch] gh/H-Huang/203/base -> origin/gh/H-Huang/203/base 2025-08-26T20:08:25.8480344Z * [new branch] gh/H-Huang/203/head -> origin/gh/H-Huang/203/head 2025-08-26T20:08:25.8480700Z * [new branch] gh/H-Huang/203/orig -> origin/gh/H-Huang/203/orig 2025-08-26T20:08:25.8481059Z * [new branch] gh/H-Huang/204/base -> origin/gh/H-Huang/204/base 2025-08-26T20:08:25.8481403Z * [new branch] gh/H-Huang/204/head -> origin/gh/H-Huang/204/head 2025-08-26T20:08:25.8481818Z * [new branch] gh/H-Huang/204/orig -> origin/gh/H-Huang/204/orig 2025-08-26T20:08:25.8482172Z * [new branch] gh/H-Huang/205/base -> origin/gh/H-Huang/205/base 2025-08-26T20:08:25.8482533Z * [new branch] gh/H-Huang/205/head -> origin/gh/H-Huang/205/head 2025-08-26T20:08:25.8482884Z * [new branch] gh/H-Huang/205/orig -> origin/gh/H-Huang/205/orig 2025-08-26T20:08:25.8483236Z * [new branch] gh/H-Huang/206/base -> origin/gh/H-Huang/206/base 2025-08-26T20:08:25.8483587Z * [new branch] gh/H-Huang/206/head -> origin/gh/H-Huang/206/head 2025-08-26T20:08:25.8483939Z * [new branch] gh/H-Huang/206/orig -> origin/gh/H-Huang/206/orig 2025-08-26T20:08:25.8484290Z * [new branch] gh/H-Huang/207/base -> origin/gh/H-Huang/207/base 2025-08-26T20:08:25.8484648Z * [new branch] gh/H-Huang/207/head -> origin/gh/H-Huang/207/head 2025-08-26T20:08:25.8485008Z * [new branch] gh/H-Huang/207/orig -> origin/gh/H-Huang/207/orig 2025-08-26T20:08:25.8485378Z * [new branch] gh/H-Huang/208/base -> origin/gh/H-Huang/208/base 2025-08-26T20:08:25.8485713Z * [new branch] gh/H-Huang/208/head -> origin/gh/H-Huang/208/head 2025-08-26T20:08:25.8486058Z * [new branch] gh/H-Huang/208/orig -> origin/gh/H-Huang/208/orig 2025-08-26T20:08:25.8486399Z * [new branch] gh/H-Huang/209/base -> origin/gh/H-Huang/209/base 2025-08-26T20:08:25.8486781Z * [new branch] gh/H-Huang/209/head -> origin/gh/H-Huang/209/head 2025-08-26T20:08:25.8487215Z * [new branch] gh/H-Huang/209/orig -> origin/gh/H-Huang/209/orig 2025-08-26T20:08:25.8487550Z * [new branch] gh/H-Huang/210/base -> origin/gh/H-Huang/210/base 2025-08-26T20:08:25.8487881Z * [new branch] gh/H-Huang/210/head -> origin/gh/H-Huang/210/head 2025-08-26T20:08:25.8488214Z * [new branch] gh/H-Huang/210/orig -> origin/gh/H-Huang/210/orig 2025-08-26T20:08:25.8488561Z * [new branch] gh/H-Huang/211/base -> origin/gh/H-Huang/211/base 2025-08-26T20:08:25.8488897Z * [new branch] gh/H-Huang/211/head -> origin/gh/H-Huang/211/head 2025-08-26T20:08:25.8489231Z * [new branch] gh/H-Huang/211/orig -> origin/gh/H-Huang/211/orig 2025-08-26T20:08:25.8489568Z * [new branch] gh/H-Huang/212/base -> origin/gh/H-Huang/212/base 2025-08-26T20:08:25.8489924Z * [new branch] gh/H-Huang/212/head -> origin/gh/H-Huang/212/head 2025-08-26T20:08:25.8490254Z * [new branch] gh/H-Huang/212/orig -> origin/gh/H-Huang/212/orig 2025-08-26T20:08:25.8495465Z * [new branch] gh/IvanKobzarev/111/base -> origin/gh/IvanKobzarev/111/base 2025-08-26T20:08:25.8495975Z * [new branch] gh/IvanKobzarev/111/head -> origin/gh/IvanKobzarev/111/head 2025-08-26T20:08:25.8496574Z * [new branch] gh/IvanKobzarev/111/orig -> origin/gh/IvanKobzarev/111/orig 2025-08-26T20:08:25.8496982Z * [new branch] gh/IvanKobzarev/112/base -> origin/gh/IvanKobzarev/112/base 2025-08-26T20:08:25.8497403Z * [new branch] gh/IvanKobzarev/112/head -> origin/gh/IvanKobzarev/112/head 2025-08-26T20:08:25.8497784Z * [new branch] gh/IvanKobzarev/112/orig -> origin/gh/IvanKobzarev/112/orig 2025-08-26T20:08:25.8498198Z * [new branch] gh/IvanKobzarev/115/base -> origin/gh/IvanKobzarev/115/base 2025-08-26T20:08:25.8498604Z * [new branch] gh/IvanKobzarev/115/head -> origin/gh/IvanKobzarev/115/head 2025-08-26T20:08:25.8499002Z * [new branch] gh/IvanKobzarev/115/orig -> origin/gh/IvanKobzarev/115/orig 2025-08-26T20:08:25.8499406Z * [new branch] gh/IvanKobzarev/116/base -> origin/gh/IvanKobzarev/116/base 2025-08-26T20:08:25.8500058Z * [new branch] gh/IvanKobzarev/116/head -> origin/gh/IvanKobzarev/116/head 2025-08-26T20:08:25.8500449Z * [new branch] gh/IvanKobzarev/116/orig -> origin/gh/IvanKobzarev/116/orig 2025-08-26T20:08:25.8500919Z * [new branch] gh/IvanKobzarev/118/base -> origin/gh/IvanKobzarev/118/base 2025-08-26T20:08:25.8501536Z * [new branch] gh/IvanKobzarev/118/head -> origin/gh/IvanKobzarev/118/head 2025-08-26T20:08:25.8503361Z * [new branch] gh/IvanKobzarev/118/orig -> origin/gh/IvanKobzarev/118/orig 2025-08-26T20:08:25.8503790Z * [new branch] gh/IvanKobzarev/124/base -> origin/gh/IvanKobzarev/124/base 2025-08-26T20:08:25.8504229Z * [new branch] gh/IvanKobzarev/124/head -> origin/gh/IvanKobzarev/124/head 2025-08-26T20:08:25.8504652Z * [new branch] gh/IvanKobzarev/124/orig -> origin/gh/IvanKobzarev/124/orig 2025-08-26T20:08:25.8507073Z * [new branch] gh/IvanKobzarev/126/base -> origin/gh/IvanKobzarev/126/base 2025-08-26T20:08:25.8507545Z * [new branch] gh/IvanKobzarev/126/head -> origin/gh/IvanKobzarev/126/head 2025-08-26T20:08:25.8507954Z * [new branch] gh/IvanKobzarev/126/orig -> origin/gh/IvanKobzarev/126/orig 2025-08-26T20:08:25.8508724Z * [new branch] gh/IvanKobzarev/127/base -> origin/gh/IvanKobzarev/127/base 2025-08-26T20:08:25.8514403Z * [new branch] gh/IvanKobzarev/127/head -> origin/gh/IvanKobzarev/127/head 2025-08-26T20:08:25.8515118Z * [new branch] gh/IvanKobzarev/127/orig -> origin/gh/IvanKobzarev/127/orig 2025-08-26T20:08:25.8515521Z * [new branch] gh/IvanKobzarev/128/base -> origin/gh/IvanKobzarev/128/base 2025-08-26T20:08:25.8515912Z * [new branch] gh/IvanKobzarev/128/head -> origin/gh/IvanKobzarev/128/head 2025-08-26T20:08:25.8516292Z * [new branch] gh/IvanKobzarev/128/orig -> origin/gh/IvanKobzarev/128/orig 2025-08-26T20:08:25.8516694Z * [new branch] gh/IvanKobzarev/132/base -> origin/gh/IvanKobzarev/132/base 2025-08-26T20:08:25.8517099Z * [new branch] gh/IvanKobzarev/132/head -> origin/gh/IvanKobzarev/132/head 2025-08-26T20:08:25.8517493Z * [new branch] gh/IvanKobzarev/132/orig -> origin/gh/IvanKobzarev/132/orig 2025-08-26T20:08:25.8517890Z * [new branch] gh/IvanKobzarev/133/base -> origin/gh/IvanKobzarev/133/base 2025-08-26T20:08:25.8518290Z * [new branch] gh/IvanKobzarev/133/head -> origin/gh/IvanKobzarev/133/head 2025-08-26T20:08:25.8518697Z * [new branch] gh/IvanKobzarev/133/orig -> origin/gh/IvanKobzarev/133/orig 2025-08-26T20:08:25.8519098Z * [new branch] gh/IvanKobzarev/134/base -> origin/gh/IvanKobzarev/134/base 2025-08-26T20:08:25.8519731Z * [new branch] gh/IvanKobzarev/134/head -> origin/gh/IvanKobzarev/134/head 2025-08-26T20:08:25.8520156Z * [new branch] gh/IvanKobzarev/134/orig -> origin/gh/IvanKobzarev/134/orig 2025-08-26T20:08:25.8520854Z * [new branch] gh/IvanKobzarev/135/base -> origin/gh/IvanKobzarev/135/base 2025-08-26T20:08:25.8521584Z * [new branch] gh/IvanKobzarev/135/head -> origin/gh/IvanKobzarev/135/head 2025-08-26T20:08:25.8522225Z * [new branch] gh/IvanKobzarev/135/orig -> origin/gh/IvanKobzarev/135/orig 2025-08-26T20:08:25.8527030Z * [new branch] gh/IvanKobzarev/136/base -> origin/gh/IvanKobzarev/136/base 2025-08-26T20:08:25.8527513Z * [new branch] gh/IvanKobzarev/136/head -> origin/gh/IvanKobzarev/136/head 2025-08-26T20:08:25.8527938Z * [new branch] gh/IvanKobzarev/136/orig -> origin/gh/IvanKobzarev/136/orig 2025-08-26T20:08:25.8528349Z * [new branch] gh/IvanKobzarev/137/base -> origin/gh/IvanKobzarev/137/base 2025-08-26T20:08:25.8528731Z * [new branch] gh/IvanKobzarev/137/head -> origin/gh/IvanKobzarev/137/head 2025-08-26T20:08:25.8529275Z * [new branch] gh/IvanKobzarev/137/orig -> origin/gh/IvanKobzarev/137/orig 2025-08-26T20:08:25.8534283Z * [new branch] gh/IvanKobzarev/138/base -> origin/gh/IvanKobzarev/138/base 2025-08-26T20:08:25.8538714Z * [new branch] gh/IvanKobzarev/138/head -> origin/gh/IvanKobzarev/138/head 2025-08-26T20:08:25.8539826Z * [new branch] gh/IvanKobzarev/138/orig -> origin/gh/IvanKobzarev/138/orig 2025-08-26T20:08:25.8540232Z * [new branch] gh/IvanKobzarev/139/base -> origin/gh/IvanKobzarev/139/base 2025-08-26T20:08:25.8540627Z * [new branch] gh/IvanKobzarev/139/head -> origin/gh/IvanKobzarev/139/head 2025-08-26T20:08:25.8541009Z * [new branch] gh/IvanKobzarev/139/orig -> origin/gh/IvanKobzarev/139/orig 2025-08-26T20:08:25.8541372Z * [new branch] gh/IvanKobzarev/140/base -> origin/gh/IvanKobzarev/140/base 2025-08-26T20:08:25.8541747Z * [new branch] gh/IvanKobzarev/140/head -> origin/gh/IvanKobzarev/140/head 2025-08-26T20:08:25.8542113Z * [new branch] gh/IvanKobzarev/140/orig -> origin/gh/IvanKobzarev/140/orig 2025-08-26T20:08:25.8542480Z * [new branch] gh/IvanKobzarev/141/base -> origin/gh/IvanKobzarev/141/base 2025-08-26T20:08:25.8542847Z * [new branch] gh/IvanKobzarev/141/head -> origin/gh/IvanKobzarev/141/head 2025-08-26T20:08:25.8543205Z * [new branch] gh/IvanKobzarev/141/orig -> origin/gh/IvanKobzarev/141/orig 2025-08-26T20:08:25.8543740Z * [new branch] gh/IvanKobzarev/142/base -> origin/gh/IvanKobzarev/142/base 2025-08-26T20:08:25.8544118Z * [new branch] gh/IvanKobzarev/142/head -> origin/gh/IvanKobzarev/142/head 2025-08-26T20:08:25.8544496Z * [new branch] gh/IvanKobzarev/142/orig -> origin/gh/IvanKobzarev/142/orig 2025-08-26T20:08:25.8544916Z * [new branch] gh/IvanKobzarev/143/base -> origin/gh/IvanKobzarev/143/base 2025-08-26T20:08:25.8545293Z * [new branch] gh/IvanKobzarev/143/head -> origin/gh/IvanKobzarev/143/head 2025-08-26T20:08:25.8545686Z * [new branch] gh/IvanKobzarev/143/orig -> origin/gh/IvanKobzarev/143/orig 2025-08-26T20:08:25.8546080Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-08-26T20:08:25.8546464Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-08-26T20:08:25.8546834Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-08-26T20:08:25.8547202Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-08-26T20:08:25.8547545Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-08-26T20:08:25.8552202Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-08-26T20:08:25.8552824Z * [new branch] gh/NikhilAPatel/8/base -> origin/gh/NikhilAPatel/8/base 2025-08-26T20:08:25.8553364Z * [new branch] gh/NikhilAPatel/8/head -> origin/gh/NikhilAPatel/8/head 2025-08-26T20:08:25.8553758Z * [new branch] gh/NikhilAPatel/8/orig -> origin/gh/NikhilAPatel/8/orig 2025-08-26T20:08:25.8561364Z * [new branch] gh/NikhilAPatel/9/base -> origin/gh/NikhilAPatel/9/base 2025-08-26T20:08:25.8561833Z * [new branch] gh/NikhilAPatel/9/head -> origin/gh/NikhilAPatel/9/head 2025-08-26T20:08:25.8562242Z * [new branch] gh/NikhilAPatel/9/orig -> origin/gh/NikhilAPatel/9/orig 2025-08-26T20:08:25.8562638Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2025-08-26T20:08:25.8562982Z * [new branch] gh/PaliC/1/head -> origin/gh/PaliC/1/head 2025-08-26T20:08:25.8563313Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2025-08-26T20:08:25.8563803Z * [new branch] gh/PaliC/12/base -> origin/gh/PaliC/12/base 2025-08-26T20:08:25.8564146Z * [new branch] gh/PaliC/12/head -> origin/gh/PaliC/12/head 2025-08-26T20:08:25.8564477Z * [new branch] gh/PaliC/12/orig -> origin/gh/PaliC/12/orig 2025-08-26T20:08:25.8564809Z * [new branch] gh/PaliC/13/base -> origin/gh/PaliC/13/base 2025-08-26T20:08:25.8565147Z * [new branch] gh/PaliC/13/head -> origin/gh/PaliC/13/head 2025-08-26T20:08:25.8565536Z * [new branch] gh/PaliC/13/orig -> origin/gh/PaliC/13/orig 2025-08-26T20:08:25.8565859Z * [new branch] gh/PaliC/14/base -> origin/gh/PaliC/14/base 2025-08-26T20:08:25.8566162Z * [new branch] gh/PaliC/14/head -> origin/gh/PaliC/14/head 2025-08-26T20:08:25.8566474Z * [new branch] gh/PaliC/14/orig -> origin/gh/PaliC/14/orig 2025-08-26T20:08:25.8566784Z * [new branch] gh/PaliC/16/base -> origin/gh/PaliC/16/base 2025-08-26T20:08:25.8567089Z * [new branch] gh/PaliC/16/head -> origin/gh/PaliC/16/head 2025-08-26T20:08:25.8567396Z * [new branch] gh/PaliC/16/orig -> origin/gh/PaliC/16/orig 2025-08-26T20:08:25.8567735Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-08-26T20:08:25.8568062Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-08-26T20:08:25.8568462Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-08-26T20:08:25.8573871Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-08-26T20:08:25.8574296Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-08-26T20:08:25.8574650Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-08-26T20:08:25.8575001Z * [new branch] gh/PaliC/19/base -> origin/gh/PaliC/19/base 2025-08-26T20:08:25.8575343Z * [new branch] gh/PaliC/19/head -> origin/gh/PaliC/19/head 2025-08-26T20:08:25.8575672Z * [new branch] gh/PaliC/19/orig -> origin/gh/PaliC/19/orig 2025-08-26T20:08:25.8576011Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2025-08-26T20:08:25.8576323Z * [new branch] gh/PaliC/2/head -> origin/gh/PaliC/2/head 2025-08-26T20:08:25.8578074Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2025-08-26T20:08:25.8578403Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-08-26T20:08:25.8579312Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-08-26T20:08:25.8579735Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-08-26T20:08:25.8580158Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-08-26T20:08:25.8580504Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-08-26T20:08:25.8580843Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-08-26T20:08:25.8585306Z * [new branch] gh/PaliC/22/base -> origin/gh/PaliC/22/base 2025-08-26T20:08:25.8585692Z * [new branch] gh/PaliC/22/head -> origin/gh/PaliC/22/head 2025-08-26T20:08:25.8586069Z * [new branch] gh/PaliC/22/orig -> origin/gh/PaliC/22/orig 2025-08-26T20:08:25.8586414Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-08-26T20:08:25.8586765Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-08-26T20:08:25.8587110Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-08-26T20:08:25.8587590Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-08-26T20:08:25.8589340Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-08-26T20:08:25.8589692Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-08-26T20:08:25.8590083Z * [new branch] gh/PaulZhang12/17/base -> origin/gh/PaulZhang12/17/base 2025-08-26T20:08:25.8590464Z * [new branch] gh/PaulZhang12/17/head -> origin/gh/PaulZhang12/17/head 2025-08-26T20:08:25.8590827Z * [new branch] gh/PaulZhang12/20/base -> origin/gh/PaulZhang12/20/base 2025-08-26T20:08:25.8591218Z * [new branch] gh/PaulZhang12/20/head -> origin/gh/PaulZhang12/20/head 2025-08-26T20:08:25.8591580Z * [new branch] gh/PaulZhang12/20/orig -> origin/gh/PaulZhang12/20/orig 2025-08-26T20:08:25.8591938Z * [new branch] gh/PaulZhang12/21/base -> origin/gh/PaulZhang12/21/base 2025-08-26T20:08:25.8592329Z * [new branch] gh/PaulZhang12/21/head -> origin/gh/PaulZhang12/21/head 2025-08-26T20:08:25.8592684Z * [new branch] gh/PaulZhang12/21/orig -> origin/gh/PaulZhang12/21/orig 2025-08-26T20:08:25.8593042Z * [new branch] gh/PaulZhang12/22/base -> origin/gh/PaulZhang12/22/base 2025-08-26T20:08:25.8593398Z * [new branch] gh/PaulZhang12/22/head -> origin/gh/PaulZhang12/22/head 2025-08-26T20:08:25.8593830Z * [new branch] gh/PaulZhang12/22/orig -> origin/gh/PaulZhang12/22/orig 2025-08-26T20:08:25.8594736Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-08-26T20:08:25.8595195Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-08-26T20:08:25.8596688Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-08-26T20:08:25.8597376Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-08-26T20:08:25.8598547Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-08-26T20:08:25.8599623Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-08-26T20:08:25.8606754Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-08-26T20:08:25.8607159Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-08-26T20:08:25.8607563Z * [new branch] gh/StrongerXi/103/base -> origin/gh/StrongerXi/103/base 2025-08-26T20:08:25.8607943Z * [new branch] gh/StrongerXi/103/head -> origin/gh/StrongerXi/103/head 2025-08-26T20:08:25.8608318Z * [new branch] gh/StrongerXi/103/orig -> origin/gh/StrongerXi/103/orig 2025-08-26T20:08:25.8608708Z * [new branch] gh/StrongerXi/133/base -> origin/gh/StrongerXi/133/base 2025-08-26T20:08:25.8609094Z * [new branch] gh/StrongerXi/133/head -> origin/gh/StrongerXi/133/head 2025-08-26T20:08:25.8609469Z * [new branch] gh/StrongerXi/133/orig -> origin/gh/StrongerXi/133/orig 2025-08-26T20:08:25.8610000Z * [new branch] gh/StrongerXi/134/base -> origin/gh/StrongerXi/134/base 2025-08-26T20:08:25.8610491Z * [new branch] gh/StrongerXi/134/head -> origin/gh/StrongerXi/134/head 2025-08-26T20:08:25.8611014Z * [new branch] gh/StrongerXi/134/orig -> origin/gh/StrongerXi/134/orig 2025-08-26T20:08:25.8611547Z * [new branch] gh/StrongerXi/136/base -> origin/gh/StrongerXi/136/base 2025-08-26T20:08:25.8612547Z * [new branch] gh/StrongerXi/136/head -> origin/gh/StrongerXi/136/head 2025-08-26T20:08:25.8613011Z * [new branch] gh/StrongerXi/136/orig -> origin/gh/StrongerXi/136/orig 2025-08-26T20:08:25.8613415Z * [new branch] gh/StrongerXi/137/base -> origin/gh/StrongerXi/137/base 2025-08-26T20:08:25.8614965Z * [new branch] gh/StrongerXi/137/head -> origin/gh/StrongerXi/137/head 2025-08-26T20:08:25.8615382Z * [new branch] gh/StrongerXi/137/orig -> origin/gh/StrongerXi/137/orig 2025-08-26T20:08:25.8615770Z * [new branch] gh/StrongerXi/138/base -> origin/gh/StrongerXi/138/base 2025-08-26T20:08:25.8616167Z * [new branch] gh/StrongerXi/138/head -> origin/gh/StrongerXi/138/head 2025-08-26T20:08:25.8616520Z * [new branch] gh/StrongerXi/138/orig -> origin/gh/StrongerXi/138/orig 2025-08-26T20:08:25.8616949Z * [new branch] gh/StrongerXi/139/base -> origin/gh/StrongerXi/139/base 2025-08-26T20:08:25.8617327Z * [new branch] gh/StrongerXi/139/head -> origin/gh/StrongerXi/139/head 2025-08-26T20:08:25.8617714Z * [new branch] gh/StrongerXi/139/orig -> origin/gh/StrongerXi/139/orig 2025-08-26T20:08:25.8620722Z * [new branch] gh/StrongerXi/140/base -> origin/gh/StrongerXi/140/base 2025-08-26T20:08:25.8621102Z * [new branch] gh/StrongerXi/140/head -> origin/gh/StrongerXi/140/head 2025-08-26T20:08:25.8621506Z * [new branch] gh/StrongerXi/140/orig -> origin/gh/StrongerXi/140/orig 2025-08-26T20:08:25.8621892Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-08-26T20:08:25.8622273Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-08-26T20:08:25.8622794Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-08-26T20:08:25.8623160Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-08-26T20:08:25.8625173Z * [new branch] gh/XilunWu/133/base -> origin/gh/XilunWu/133/base 2025-08-26T20:08:25.8625662Z * [new branch] gh/XilunWu/133/head -> origin/gh/XilunWu/133/head 2025-08-26T20:08:25.8626161Z * [new branch] gh/XilunWu/133/orig -> origin/gh/XilunWu/133/orig 2025-08-26T20:08:25.8626662Z * [new branch] gh/XilunWu/139/base -> origin/gh/XilunWu/139/base 2025-08-26T20:08:25.8627051Z * [new branch] gh/XilunWu/139/head -> origin/gh/XilunWu/139/head 2025-08-26T20:08:25.8627399Z * [new branch] gh/XilunWu/139/orig -> origin/gh/XilunWu/139/orig 2025-08-26T20:08:25.8630879Z * [new branch] gh/XilunWu/143/base -> origin/gh/XilunWu/143/base 2025-08-26T20:08:25.8631238Z * [new branch] gh/XilunWu/143/head -> origin/gh/XilunWu/143/head 2025-08-26T20:08:25.8631591Z * [new branch] gh/XilunWu/143/orig -> origin/gh/XilunWu/143/orig 2025-08-26T20:08:25.8631937Z * [new branch] gh/XilunWu/144/base -> origin/gh/XilunWu/144/base 2025-08-26T20:08:25.8632309Z * [new branch] gh/XilunWu/144/head -> origin/gh/XilunWu/144/head 2025-08-26T20:08:25.8632628Z * [new branch] gh/XilunWu/144/orig -> origin/gh/XilunWu/144/orig 2025-08-26T20:08:25.8632957Z * [new branch] gh/XilunWu/145/base -> origin/gh/XilunWu/145/base 2025-08-26T20:08:25.8633286Z * [new branch] gh/XilunWu/145/head -> origin/gh/XilunWu/145/head 2025-08-26T20:08:25.8637110Z * [new branch] gh/XilunWu/145/orig -> origin/gh/XilunWu/145/orig 2025-08-26T20:08:25.8637463Z * [new branch] gh/XilunWu/146/base -> origin/gh/XilunWu/146/base 2025-08-26T20:08:25.8637799Z * [new branch] gh/XilunWu/146/head -> origin/gh/XilunWu/146/head 2025-08-26T20:08:25.8638163Z * [new branch] gh/XilunWu/146/orig -> origin/gh/XilunWu/146/orig 2025-08-26T20:08:25.8638504Z * [new branch] gh/XilunWu/147/base -> origin/gh/XilunWu/147/base 2025-08-26T20:08:25.8638852Z * [new branch] gh/XilunWu/147/head -> origin/gh/XilunWu/147/head 2025-08-26T20:08:25.8639365Z * [new branch] gh/XilunWu/147/orig -> origin/gh/XilunWu/147/orig 2025-08-26T20:08:25.8639726Z * [new branch] gh/XilunWu/148/base -> origin/gh/XilunWu/148/base 2025-08-26T20:08:25.8640081Z * [new branch] gh/XilunWu/148/head -> origin/gh/XilunWu/148/head 2025-08-26T20:08:25.8640434Z * [new branch] gh/XilunWu/148/orig -> origin/gh/XilunWu/148/orig 2025-08-26T20:08:25.8640754Z * [new branch] gh/XilunWu/149/base -> origin/gh/XilunWu/149/base 2025-08-26T20:08:25.8641081Z * [new branch] gh/XilunWu/149/head -> origin/gh/XilunWu/149/head 2025-08-26T20:08:25.8641393Z * [new branch] gh/XilunWu/149/orig -> origin/gh/XilunWu/149/orig 2025-08-26T20:08:25.8647482Z * [new branch] gh/XilunWu/150/base -> origin/gh/XilunWu/150/base 2025-08-26T20:08:25.8647851Z * [new branch] gh/XilunWu/150/head -> origin/gh/XilunWu/150/head 2025-08-26T20:08:25.8648207Z * [new branch] gh/XilunWu/150/orig -> origin/gh/XilunWu/150/orig 2025-08-26T20:08:25.8648556Z * [new branch] gh/XilunWu/151/base -> origin/gh/XilunWu/151/base 2025-08-26T20:08:25.8648898Z * [new branch] gh/XilunWu/151/head -> origin/gh/XilunWu/151/head 2025-08-26T20:08:25.8649226Z * [new branch] gh/XilunWu/151/orig -> origin/gh/XilunWu/151/orig 2025-08-26T20:08:25.8649573Z * [new branch] gh/XilunWu/152/base -> origin/gh/XilunWu/152/base 2025-08-26T20:08:25.8649959Z * [new branch] gh/XilunWu/152/head -> origin/gh/XilunWu/152/head 2025-08-26T20:08:25.8653915Z * [new branch] gh/XilunWu/152/orig -> origin/gh/XilunWu/152/orig 2025-08-26T20:08:25.8658817Z * [new branch] gh/XilunWu/153/base -> origin/gh/XilunWu/153/base 2025-08-26T20:08:25.8663304Z * [new branch] gh/XilunWu/153/head -> origin/gh/XilunWu/153/head 2025-08-26T20:08:25.8664477Z * [new branch] gh/XilunWu/153/orig -> origin/gh/XilunWu/153/orig 2025-08-26T20:08:25.8665264Z * [new branch] gh/XilunWu/159/base -> origin/gh/XilunWu/159/base 2025-08-26T20:08:25.8667644Z * [new branch] gh/XilunWu/159/head -> origin/gh/XilunWu/159/head 2025-08-26T20:08:25.8668138Z * [new branch] gh/XilunWu/159/orig -> origin/gh/XilunWu/159/orig 2025-08-26T20:08:25.8671947Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-08-26T20:08:25.8672395Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-08-26T20:08:25.8672756Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-08-26T20:08:25.8673105Z * [new branch] gh/XilunWu/161/base -> origin/gh/XilunWu/161/base 2025-08-26T20:08:25.8673465Z * [new branch] gh/XilunWu/161/head -> origin/gh/XilunWu/161/head 2025-08-26T20:08:25.8673805Z * [new branch] gh/XilunWu/161/orig -> origin/gh/XilunWu/161/orig 2025-08-26T20:08:25.8674162Z * [new branch] gh/XilunWu/162/base -> origin/gh/XilunWu/162/base 2025-08-26T20:08:25.8674512Z * [new branch] gh/XilunWu/162/head -> origin/gh/XilunWu/162/head 2025-08-26T20:08:25.8674844Z * [new branch] gh/XilunWu/162/orig -> origin/gh/XilunWu/162/orig 2025-08-26T20:08:25.8675185Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-08-26T20:08:25.8675518Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-08-26T20:08:25.8675852Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-08-26T20:08:25.8676185Z * [new branch] gh/XilunWu/164/base -> origin/gh/XilunWu/164/base 2025-08-26T20:08:25.8676670Z * [new branch] gh/XilunWu/164/head -> origin/gh/XilunWu/164/head 2025-08-26T20:08:25.8677011Z * [new branch] gh/XilunWu/164/orig -> origin/gh/XilunWu/164/orig 2025-08-26T20:08:25.8677353Z * [new branch] gh/XilunWu/165/base -> origin/gh/XilunWu/165/base 2025-08-26T20:08:25.8677690Z * [new branch] gh/XilunWu/165/head -> origin/gh/XilunWu/165/head 2025-08-26T20:08:25.8678033Z * [new branch] gh/XilunWu/165/orig -> origin/gh/XilunWu/165/orig 2025-08-26T20:08:25.8678379Z * [new branch] gh/XilunWu/166/base -> origin/gh/XilunWu/166/base 2025-08-26T20:08:25.8678723Z * [new branch] gh/XilunWu/166/head -> origin/gh/XilunWu/166/head 2025-08-26T20:08:25.8679068Z * [new branch] gh/XilunWu/166/orig -> origin/gh/XilunWu/166/orig 2025-08-26T20:08:25.8679676Z * [new branch] gh/XilunWu/167/base -> origin/gh/XilunWu/167/base 2025-08-26T20:08:25.8680036Z * [new branch] gh/XilunWu/167/head -> origin/gh/XilunWu/167/head 2025-08-26T20:08:25.8680395Z * [new branch] gh/XilunWu/167/orig -> origin/gh/XilunWu/167/orig 2025-08-26T20:08:25.8680752Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-08-26T20:08:25.8681130Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-08-26T20:08:25.8681491Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-08-26T20:08:25.8681909Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-08-26T20:08:25.8682270Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-08-26T20:08:25.8682628Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-08-26T20:08:25.8682980Z * [new branch] gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-08-26T20:08:25.8683342Z * [new branch] gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-08-26T20:08:25.8683721Z * [new branch] gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig 2025-08-26T20:08:25.8684069Z * [new branch] gh/XuehaiPan/227/base -> origin/gh/XuehaiPan/227/base 2025-08-26T20:08:25.8684427Z * [new branch] gh/XuehaiPan/227/head -> origin/gh/XuehaiPan/227/head 2025-08-26T20:08:25.8684777Z * [new branch] gh/XuehaiPan/227/orig -> origin/gh/XuehaiPan/227/orig 2025-08-26T20:08:25.8685128Z * [new branch] gh/XuehaiPan/231/base -> origin/gh/XuehaiPan/231/base 2025-08-26T20:08:25.8685477Z * [new branch] gh/XuehaiPan/231/head -> origin/gh/XuehaiPan/231/head 2025-08-26T20:08:25.8685823Z * [new branch] gh/XuehaiPan/231/orig -> origin/gh/XuehaiPan/231/orig 2025-08-26T20:08:25.8686186Z * [new branch] gh/XuehaiPan/232/base -> origin/gh/XuehaiPan/232/base 2025-08-26T20:08:25.8686548Z * [new branch] gh/XuehaiPan/232/head -> origin/gh/XuehaiPan/232/head 2025-08-26T20:08:25.8686899Z * [new branch] gh/XuehaiPan/232/orig -> origin/gh/XuehaiPan/232/orig 2025-08-26T20:08:25.8687249Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-08-26T20:08:25.8687589Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-08-26T20:08:25.8687945Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-08-26T20:08:25.8688309Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-08-26T20:08:25.8688673Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-08-26T20:08:25.8689037Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-08-26T20:08:25.8689444Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-08-26T20:08:25.8695309Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-08-26T20:08:25.8695693Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-08-26T20:08:25.8696069Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-08-26T20:08:25.8696685Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-08-26T20:08:25.8697054Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-08-26T20:08:25.8697420Z * [new branch] gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-08-26T20:08:25.8697781Z * [new branch] gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-08-26T20:08:25.8702892Z * [new branch] gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig 2025-08-26T20:08:25.8703522Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-08-26T20:08:25.8704032Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-08-26T20:08:25.8709523Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-08-26T20:08:25.8710132Z * [new branch] gh/XuehaiPan/290/base -> origin/gh/XuehaiPan/290/base 2025-08-26T20:08:25.8710643Z * [new branch] gh/XuehaiPan/290/head -> origin/gh/XuehaiPan/290/head 2025-08-26T20:08:25.8711257Z * [new branch] gh/XuehaiPan/290/orig -> origin/gh/XuehaiPan/290/orig 2025-08-26T20:08:25.8711650Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-08-26T20:08:25.8712014Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-08-26T20:08:25.8712374Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-08-26T20:08:25.8712727Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-08-26T20:08:25.8713081Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-08-26T20:08:25.8713436Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-08-26T20:08:25.8713780Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-08-26T20:08:25.8714134Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-08-26T20:08:25.8714493Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-08-26T20:08:25.8714857Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-08-26T20:08:25.8715229Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-08-26T20:08:25.8715578Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-08-26T20:08:25.8715943Z * [new branch] gh/XuehaiPan/356/base -> origin/gh/XuehaiPan/356/base 2025-08-26T20:08:25.8716306Z * [new branch] gh/XuehaiPan/356/head -> origin/gh/XuehaiPan/356/head 2025-08-26T20:08:25.8716673Z * [new branch] gh/XuehaiPan/356/orig -> origin/gh/XuehaiPan/356/orig 2025-08-26T20:08:25.8717033Z * [new branch] gh/XuehaiPan/357/base -> origin/gh/XuehaiPan/357/base 2025-08-26T20:08:25.8717387Z * [new branch] gh/XuehaiPan/357/head -> origin/gh/XuehaiPan/357/head 2025-08-26T20:08:25.8717750Z * [new branch] gh/XuehaiPan/357/orig -> origin/gh/XuehaiPan/357/orig 2025-08-26T20:08:25.8718112Z * [new branch] gh/XuehaiPan/358/base -> origin/gh/XuehaiPan/358/base 2025-08-26T20:08:25.8718476Z * [new branch] gh/XuehaiPan/358/head -> origin/gh/XuehaiPan/358/head 2025-08-26T20:08:25.8718939Z * [new branch] gh/XuehaiPan/358/orig -> origin/gh/XuehaiPan/358/orig 2025-08-26T20:08:25.8719531Z * [new branch] gh/XuehaiPan/359/base -> origin/gh/XuehaiPan/359/base 2025-08-26T20:08:25.8719898Z * [new branch] gh/XuehaiPan/359/head -> origin/gh/XuehaiPan/359/head 2025-08-26T20:08:25.8720257Z * [new branch] gh/XuehaiPan/359/orig -> origin/gh/XuehaiPan/359/orig 2025-08-26T20:08:25.8720616Z * [new branch] gh/XuehaiPan/360/base -> origin/gh/XuehaiPan/360/base 2025-08-26T20:08:25.8720988Z * [new branch] gh/XuehaiPan/360/head -> origin/gh/XuehaiPan/360/head 2025-08-26T20:08:25.8721382Z * [new branch] gh/XuehaiPan/360/orig -> origin/gh/XuehaiPan/360/orig 2025-08-26T20:08:25.8721749Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-08-26T20:08:25.8722119Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-08-26T20:08:25.8722481Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-08-26T20:08:25.8722845Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-08-26T20:08:25.8723199Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-08-26T20:08:25.8723560Z * [new branch] gh/XuehaiPan/369/base -> origin/gh/XuehaiPan/369/base 2025-08-26T20:08:25.8724461Z * [new branch] gh/XuehaiPan/369/head -> origin/gh/XuehaiPan/369/head 2025-08-26T20:08:25.8724822Z * [new branch] gh/XuehaiPan/369/orig -> origin/gh/XuehaiPan/369/orig 2025-08-26T20:08:25.8725181Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-08-26T20:08:25.8728849Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-08-26T20:08:25.8729227Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-08-26T20:08:25.8729585Z * [new branch] gh/XuehaiPan/371/base -> origin/gh/XuehaiPan/371/base 2025-08-26T20:08:25.8729961Z * [new branch] gh/XuehaiPan/371/head -> origin/gh/XuehaiPan/371/head 2025-08-26T20:08:25.8730333Z * [new branch] gh/XuehaiPan/371/orig -> origin/gh/XuehaiPan/371/orig 2025-08-26T20:08:25.8730698Z * [new branch] gh/XuehaiPan/377/base -> origin/gh/XuehaiPan/377/base 2025-08-26T20:08:25.8732516Z * [new branch] gh/XuehaiPan/377/head -> origin/gh/XuehaiPan/377/head 2025-08-26T20:08:25.8732883Z * [new branch] gh/XuehaiPan/377/orig -> origin/gh/XuehaiPan/377/orig 2025-08-26T20:08:25.8733234Z * [new branch] gh/XuehaiPan/378/base -> origin/gh/XuehaiPan/378/base 2025-08-26T20:08:25.8733585Z * [new branch] gh/XuehaiPan/378/head -> origin/gh/XuehaiPan/378/head 2025-08-26T20:08:25.8733946Z * [new branch] gh/XuehaiPan/378/orig -> origin/gh/XuehaiPan/378/orig 2025-08-26T20:08:25.8738367Z * [new branch] gh/XuehaiPan/379/base -> origin/gh/XuehaiPan/379/base 2025-08-26T20:08:25.8740241Z * [new branch] gh/XuehaiPan/379/head -> origin/gh/XuehaiPan/379/head 2025-08-26T20:08:25.8741052Z * [new branch] gh/XuehaiPan/379/orig -> origin/gh/XuehaiPan/379/orig 2025-08-26T20:08:25.8744973Z * [new branch] gh/XuehaiPan/380/base -> origin/gh/XuehaiPan/380/base 2025-08-26T20:08:25.8745405Z * [new branch] gh/XuehaiPan/380/head -> origin/gh/XuehaiPan/380/head 2025-08-26T20:08:25.8745836Z * [new branch] gh/XuehaiPan/380/orig -> origin/gh/XuehaiPan/380/orig 2025-08-26T20:08:25.8746213Z * [new branch] gh/XuehaiPan/381/base -> origin/gh/XuehaiPan/381/base 2025-08-26T20:08:25.8746752Z * [new branch] gh/XuehaiPan/381/head -> origin/gh/XuehaiPan/381/head 2025-08-26T20:08:25.8747122Z * [new branch] gh/XuehaiPan/382/base -> origin/gh/XuehaiPan/382/base 2025-08-26T20:08:25.8747494Z * [new branch] gh/XuehaiPan/382/head -> origin/gh/XuehaiPan/382/head 2025-08-26T20:08:25.8747879Z * [new branch] gh/XuehaiPan/382/orig -> origin/gh/XuehaiPan/382/orig 2025-08-26T20:08:25.8748233Z * [new branch] gh/XuehaiPan/383/base -> origin/gh/XuehaiPan/383/base 2025-08-26T20:08:25.8748618Z * [new branch] gh/XuehaiPan/383/head -> origin/gh/XuehaiPan/383/head 2025-08-26T20:08:25.8748987Z * [new branch] gh/XuehaiPan/383/orig -> origin/gh/XuehaiPan/383/orig 2025-08-26T20:08:25.8749356Z * [new branch] gh/XuehaiPan/384/base -> origin/gh/XuehaiPan/384/base 2025-08-26T20:08:25.8749729Z * [new branch] gh/XuehaiPan/384/head -> origin/gh/XuehaiPan/384/head 2025-08-26T20:08:25.8752659Z * [new branch] gh/XuehaiPan/384/orig -> origin/gh/XuehaiPan/384/orig 2025-08-26T20:08:25.8753072Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-08-26T20:08:25.8753469Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-08-26T20:08:25.8753849Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-08-26T20:08:25.8754227Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-08-26T20:08:25.8754652Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-08-26T20:08:25.8755030Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-08-26T20:08:25.8755408Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-08-26T20:08:25.8755790Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-08-26T20:08:25.8756174Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-08-26T20:08:25.8756557Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-08-26T20:08:25.8756938Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-08-26T20:08:25.8757312Z * [new branch] gh/ZhiweiYan-96/64/base -> origin/gh/ZhiweiYan-96/64/base 2025-08-26T20:08:25.8757693Z * [new branch] gh/ZhiweiYan-96/64/head -> origin/gh/ZhiweiYan-96/64/head 2025-08-26T20:08:25.8758066Z * [new branch] gh/ZhiweiYan-96/64/orig -> origin/gh/ZhiweiYan-96/64/orig 2025-08-26T20:08:25.8758462Z * [new branch] gh/ZhiweiYan-96/65/base -> origin/gh/ZhiweiYan-96/65/base 2025-08-26T20:08:25.8758849Z * [new branch] gh/ZhiweiYan-96/65/head -> origin/gh/ZhiweiYan-96/65/head 2025-08-26T20:08:25.8759883Z * [new branch] gh/ZhiweiYan-96/65/orig -> origin/gh/ZhiweiYan-96/65/orig 2025-08-26T20:08:25.8760699Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-08-26T20:08:25.8761433Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-08-26T20:08:25.8763743Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-08-26T20:08:25.8764210Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-08-26T20:08:25.8764640Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-08-26T20:08:25.8765352Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-08-26T20:08:25.8765745Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-08-26T20:08:25.8766530Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-08-26T20:08:25.8770758Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-08-26T20:08:25.8771160Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-08-26T20:08:25.8771544Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-08-26T20:08:25.8771928Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-08-26T20:08:25.8774681Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-08-26T20:08:25.8775153Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-08-26T20:08:25.8775723Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-08-26T20:08:25.8776266Z * [new branch] gh/alexsamardzic/8/base -> origin/gh/alexsamardzic/8/base 2025-08-26T20:08:25.8776688Z * [new branch] gh/alexsamardzic/8/head -> origin/gh/alexsamardzic/8/head 2025-08-26T20:08:25.8780645Z * [new branch] gh/alexsamardzic/8/orig -> origin/gh/alexsamardzic/8/orig 2025-08-26T20:08:25.8781108Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-08-26T20:08:25.8781488Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-08-26T20:08:25.8781862Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-08-26T20:08:25.8782399Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-08-26T20:08:25.8783417Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-08-26T20:08:25.8783863Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-08-26T20:08:25.8784254Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-08-26T20:08:25.8784655Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-08-26T20:08:25.8785028Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-08-26T20:08:25.8789100Z * [new branch] gh/andyanwang/1/base -> origin/gh/andyanwang/1/base 2025-08-26T20:08:25.8789581Z * [new branch] gh/andyanwang/1/head -> origin/gh/andyanwang/1/head 2025-08-26T20:08:25.8789964Z * [new branch] gh/andyanwang/1/orig -> origin/gh/andyanwang/1/orig 2025-08-26T20:08:25.8790378Z * [new branch] gh/andyanwang/13/base -> origin/gh/andyanwang/13/base 2025-08-26T20:08:25.8790762Z * [new branch] gh/andyanwang/13/head -> origin/gh/andyanwang/13/head 2025-08-26T20:08:25.8791157Z * [new branch] gh/andyanwang/13/orig -> origin/gh/andyanwang/13/orig 2025-08-26T20:08:25.8791552Z * [new branch] gh/andyanwang/2/base -> origin/gh/andyanwang/2/base 2025-08-26T20:08:25.8791929Z * [new branch] gh/andyanwang/2/head -> origin/gh/andyanwang/2/head 2025-08-26T20:08:25.8792295Z * [new branch] gh/andyanwang/2/orig -> origin/gh/andyanwang/2/orig 2025-08-26T20:08:25.8792667Z * [new branch] gh/andyanwang/28/base -> origin/gh/andyanwang/28/base 2025-08-26T20:08:25.8793045Z * [new branch] gh/andyanwang/28/head -> origin/gh/andyanwang/28/head 2025-08-26T20:08:25.8793429Z * [new branch] gh/andyanwang/28/orig -> origin/gh/andyanwang/28/orig 2025-08-26T20:08:25.8793799Z * [new branch] gh/andyanwang/3/base -> origin/gh/andyanwang/3/base 2025-08-26T20:08:25.8794579Z * [new branch] gh/andyanwang/3/head -> origin/gh/andyanwang/3/head 2025-08-26T20:08:25.8795016Z * [new branch] gh/andyanwang/3/orig -> origin/gh/andyanwang/3/orig 2025-08-26T20:08:25.8795887Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-08-26T20:08:25.8796863Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-08-26T20:08:25.8797616Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-08-26T20:08:25.8798798Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-08-26T20:08:25.8800803Z * [new branch] gh/andyanwang/32/base -> origin/gh/andyanwang/32/base 2025-08-26T20:08:25.8801222Z * [new branch] gh/andyanwang/32/head -> origin/gh/andyanwang/32/head 2025-08-26T20:08:25.8801722Z * [new branch] gh/andyanwang/32/orig -> origin/gh/andyanwang/32/orig 2025-08-26T20:08:25.8803541Z * [new branch] gh/andyanwang/35/base -> origin/gh/andyanwang/35/base 2025-08-26T20:08:25.8804004Z * [new branch] gh/andyanwang/35/head -> origin/gh/andyanwang/35/head 2025-08-26T20:08:25.8804622Z * [new branch] gh/andyanwang/35/orig -> origin/gh/andyanwang/35/orig 2025-08-26T20:08:25.8805858Z * [new branch] gh/andyanwang/36/base -> origin/gh/andyanwang/36/base 2025-08-26T20:08:25.8809338Z * [new branch] gh/andyanwang/36/head -> origin/gh/andyanwang/36/head 2025-08-26T20:08:25.8809789Z * [new branch] gh/andyanwang/36/orig -> origin/gh/andyanwang/36/orig 2025-08-26T20:08:25.8810231Z * [new branch] gh/andyanwang/37/base -> origin/gh/andyanwang/37/base 2025-08-26T20:08:25.8810888Z * [new branch] gh/andyanwang/37/head -> origin/gh/andyanwang/37/head 2025-08-26T20:08:25.8811264Z * [new branch] gh/andyanwang/37/orig -> origin/gh/andyanwang/37/orig 2025-08-26T20:08:25.8811637Z * [new branch] gh/andyanwang/38/base -> origin/gh/andyanwang/38/base 2025-08-26T20:08:25.8812011Z * [new branch] gh/andyanwang/38/head -> origin/gh/andyanwang/38/head 2025-08-26T20:08:25.8812641Z * [new branch] gh/andyanwang/38/orig -> origin/gh/andyanwang/38/orig 2025-08-26T20:08:25.8813342Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-08-26T20:08:25.8813989Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-08-26T20:08:25.8816408Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-08-26T20:08:25.8816890Z * [new branch] gh/andyanwang/4/base -> origin/gh/andyanwang/4/base 2025-08-26T20:08:25.8817279Z * [new branch] gh/andyanwang/4/head -> origin/gh/andyanwang/4/head 2025-08-26T20:08:25.8817655Z * [new branch] gh/andyanwang/4/orig -> origin/gh/andyanwang/4/orig 2025-08-26T20:08:25.8818303Z * [new branch] gh/andyanwang/40/base -> origin/gh/andyanwang/40/base 2025-08-26T20:08:25.8819021Z * [new branch] gh/andyanwang/40/head -> origin/gh/andyanwang/40/head 2025-08-26T20:08:25.8819816Z * [new branch] gh/andyanwang/40/orig -> origin/gh/andyanwang/40/orig 2025-08-26T20:08:25.8821322Z * [new branch] gh/angelayi/106/base -> origin/gh/angelayi/106/base 2025-08-26T20:08:25.8821725Z * [new branch] gh/angelayi/106/head -> origin/gh/angelayi/106/head 2025-08-26T20:08:25.8822514Z * [new branch] gh/angelayi/106/orig -> origin/gh/angelayi/106/orig 2025-08-26T20:08:25.8823804Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-08-26T20:08:25.8824405Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-08-26T20:08:25.8825216Z * [new branch] gh/angelayi/108/base -> origin/gh/angelayi/108/base 2025-08-26T20:08:25.8825903Z * [new branch] gh/angelayi/108/head -> origin/gh/angelayi/108/head 2025-08-26T20:08:25.8826518Z * [new branch] gh/angelayi/108/orig -> origin/gh/angelayi/108/orig 2025-08-26T20:08:25.8828057Z * [new branch] gh/angelayi/109/base -> origin/gh/angelayi/109/base 2025-08-26T20:08:25.8828424Z * [new branch] gh/angelayi/109/head -> origin/gh/angelayi/109/head 2025-08-26T20:08:25.8829091Z * [new branch] gh/angelayi/109/orig -> origin/gh/angelayi/109/orig 2025-08-26T20:08:25.8830691Z * [new branch] gh/angelayi/110/base -> origin/gh/angelayi/110/base 2025-08-26T20:08:25.8831099Z * [new branch] gh/angelayi/110/head -> origin/gh/angelayi/110/head 2025-08-26T20:08:25.8831691Z * [new branch] gh/angelayi/110/orig -> origin/gh/angelayi/110/orig 2025-08-26T20:08:25.8832710Z * [new branch] gh/angelayi/111/base -> origin/gh/angelayi/111/base 2025-08-26T20:08:25.8833236Z * [new branch] gh/angelayi/111/head -> origin/gh/angelayi/111/head 2025-08-26T20:08:25.8833971Z * [new branch] gh/angelayi/111/orig -> origin/gh/angelayi/111/orig 2025-08-26T20:08:25.8835158Z * [new branch] gh/angelayi/112/base -> origin/gh/angelayi/112/base 2025-08-26T20:08:25.8835541Z * [new branch] gh/angelayi/112/head -> origin/gh/angelayi/112/head 2025-08-26T20:08:25.8836637Z * [new branch] gh/angelayi/112/orig -> origin/gh/angelayi/112/orig 2025-08-26T20:08:25.8837576Z * [new branch] gh/angelayi/113/base -> origin/gh/angelayi/113/base 2025-08-26T20:08:25.8838225Z * [new branch] gh/angelayi/113/head -> origin/gh/angelayi/113/head 2025-08-26T20:08:25.8841216Z * [new branch] gh/angelayi/113/orig -> origin/gh/angelayi/113/orig 2025-08-26T20:08:25.8841641Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-08-26T20:08:25.8841993Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-08-26T20:08:25.8842388Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-08-26T20:08:25.8842911Z * [new branch] gh/angelayi/115/base -> origin/gh/angelayi/115/base 2025-08-26T20:08:25.8843422Z * [new branch] gh/angelayi/115/head -> origin/gh/angelayi/115/head 2025-08-26T20:08:25.8844242Z * [new branch] gh/angelayi/115/orig -> origin/gh/angelayi/115/orig 2025-08-26T20:08:25.8845056Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-08-26T20:08:25.8845506Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-08-26T20:08:25.8846342Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-08-26T20:08:25.8847648Z * [new branch] gh/anijain2305/766/base -> origin/gh/anijain2305/766/base 2025-08-26T20:08:25.8848039Z * [new branch] gh/anijain2305/766/head -> origin/gh/anijain2305/766/head 2025-08-26T20:08:25.8849355Z * [new branch] gh/anijain2305/766/orig -> origin/gh/anijain2305/766/orig 2025-08-26T20:08:25.8849742Z * [new branch] gh/anijain2305/790/base -> origin/gh/anijain2305/790/base 2025-08-26T20:08:25.8850134Z * [new branch] gh/anijain2305/790/head -> origin/gh/anijain2305/790/head 2025-08-26T20:08:25.8850820Z * [new branch] gh/anijain2305/790/orig -> origin/gh/anijain2305/790/orig 2025-08-26T20:08:25.8851936Z * [new branch] gh/anijain2305/792/base -> origin/gh/anijain2305/792/base 2025-08-26T20:08:25.8852667Z * [new branch] gh/anijain2305/792/head -> origin/gh/anijain2305/792/head 2025-08-26T20:08:25.8856293Z * [new branch] gh/anijain2305/792/orig -> origin/gh/anijain2305/792/orig 2025-08-26T20:08:25.8856763Z * [new branch] gh/anijain2305/803/base -> origin/gh/anijain2305/803/base 2025-08-26T20:08:25.8857381Z * [new branch] gh/anijain2305/803/head -> origin/gh/anijain2305/803/head 2025-08-26T20:08:25.8857783Z * [new branch] gh/anijain2305/803/orig -> origin/gh/anijain2305/803/orig 2025-08-26T20:08:25.8858175Z * [new branch] gh/anijain2305/804/base -> origin/gh/anijain2305/804/base 2025-08-26T20:08:25.8858566Z * [new branch] gh/anijain2305/804/head -> origin/gh/anijain2305/804/head 2025-08-26T20:08:25.8863784Z * [new branch] gh/anijain2305/804/orig -> origin/gh/anijain2305/804/orig 2025-08-26T20:08:25.8864276Z * [new branch] gh/anijain2305/805/base -> origin/gh/anijain2305/805/base 2025-08-26T20:08:25.8864656Z * [new branch] gh/anijain2305/805/head -> origin/gh/anijain2305/805/head 2025-08-26T20:08:25.8865027Z * [new branch] gh/anijain2305/805/orig -> origin/gh/anijain2305/805/orig 2025-08-26T20:08:25.8865416Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-08-26T20:08:25.8865812Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-08-26T20:08:25.8866201Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-08-26T20:08:25.8866590Z * [new branch] gh/anijain2305/812/base -> origin/gh/anijain2305/812/base 2025-08-26T20:08:25.8867299Z * [new branch] gh/anijain2305/812/head -> origin/gh/anijain2305/812/head 2025-08-26T20:08:25.8867825Z * [new branch] gh/anijain2305/812/orig -> origin/gh/anijain2305/812/orig 2025-08-26T20:08:25.8868207Z * [new branch] gh/anijain2305/817/base -> origin/gh/anijain2305/817/base 2025-08-26T20:08:25.8868572Z * [new branch] gh/anijain2305/817/head -> origin/gh/anijain2305/817/head 2025-08-26T20:08:25.8868935Z * [new branch] gh/anijain2305/817/orig -> origin/gh/anijain2305/817/orig 2025-08-26T20:08:25.8869302Z * [new branch] gh/anijain2305/823/base -> origin/gh/anijain2305/823/base 2025-08-26T20:08:25.8869650Z * [new branch] gh/anijain2305/823/head -> origin/gh/anijain2305/823/head 2025-08-26T20:08:25.8871904Z * [new branch] gh/anijain2305/823/orig -> origin/gh/anijain2305/823/orig 2025-08-26T20:08:25.8872303Z * [new branch] gh/anijain2305/829/base -> origin/gh/anijain2305/829/base 2025-08-26T20:08:25.8872714Z * [new branch] gh/anijain2305/829/head -> origin/gh/anijain2305/829/head 2025-08-26T20:08:25.8873084Z * [new branch] gh/anijain2305/829/orig -> origin/gh/anijain2305/829/orig 2025-08-26T20:08:25.8873455Z * [new branch] gh/anijain2305/830/base -> origin/gh/anijain2305/830/base 2025-08-26T20:08:25.8873816Z * [new branch] gh/anijain2305/830/head -> origin/gh/anijain2305/830/head 2025-08-26T20:08:25.8874185Z * [new branch] gh/anijain2305/830/orig -> origin/gh/anijain2305/830/orig 2025-08-26T20:08:25.8874560Z * [new branch] gh/anijain2305/831/base -> origin/gh/anijain2305/831/base 2025-08-26T20:08:25.8874923Z * [new branch] gh/anijain2305/831/head -> origin/gh/anijain2305/831/head 2025-08-26T20:08:25.8875290Z * [new branch] gh/anijain2305/831/orig -> origin/gh/anijain2305/831/orig 2025-08-26T20:08:25.8876190Z * [new branch] gh/anijain2305/832/base -> origin/gh/anijain2305/832/base 2025-08-26T20:08:25.8876691Z * [new branch] gh/anijain2305/832/head -> origin/gh/anijain2305/832/head 2025-08-26T20:08:25.8877260Z * [new branch] gh/anijain2305/832/orig -> origin/gh/anijain2305/832/orig 2025-08-26T20:08:25.8877952Z * [new branch] gh/anijain2305/833/base -> origin/gh/anijain2305/833/base 2025-08-26T20:08:25.8878752Z * [new branch] gh/anijain2305/833/head -> origin/gh/anijain2305/833/head 2025-08-26T20:08:25.8879682Z * [new branch] gh/anijain2305/833/orig -> origin/gh/anijain2305/833/orig 2025-08-26T20:08:25.8881245Z * [new branch] gh/anijain2305/834/base -> origin/gh/anijain2305/834/base 2025-08-26T20:08:25.8881637Z * [new branch] gh/anijain2305/834/head -> origin/gh/anijain2305/834/head 2025-08-26T20:08:25.8883282Z * [new branch] gh/anijain2305/834/orig -> origin/gh/anijain2305/834/orig 2025-08-26T20:08:25.8883749Z * [new branch] gh/anijain2305/835/base -> origin/gh/anijain2305/835/base 2025-08-26T20:08:25.8884172Z * [new branch] gh/anijain2305/835/head -> origin/gh/anijain2305/835/head 2025-08-26T20:08:25.8884759Z * [new branch] gh/anijain2305/835/orig -> origin/gh/anijain2305/835/orig 2025-08-26T20:08:25.8886072Z * [new branch] gh/anijain2305/836/base -> origin/gh/anijain2305/836/base 2025-08-26T20:08:25.8886592Z * [new branch] gh/anijain2305/836/head -> origin/gh/anijain2305/836/head 2025-08-26T20:08:25.8887223Z * [new branch] gh/anijain2305/836/orig -> origin/gh/anijain2305/836/orig 2025-08-26T20:08:25.8888457Z * [new branch] gh/anijain2305/837/base -> origin/gh/anijain2305/837/base 2025-08-26T20:08:25.8888968Z * [new branch] gh/anijain2305/837/head -> origin/gh/anijain2305/837/head 2025-08-26T20:08:25.8889559Z * [new branch] gh/anijain2305/837/orig -> origin/gh/anijain2305/837/orig 2025-08-26T20:08:25.8890789Z * [new branch] gh/anijain2305/838/base -> origin/gh/anijain2305/838/base 2025-08-26T20:08:25.8891348Z * [new branch] gh/anijain2305/838/head -> origin/gh/anijain2305/838/head 2025-08-26T20:08:25.8891746Z * [new branch] gh/anijain2305/838/orig -> origin/gh/anijain2305/838/orig 2025-08-26T20:08:25.8893012Z * [new branch] gh/anijain2305/839/base -> origin/gh/anijain2305/839/base 2025-08-26T20:08:25.8893397Z * [new branch] gh/anijain2305/839/head -> origin/gh/anijain2305/839/head 2025-08-26T20:08:25.8894155Z * [new branch] gh/anijain2305/839/orig -> origin/gh/anijain2305/839/orig 2025-08-26T20:08:25.8895378Z * [new branch] gh/anijain2305/840/base -> origin/gh/anijain2305/840/base 2025-08-26T20:08:25.8896323Z * [new branch] gh/anijain2305/840/head -> origin/gh/anijain2305/840/head 2025-08-26T20:08:25.8896778Z * [new branch] gh/anijain2305/840/orig -> origin/gh/anijain2305/840/orig 2025-08-26T20:08:25.8900327Z * [new branch] gh/anijain2305/841/base -> origin/gh/anijain2305/841/base 2025-08-26T20:08:25.8900711Z * [new branch] gh/anijain2305/841/head -> origin/gh/anijain2305/841/head 2025-08-26T20:08:25.8901103Z * [new branch] gh/anijain2305/841/orig -> origin/gh/anijain2305/841/orig 2025-08-26T20:08:25.8901520Z * [new branch] gh/anijain2305/842/base -> origin/gh/anijain2305/842/base 2025-08-26T20:08:25.8901898Z * [new branch] gh/anijain2305/842/head -> origin/gh/anijain2305/842/head 2025-08-26T20:08:25.8902281Z * [new branch] gh/anijain2305/842/orig -> origin/gh/anijain2305/842/orig 2025-08-26T20:08:25.8904471Z * [new branch] gh/anijain2305/843/base -> origin/gh/anijain2305/843/base 2025-08-26T20:08:25.8904885Z * [new branch] gh/anijain2305/843/head -> origin/gh/anijain2305/843/head 2025-08-26T20:08:25.8905272Z * [new branch] gh/anijain2305/843/orig -> origin/gh/anijain2305/843/orig 2025-08-26T20:08:25.8905659Z * [new branch] gh/anijain2305/844/base -> origin/gh/anijain2305/844/base 2025-08-26T20:08:25.8906031Z * [new branch] gh/anijain2305/844/head -> origin/gh/anijain2305/844/head 2025-08-26T20:08:25.8906409Z * [new branch] gh/anijain2305/844/orig -> origin/gh/anijain2305/844/orig 2025-08-26T20:08:25.8908240Z * [new branch] gh/anijain2305/845/base -> origin/gh/anijain2305/845/base 2025-08-26T20:08:25.8908626Z * [new branch] gh/anijain2305/845/head -> origin/gh/anijain2305/845/head 2025-08-26T20:08:25.8909009Z * [new branch] gh/anijain2305/845/orig -> origin/gh/anijain2305/845/orig 2025-08-26T20:08:25.8909389Z * [new branch] gh/anijain2305/846/base -> origin/gh/anijain2305/846/base 2025-08-26T20:08:25.8909765Z * [new branch] gh/anijain2305/846/head -> origin/gh/anijain2305/846/head 2025-08-26T20:08:25.8911540Z * [new branch] gh/anijain2305/846/orig -> origin/gh/anijain2305/846/orig 2025-08-26T20:08:25.8914834Z * [new branch] gh/anijain2305/847/base -> origin/gh/anijain2305/847/base 2025-08-26T20:08:25.8915299Z * [new branch] gh/anijain2305/847/head -> origin/gh/anijain2305/847/head 2025-08-26T20:08:25.8915724Z * [new branch] gh/anijain2305/847/orig -> origin/gh/anijain2305/847/orig 2025-08-26T20:08:25.8916113Z * [new branch] gh/anijain2305/848/base -> origin/gh/anijain2305/848/base 2025-08-26T20:08:25.8916505Z * [new branch] gh/anijain2305/848/head -> origin/gh/anijain2305/848/head 2025-08-26T20:08:25.8930630Z * [new branch] gh/anijain2305/848/orig -> origin/gh/anijain2305/848/orig 2025-08-26T20:08:25.8933331Z * [new branch] gh/anijain2305/849/base -> origin/gh/anijain2305/849/base 2025-08-26T20:08:25.8934182Z * [new branch] gh/anijain2305/849/head -> origin/gh/anijain2305/849/head 2025-08-26T20:08:25.8934753Z * [new branch] gh/anijain2305/849/orig -> origin/gh/anijain2305/849/orig 2025-08-26T20:08:25.8935287Z * [new branch] gh/anijain2305/850/base -> origin/gh/anijain2305/850/base 2025-08-26T20:08:25.8935827Z * [new branch] gh/anijain2305/850/head -> origin/gh/anijain2305/850/head 2025-08-26T20:08:25.8936243Z * [new branch] gh/anijain2305/850/orig -> origin/gh/anijain2305/850/orig 2025-08-26T20:08:25.8936648Z * [new branch] gh/anijain2305/851/base -> origin/gh/anijain2305/851/base 2025-08-26T20:08:25.8937038Z * [new branch] gh/anijain2305/851/head -> origin/gh/anijain2305/851/head 2025-08-26T20:08:25.8937416Z * [new branch] gh/anijain2305/851/orig -> origin/gh/anijain2305/851/orig 2025-08-26T20:08:25.8937812Z * [new branch] gh/anijain2305/852/base -> origin/gh/anijain2305/852/base 2025-08-26T20:08:25.8938182Z * [new branch] gh/anijain2305/852/head -> origin/gh/anijain2305/852/head 2025-08-26T20:08:25.8938556Z * [new branch] gh/anijain2305/852/orig -> origin/gh/anijain2305/852/orig 2025-08-26T20:08:25.8938931Z * [new branch] gh/anijain2305/853/base -> origin/gh/anijain2305/853/base 2025-08-26T20:08:25.8939300Z * [new branch] gh/anijain2305/853/head -> origin/gh/anijain2305/853/head 2025-08-26T20:08:25.8939707Z * [new branch] gh/anijain2305/853/orig -> origin/gh/anijain2305/853/orig 2025-08-26T20:08:25.8940074Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-08-26T20:08:25.8940545Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-08-26T20:08:25.8940933Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-08-26T20:08:25.8941316Z * [new branch] gh/anijain2305/855/base -> origin/gh/anijain2305/855/base 2025-08-26T20:08:25.8941696Z * [new branch] gh/anijain2305/855/head -> origin/gh/anijain2305/855/head 2025-08-26T20:08:25.8942071Z * [new branch] gh/anijain2305/855/orig -> origin/gh/anijain2305/855/orig 2025-08-26T20:08:25.8942438Z * [new branch] gh/anijain2305/856/base -> origin/gh/anijain2305/856/base 2025-08-26T20:08:25.8942889Z * [new branch] gh/anijain2305/856/head -> origin/gh/anijain2305/856/head 2025-08-26T20:08:25.8943265Z * [new branch] gh/anijain2305/856/orig -> origin/gh/anijain2305/856/orig 2025-08-26T20:08:25.8943436Z * [new branch] gh/anijain2305/857/base -> origin/gh/anijain2305/857/base 2025-08-26T20:08:25.8943600Z * [new branch] gh/anijain2305/857/head -> origin/gh/anijain2305/857/head 2025-08-26T20:08:25.8943754Z * [new branch] gh/anijain2305/857/orig -> origin/gh/anijain2305/857/orig 2025-08-26T20:08:25.8943910Z * [new branch] gh/anijain2305/858/base -> origin/gh/anijain2305/858/base 2025-08-26T20:08:25.8944064Z * [new branch] gh/anijain2305/858/head -> origin/gh/anijain2305/858/head 2025-08-26T20:08:25.8944209Z * [new branch] gh/anijain2305/858/orig -> origin/gh/anijain2305/858/orig 2025-08-26T20:08:25.8944363Z * [new branch] gh/anijain2305/859/base -> origin/gh/anijain2305/859/base 2025-08-26T20:08:25.8944512Z * [new branch] gh/anijain2305/859/head -> origin/gh/anijain2305/859/head 2025-08-26T20:08:25.8944673Z * [new branch] gh/anijain2305/859/orig -> origin/gh/anijain2305/859/orig 2025-08-26T20:08:25.8944821Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-08-26T20:08:25.8944966Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-08-26T20:08:25.8945166Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-08-26T20:08:25.8945326Z * [new branch] gh/ankitageorge/13/base -> origin/gh/ankitageorge/13/base 2025-08-26T20:08:25.8945496Z * [new branch] gh/ankitageorge/13/head -> origin/gh/ankitageorge/13/head 2025-08-26T20:08:25.8945662Z * [new branch] gh/ankitageorge/13/orig -> origin/gh/ankitageorge/13/orig 2025-08-26T20:08:25.8946030Z * [new branch] gh/ankitageorge/14/base -> origin/gh/ankitageorge/14/base 2025-08-26T20:08:25.8946179Z * [new branch] gh/ankitageorge/14/head -> origin/gh/ankitageorge/14/head 2025-08-26T20:08:25.8946474Z * [new branch] gh/ankitageorge/14/orig -> origin/gh/ankitageorge/14/orig 2025-08-26T20:08:25.8946726Z * [new branch] gh/ankitageorge/15/base -> origin/gh/ankitageorge/15/base 2025-08-26T20:08:25.8946967Z * [new branch] gh/ankitageorge/15/head -> origin/gh/ankitageorge/15/head 2025-08-26T20:08:25.8947188Z * [new branch] gh/ankitageorge/15/orig -> origin/gh/ankitageorge/15/orig 2025-08-26T20:08:25.8952088Z * [new branch] gh/ankitageorge/16/base -> origin/gh/ankitageorge/16/base 2025-08-26T20:08:25.8952785Z * [new branch] gh/ankitageorge/16/head -> origin/gh/ankitageorge/16/head 2025-08-26T20:08:25.8952987Z * [new branch] gh/ankitageorge/16/orig -> origin/gh/ankitageorge/16/orig 2025-08-26T20:08:25.8953176Z * [new branch] gh/ankitageorge/17/base -> origin/gh/ankitageorge/17/base 2025-08-26T20:08:25.8953337Z * [new branch] gh/ankitageorge/17/head -> origin/gh/ankitageorge/17/head 2025-08-26T20:08:25.8953507Z * [new branch] gh/ankitageorge/17/orig -> origin/gh/ankitageorge/17/orig 2025-08-26T20:08:25.8953667Z * [new branch] gh/ankitageorge/18/base -> origin/gh/ankitageorge/18/base 2025-08-26T20:08:25.8957109Z * [new branch] gh/ankitageorge/18/head -> origin/gh/ankitageorge/18/head 2025-08-26T20:08:25.8957733Z * [new branch] gh/ankitageorge/18/orig -> origin/gh/ankitageorge/18/orig 2025-08-26T20:08:25.8957943Z * [new branch] gh/ankitageorge/19/base -> origin/gh/ankitageorge/19/base 2025-08-26T20:08:25.8958122Z * [new branch] gh/ankitageorge/19/head -> origin/gh/ankitageorge/19/head 2025-08-26T20:08:25.8958447Z * [new branch] gh/ankitageorge/19/orig -> origin/gh/ankitageorge/19/orig 2025-08-26T20:08:25.8958618Z * [new branch] gh/ankitageorge/20/base -> origin/gh/ankitageorge/20/base 2025-08-26T20:08:25.8958794Z * [new branch] gh/ankitageorge/20/head -> origin/gh/ankitageorge/20/head 2025-08-26T20:08:25.8959062Z * [new branch] gh/ankitageorge/20/orig -> origin/gh/ankitageorge/20/orig 2025-08-26T20:08:25.8963713Z * [new branch] gh/ankitageorge/21/base -> origin/gh/ankitageorge/21/base 2025-08-26T20:08:25.8963943Z * [new branch] gh/ankitageorge/21/head -> origin/gh/ankitageorge/21/head 2025-08-26T20:08:25.8964143Z * [new branch] gh/ankitageorge/21/orig -> origin/gh/ankitageorge/21/orig 2025-08-26T20:08:25.8964312Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-08-26T20:08:25.8964470Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-08-26T20:08:25.8964644Z * [new branch] gh/anshul-si/10/base -> origin/gh/anshul-si/10/base 2025-08-26T20:08:25.8964994Z * [new branch] gh/anshul-si/10/head -> origin/gh/anshul-si/10/head 2025-08-26T20:08:25.8970830Z * [new branch] gh/anshul-si/10/orig -> origin/gh/anshul-si/10/orig 2025-08-26T20:08:25.8971042Z * [new branch] gh/anshul-si/11/base -> origin/gh/anshul-si/11/base 2025-08-26T20:08:25.8971197Z * [new branch] gh/anshul-si/11/head -> origin/gh/anshul-si/11/head 2025-08-26T20:08:25.8971594Z * [new branch] gh/anshul-si/11/orig -> origin/gh/anshul-si/11/orig 2025-08-26T20:08:25.8971764Z * [new branch] gh/anshul-si/12/base -> origin/gh/anshul-si/12/base 2025-08-26T20:08:25.8971918Z * [new branch] gh/anshul-si/12/head -> origin/gh/anshul-si/12/head 2025-08-26T20:08:25.8972421Z * [new branch] gh/anshul-si/12/orig -> origin/gh/anshul-si/12/orig 2025-08-26T20:08:25.8972586Z * [new branch] gh/anshul-si/13/base -> origin/gh/anshul-si/13/base 2025-08-26T20:08:25.8972749Z * [new branch] gh/anshul-si/13/head -> origin/gh/anshul-si/13/head 2025-08-26T20:08:25.8972907Z * [new branch] gh/anshul-si/13/orig -> origin/gh/anshul-si/13/orig 2025-08-26T20:08:25.8973070Z * [new branch] gh/anshul-si/14/base -> origin/gh/anshul-si/14/base 2025-08-26T20:08:25.8974266Z * [new branch] gh/anshul-si/14/head -> origin/gh/anshul-si/14/head 2025-08-26T20:08:25.8974578Z * [new branch] gh/anshul-si/14/orig -> origin/gh/anshul-si/14/orig 2025-08-26T20:08:25.8975698Z * [new branch] gh/anshul-si/15/base -> origin/gh/anshul-si/15/base 2025-08-26T20:08:25.8976031Z * [new branch] gh/anshul-si/15/head -> origin/gh/anshul-si/15/head 2025-08-26T20:08:25.8977017Z * [new branch] gh/anshul-si/15/orig -> origin/gh/anshul-si/15/orig 2025-08-26T20:08:25.8978169Z * [new branch] gh/anshul-si/16/base -> origin/gh/anshul-si/16/base 2025-08-26T20:08:25.8978554Z * [new branch] gh/anshul-si/16/head -> origin/gh/anshul-si/16/head 2025-08-26T20:08:25.8979384Z * [new branch] gh/anshul-si/16/orig -> origin/gh/anshul-si/16/orig 2025-08-26T20:08:25.8981755Z * [new branch] gh/anshul-si/17/base -> origin/gh/anshul-si/17/base 2025-08-26T20:08:25.8981929Z * [new branch] gh/anshul-si/17/head -> origin/gh/anshul-si/17/head 2025-08-26T20:08:25.8982099Z * [new branch] gh/anshul-si/17/orig -> origin/gh/anshul-si/17/orig 2025-08-26T20:08:25.8983311Z * [new branch] gh/anshul-si/18/base -> origin/gh/anshul-si/18/base 2025-08-26T20:08:25.8983879Z * [new branch] gh/anshul-si/18/head -> origin/gh/anshul-si/18/head 2025-08-26T20:08:25.8984572Z * [new branch] gh/anshul-si/18/orig -> origin/gh/anshul-si/18/orig 2025-08-26T20:08:25.8987220Z * [new branch] gh/anshul-si/19/base -> origin/gh/anshul-si/19/base 2025-08-26T20:08:25.8987417Z * [new branch] gh/anshul-si/19/head -> origin/gh/anshul-si/19/head 2025-08-26T20:08:25.8987576Z * [new branch] gh/anshul-si/19/orig -> origin/gh/anshul-si/19/orig 2025-08-26T20:08:25.8987744Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-08-26T20:08:25.8988585Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-08-26T20:08:25.8990499Z * [new branch] gh/anshul-si/20/base -> origin/gh/anshul-si/20/base 2025-08-26T20:08:25.8990692Z * [new branch] gh/anshul-si/20/head -> origin/gh/anshul-si/20/head 2025-08-26T20:08:25.8990858Z * [new branch] gh/anshul-si/20/orig -> origin/gh/anshul-si/20/orig 2025-08-26T20:08:25.8991959Z * [new branch] gh/anshul-si/21/base -> origin/gh/anshul-si/21/base 2025-08-26T20:08:25.8992383Z * [new branch] gh/anshul-si/21/head -> origin/gh/anshul-si/21/head 2025-08-26T20:08:25.8994071Z * [new branch] gh/anshul-si/21/orig -> origin/gh/anshul-si/21/orig 2025-08-26T20:08:25.8994420Z * [new branch] gh/anshul-si/22/base -> origin/gh/anshul-si/22/base 2025-08-26T20:08:25.8994796Z * [new branch] gh/anshul-si/22/head -> origin/gh/anshul-si/22/head 2025-08-26T20:08:25.8997399Z * [new branch] gh/anshul-si/22/orig -> origin/gh/anshul-si/22/orig 2025-08-26T20:08:25.8997615Z * [new branch] gh/anshul-si/23/base -> origin/gh/anshul-si/23/base 2025-08-26T20:08:25.8997772Z * [new branch] gh/anshul-si/23/head -> origin/gh/anshul-si/23/head 2025-08-26T20:08:25.8998872Z * [new branch] gh/anshul-si/23/orig -> origin/gh/anshul-si/23/orig 2025-08-26T20:08:25.9000639Z * [new branch] gh/anshul-si/24/base -> origin/gh/anshul-si/24/base 2025-08-26T20:08:25.9000839Z * [new branch] gh/anshul-si/24/head -> origin/gh/anshul-si/24/head 2025-08-26T20:08:25.9001097Z * [new branch] gh/anshul-si/24/orig -> origin/gh/anshul-si/24/orig 2025-08-26T20:08:25.9002868Z * [new branch] gh/anshul-si/25/base -> origin/gh/anshul-si/25/base 2025-08-26T20:08:25.9003062Z * [new branch] gh/anshul-si/25/head -> origin/gh/anshul-si/25/head 2025-08-26T20:08:25.9007128Z * [new branch] gh/anshul-si/25/orig -> origin/gh/anshul-si/25/orig 2025-08-26T20:08:25.9007334Z * [new branch] gh/anshul-si/26/base -> origin/gh/anshul-si/26/base 2025-08-26T20:08:25.9007607Z * [new branch] gh/anshul-si/26/head -> origin/gh/anshul-si/26/head 2025-08-26T20:08:25.9007760Z * [new branch] gh/anshul-si/26/orig -> origin/gh/anshul-si/26/orig 2025-08-26T20:08:25.9007899Z * [new branch] gh/anshul-si/27/base -> origin/gh/anshul-si/27/base 2025-08-26T20:08:25.9008053Z * [new branch] gh/anshul-si/27/head -> origin/gh/anshul-si/27/head 2025-08-26T20:08:25.9008191Z * [new branch] gh/anshul-si/27/orig -> origin/gh/anshul-si/27/orig 2025-08-26T20:08:25.9009782Z * [new branch] gh/anshul-si/28/base -> origin/gh/anshul-si/28/base 2025-08-26T20:08:25.9010192Z * [new branch] gh/anshul-si/28/head -> origin/gh/anshul-si/28/head 2025-08-26T20:08:25.9010372Z * [new branch] gh/anshul-si/28/orig -> origin/gh/anshul-si/28/orig 2025-08-26T20:08:25.9012467Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-08-26T20:08:25.9012842Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-08-26T20:08:25.9013262Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-08-26T20:08:25.9013815Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-08-26T20:08:25.9014242Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-08-26T20:08:25.9015430Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-08-26T20:08:25.9020948Z * [new branch] gh/anshul-si/7/base -> origin/gh/anshul-si/7/base 2025-08-26T20:08:25.9021169Z * [new branch] gh/anshul-si/7/head -> origin/gh/anshul-si/7/head 2025-08-26T20:08:25.9021326Z * [new branch] gh/anshul-si/7/orig -> origin/gh/anshul-si/7/orig 2025-08-26T20:08:25.9021480Z * [new branch] gh/anshul-si/8/base -> origin/gh/anshul-si/8/base 2025-08-26T20:08:25.9021643Z * [new branch] gh/anshul-si/8/head -> origin/gh/anshul-si/8/head 2025-08-26T20:08:25.9021806Z * [new branch] gh/anshul-si/8/orig -> origin/gh/anshul-si/8/orig 2025-08-26T20:08:25.9027314Z * [new branch] gh/anshul-si/9/base -> origin/gh/anshul-si/9/base 2025-08-26T20:08:25.9027680Z * [new branch] gh/anshul-si/9/head -> origin/gh/anshul-si/9/head 2025-08-26T20:08:25.9027859Z * [new branch] gh/anshul-si/9/orig -> origin/gh/anshul-si/9/orig 2025-08-26T20:08:25.9028021Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-08-26T20:08:25.9028699Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-08-26T20:08:25.9028953Z * [new branch] gh/aorenste/237/base -> origin/gh/aorenste/237/base 2025-08-26T20:08:25.9029325Z * [new branch] gh/aorenste/237/head -> origin/gh/aorenste/237/head 2025-08-26T20:08:25.9034136Z * [new branch] gh/aorenste/237/orig -> origin/gh/aorenste/237/orig 2025-08-26T20:08:25.9034379Z * [new branch] gh/aorenste/238/base -> origin/gh/aorenste/238/base 2025-08-26T20:08:25.9034529Z * [new branch] gh/aorenste/238/head -> origin/gh/aorenste/238/head 2025-08-26T20:08:25.9034676Z * [new branch] gh/aorenste/238/orig -> origin/gh/aorenste/238/orig 2025-08-26T20:08:25.9034977Z * [new branch] gh/bdhirsh/650/base -> origin/gh/bdhirsh/650/base 2025-08-26T20:08:25.9035210Z * [new branch] gh/bdhirsh/650/head -> origin/gh/bdhirsh/650/head 2025-08-26T20:08:25.9035359Z * [new branch] gh/bdhirsh/650/orig -> origin/gh/bdhirsh/650/orig 2025-08-26T20:08:25.9035634Z * [new branch] gh/bdhirsh/656/base -> origin/gh/bdhirsh/656/base 2025-08-26T20:08:25.9038502Z * [new branch] gh/bdhirsh/656/head -> origin/gh/bdhirsh/656/head 2025-08-26T20:08:25.9038688Z * [new branch] gh/bdhirsh/657/base -> origin/gh/bdhirsh/657/base 2025-08-26T20:08:25.9039404Z * [new branch] gh/bdhirsh/657/head -> origin/gh/bdhirsh/657/head 2025-08-26T20:08:25.9039654Z * [new branch] gh/bdhirsh/663/base -> origin/gh/bdhirsh/663/base 2025-08-26T20:08:25.9039814Z * [new branch] gh/bdhirsh/663/head -> origin/gh/bdhirsh/663/head 2025-08-26T20:08:25.9039974Z * [new branch] gh/bdhirsh/663/orig -> origin/gh/bdhirsh/663/orig 2025-08-26T20:08:25.9040124Z * [new branch] gh/bdhirsh/665/base -> origin/gh/bdhirsh/665/base 2025-08-26T20:08:25.9040285Z * [new branch] gh/bdhirsh/665/head -> origin/gh/bdhirsh/665/head 2025-08-26T20:08:25.9040511Z * [new branch] gh/bdhirsh/665/orig -> origin/gh/bdhirsh/665/orig 2025-08-26T20:08:25.9040662Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-08-26T20:08:25.9047149Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-08-26T20:08:25.9049863Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-08-26T20:08:25.9050061Z * [new branch] gh/bdhirsh/667/base -> origin/gh/bdhirsh/667/base 2025-08-26T20:08:25.9050248Z * [new branch] gh/bdhirsh/667/head -> origin/gh/bdhirsh/667/head 2025-08-26T20:08:25.9050398Z * [new branch] gh/bdhirsh/667/orig -> origin/gh/bdhirsh/667/orig 2025-08-26T20:08:25.9050570Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-08-26T20:08:25.9050722Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-08-26T20:08:25.9050870Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-08-26T20:08:25.9051064Z * [new branch] gh/benjaminglass1/100/base -> origin/gh/benjaminglass1/100/base 2025-08-26T20:08:25.9051235Z * [new branch] gh/benjaminglass1/100/head -> origin/gh/benjaminglass1/100/head 2025-08-26T20:08:25.9051406Z * [new branch] gh/benjaminglass1/100/orig -> origin/gh/benjaminglass1/100/orig 2025-08-26T20:08:25.9051579Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-08-26T20:08:25.9051759Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-08-26T20:08:25.9061357Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-08-26T20:08:25.9066727Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-08-26T20:08:25.9066958Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-08-26T20:08:25.9067162Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-08-26T20:08:25.9067333Z * [new branch] gh/benjaminglass1/103/base -> origin/gh/benjaminglass1/103/base 2025-08-26T20:08:25.9067525Z * [new branch] gh/benjaminglass1/103/head -> origin/gh/benjaminglass1/103/head 2025-08-26T20:08:25.9067698Z * [new branch] gh/benjaminglass1/103/orig -> origin/gh/benjaminglass1/103/orig 2025-08-26T20:08:25.9067886Z * [new branch] gh/benjaminglass1/79/base -> origin/gh/benjaminglass1/79/base 2025-08-26T20:08:25.9068060Z * [new branch] gh/benjaminglass1/79/head -> origin/gh/benjaminglass1/79/head 2025-08-26T20:08:25.9068235Z * [new branch] gh/benjaminglass1/79/orig -> origin/gh/benjaminglass1/79/orig 2025-08-26T20:08:25.9068420Z * [new branch] gh/benjaminglass1/86/base -> origin/gh/benjaminglass1/86/base 2025-08-26T20:08:25.9068593Z * [new branch] gh/benjaminglass1/86/head -> origin/gh/benjaminglass1/86/head 2025-08-26T20:08:25.9068760Z * [new branch] gh/benjaminglass1/86/orig -> origin/gh/benjaminglass1/86/orig 2025-08-26T20:08:25.9068919Z * [new branch] gh/benjaminglass1/89/base -> origin/gh/benjaminglass1/89/base 2025-08-26T20:08:25.9069087Z * [new branch] gh/benjaminglass1/89/head -> origin/gh/benjaminglass1/89/head 2025-08-26T20:08:25.9069252Z * [new branch] gh/benjaminglass1/89/orig -> origin/gh/benjaminglass1/89/orig 2025-08-26T20:08:25.9069417Z * [new branch] gh/benjaminglass1/91/base -> origin/gh/benjaminglass1/91/base 2025-08-26T20:08:25.9069588Z * [new branch] gh/benjaminglass1/91/head -> origin/gh/benjaminglass1/91/head 2025-08-26T20:08:25.9069768Z * [new branch] gh/benjaminglass1/91/orig -> origin/gh/benjaminglass1/91/orig 2025-08-26T20:08:25.9069940Z * [new branch] gh/benjaminglass1/93/base -> origin/gh/benjaminglass1/93/base 2025-08-26T20:08:25.9070114Z * [new branch] gh/benjaminglass1/93/head -> origin/gh/benjaminglass1/93/head 2025-08-26T20:08:25.9070336Z * [new branch] gh/benjaminglass1/93/orig -> origin/gh/benjaminglass1/93/orig 2025-08-26T20:08:25.9070495Z * [new branch] gh/benjaminglass1/95/base -> origin/gh/benjaminglass1/95/base 2025-08-26T20:08:25.9070657Z * [new branch] gh/benjaminglass1/95/head -> origin/gh/benjaminglass1/95/head 2025-08-26T20:08:25.9070822Z * [new branch] gh/benjaminglass1/95/orig -> origin/gh/benjaminglass1/95/orig 2025-08-26T20:08:25.9070981Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-08-26T20:08:25.9071159Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-08-26T20:08:25.9071323Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-08-26T20:08:25.9071969Z * [new branch] gh/benjaminglass1/98/base -> origin/gh/benjaminglass1/98/base 2025-08-26T20:08:25.9072699Z * [new branch] gh/benjaminglass1/98/head -> origin/gh/benjaminglass1/98/head 2025-08-26T20:08:25.9073290Z * [new branch] gh/benjaminglass1/98/orig -> origin/gh/benjaminglass1/98/orig 2025-08-26T20:08:25.9074531Z * [new branch] gh/benjaminglass1/99/base -> origin/gh/benjaminglass1/99/base 2025-08-26T20:08:25.9074999Z * [new branch] gh/benjaminglass1/99/head -> origin/gh/benjaminglass1/99/head 2025-08-26T20:08:25.9075994Z * [new branch] gh/benjaminglass1/99/orig -> origin/gh/benjaminglass1/99/orig 2025-08-26T20:08:25.9077317Z * [new branch] gh/bobrenjc93/514/base -> origin/gh/bobrenjc93/514/base 2025-08-26T20:08:25.9077590Z * [new branch] gh/bobrenjc93/514/head -> origin/gh/bobrenjc93/514/head 2025-08-26T20:08:25.9078987Z * [new branch] gh/bobrenjc93/514/orig -> origin/gh/bobrenjc93/514/orig 2025-08-26T20:08:25.9079518Z * [new branch] gh/bobrenjc93/521/base -> origin/gh/bobrenjc93/521/base 2025-08-26T20:08:25.9080073Z * [new branch] gh/bobrenjc93/521/head -> origin/gh/bobrenjc93/521/head 2025-08-26T20:08:25.9080847Z * [new branch] gh/bobrenjc93/521/orig -> origin/gh/bobrenjc93/521/orig 2025-08-26T20:08:25.9086395Z * [new branch] gh/bobrenjc93/522/base -> origin/gh/bobrenjc93/522/base 2025-08-26T20:08:25.9086611Z * [new branch] gh/bobrenjc93/522/head -> origin/gh/bobrenjc93/522/head 2025-08-26T20:08:25.9086775Z * [new branch] gh/bobrenjc93/522/orig -> origin/gh/bobrenjc93/522/orig 2025-08-26T20:08:25.9086974Z * [new branch] gh/bobrenjc93/525/base -> origin/gh/bobrenjc93/525/base 2025-08-26T20:08:25.9087132Z * [new branch] gh/bobrenjc93/525/head -> origin/gh/bobrenjc93/525/head 2025-08-26T20:08:25.9087292Z * [new branch] gh/bobrenjc93/525/orig -> origin/gh/bobrenjc93/525/orig 2025-08-26T20:08:25.9087446Z * [new branch] gh/bobrenjc93/526/base -> origin/gh/bobrenjc93/526/base 2025-08-26T20:08:25.9087608Z * [new branch] gh/bobrenjc93/526/head -> origin/gh/bobrenjc93/526/head 2025-08-26T20:08:25.9087816Z * [new branch] gh/bobrenjc93/526/orig -> origin/gh/bobrenjc93/526/orig 2025-08-26T20:08:25.9092737Z * [new branch] gh/bobrenjc93/527/base -> origin/gh/bobrenjc93/527/base 2025-08-26T20:08:25.9092944Z * [new branch] gh/bobrenjc93/527/head -> origin/gh/bobrenjc93/527/head 2025-08-26T20:08:25.9093104Z * [new branch] gh/bobrenjc93/527/orig -> origin/gh/bobrenjc93/527/orig 2025-08-26T20:08:25.9093280Z * [new branch] gh/bobrenjc93/528/base -> origin/gh/bobrenjc93/528/base 2025-08-26T20:08:25.9093432Z * [new branch] gh/bobrenjc93/528/head -> origin/gh/bobrenjc93/528/head 2025-08-26T20:08:25.9093594Z * [new branch] gh/bobrenjc93/528/orig -> origin/gh/bobrenjc93/528/orig 2025-08-26T20:08:25.9093899Z * [new branch] gh/bobrenjc93/529/base -> origin/gh/bobrenjc93/529/base 2025-08-26T20:08:25.9102282Z * [new branch] gh/bobrenjc93/529/head -> origin/gh/bobrenjc93/529/head 2025-08-26T20:08:25.9104551Z * [new branch] gh/bobrenjc93/529/orig -> origin/gh/bobrenjc93/529/orig 2025-08-26T20:08:25.9104916Z * [new branch] gh/bobrenjc93/535/base -> origin/gh/bobrenjc93/535/base 2025-08-26T20:08:25.9105103Z * [new branch] gh/bobrenjc93/535/head -> origin/gh/bobrenjc93/535/head 2025-08-26T20:08:25.9105286Z * [new branch] gh/bobrenjc93/535/orig -> origin/gh/bobrenjc93/535/orig 2025-08-26T20:08:25.9105541Z * [new branch] gh/bobrenjc93/537/base -> origin/gh/bobrenjc93/537/base 2025-08-26T20:08:25.9105739Z * [new branch] gh/bobrenjc93/537/head -> origin/gh/bobrenjc93/537/head 2025-08-26T20:08:25.9105897Z * [new branch] gh/bobrenjc93/537/orig -> origin/gh/bobrenjc93/537/orig 2025-08-26T20:08:25.9106198Z * [new branch] gh/bobrenjc93/538/base -> origin/gh/bobrenjc93/538/base 2025-08-26T20:08:25.9106940Z * [new branch] gh/bobrenjc93/538/head -> origin/gh/bobrenjc93/538/head 2025-08-26T20:08:25.9107392Z * [new branch] gh/bobrenjc93/538/orig -> origin/gh/bobrenjc93/538/orig 2025-08-26T20:08:25.9107719Z * [new branch] gh/bobrenjc93/539/base -> origin/gh/bobrenjc93/539/base 2025-08-26T20:08:25.9107898Z * [new branch] gh/bobrenjc93/539/head -> origin/gh/bobrenjc93/539/head 2025-08-26T20:08:25.9108420Z * [new branch] gh/bobrenjc93/539/orig -> origin/gh/bobrenjc93/539/orig 2025-08-26T20:08:25.9108596Z * [new branch] gh/bobrenjc93/540/base -> origin/gh/bobrenjc93/540/base 2025-08-26T20:08:25.9108762Z * [new branch] gh/bobrenjc93/540/head -> origin/gh/bobrenjc93/540/head 2025-08-26T20:08:25.9108917Z * [new branch] gh/bobrenjc93/540/orig -> origin/gh/bobrenjc93/540/orig 2025-08-26T20:08:25.9112388Z * [new branch] gh/bobrenjc93/541/base -> origin/gh/bobrenjc93/541/base 2025-08-26T20:08:25.9112630Z * [new branch] gh/bobrenjc93/541/head -> origin/gh/bobrenjc93/541/head 2025-08-26T20:08:25.9112788Z * [new branch] gh/bobrenjc93/541/orig -> origin/gh/bobrenjc93/541/orig 2025-08-26T20:08:25.9112959Z * [new branch] gh/bobrenjc93/542/base -> origin/gh/bobrenjc93/542/base 2025-08-26T20:08:25.9113122Z * [new branch] gh/bobrenjc93/542/head -> origin/gh/bobrenjc93/542/head 2025-08-26T20:08:25.9113282Z * [new branch] gh/bobrenjc93/542/orig -> origin/gh/bobrenjc93/542/orig 2025-08-26T20:08:25.9113468Z * [new branch] gh/bobrenjc93/543/base -> origin/gh/bobrenjc93/543/base 2025-08-26T20:08:25.9113625Z * [new branch] gh/bobrenjc93/543/head -> origin/gh/bobrenjc93/543/head 2025-08-26T20:08:25.9114282Z * [new branch] gh/bobrenjc93/543/orig -> origin/gh/bobrenjc93/543/orig 2025-08-26T20:08:25.9114770Z * [new branch] gh/bobrenjc93/544/base -> origin/gh/bobrenjc93/544/base 2025-08-26T20:08:25.9116656Z * [new branch] gh/bobrenjc93/544/head -> origin/gh/bobrenjc93/544/head 2025-08-26T20:08:25.9116916Z * [new branch] gh/bobrenjc93/544/orig -> origin/gh/bobrenjc93/544/orig 2025-08-26T20:08:25.9117086Z * [new branch] gh/bobrenjc93/545/base -> origin/gh/bobrenjc93/545/base 2025-08-26T20:08:25.9117676Z * [new branch] gh/bobrenjc93/545/head -> origin/gh/bobrenjc93/545/head 2025-08-26T20:08:25.9118633Z * [new branch] gh/bobrenjc93/545/orig -> origin/gh/bobrenjc93/545/orig 2025-08-26T20:08:25.9119859Z * [new branch] gh/bobrenjc93/546/base -> origin/gh/bobrenjc93/546/base 2025-08-26T20:08:25.9120073Z * [new branch] gh/bobrenjc93/546/head -> origin/gh/bobrenjc93/546/head 2025-08-26T20:08:25.9124666Z * [new branch] gh/bobrenjc93/546/orig -> origin/gh/bobrenjc93/546/orig 2025-08-26T20:08:25.9128614Z * [new branch] gh/bobrenjc93/547/base -> origin/gh/bobrenjc93/547/base 2025-08-26T20:08:25.9135231Z * [new branch] gh/bobrenjc93/547/head -> origin/gh/bobrenjc93/547/head 2025-08-26T20:08:25.9135445Z * [new branch] gh/bobrenjc93/547/orig -> origin/gh/bobrenjc93/547/orig 2025-08-26T20:08:25.9135679Z * [new branch] gh/bobrenjc93/548/base -> origin/gh/bobrenjc93/548/base 2025-08-26T20:08:25.9135881Z * [new branch] gh/bobrenjc93/548/head -> origin/gh/bobrenjc93/548/head 2025-08-26T20:08:25.9136052Z * [new branch] gh/bobrenjc93/548/orig -> origin/gh/bobrenjc93/548/orig 2025-08-26T20:08:25.9136205Z * [new branch] gh/bobrenjc93/549/base -> origin/gh/bobrenjc93/549/base 2025-08-26T20:08:25.9136357Z * [new branch] gh/bobrenjc93/549/head -> origin/gh/bobrenjc93/549/head 2025-08-26T20:08:25.9136531Z * [new branch] gh/bobrenjc93/549/orig -> origin/gh/bobrenjc93/549/orig 2025-08-26T20:08:25.9136700Z * [new branch] gh/bobrenjc93/550/base -> origin/gh/bobrenjc93/550/base 2025-08-26T20:08:25.9136863Z * [new branch] gh/bobrenjc93/550/head -> origin/gh/bobrenjc93/550/head 2025-08-26T20:08:25.9137027Z * [new branch] gh/bobrenjc93/550/orig -> origin/gh/bobrenjc93/550/orig 2025-08-26T20:08:25.9137329Z * [new branch] gh/bobrenjc93/551/base -> origin/gh/bobrenjc93/551/base 2025-08-26T20:08:25.9137498Z * [new branch] gh/bobrenjc93/551/head -> origin/gh/bobrenjc93/551/head 2025-08-26T20:08:25.9137657Z * [new branch] gh/bobrenjc93/551/orig -> origin/gh/bobrenjc93/551/orig 2025-08-26T20:08:25.9137844Z * [new branch] gh/briancoutinho/2/base -> origin/gh/briancoutinho/2/base 2025-08-26T20:08:25.9138029Z * [new branch] gh/briancoutinho/2/head -> origin/gh/briancoutinho/2/head 2025-08-26T20:08:25.9138178Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-08-26T20:08:25.9144795Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-08-26T20:08:25.9144975Z * [new branch] gh/c00w/38/base -> origin/gh/c00w/38/base 2025-08-26T20:08:25.9145397Z * [new branch] gh/c00w/38/head -> origin/gh/c00w/38/head 2025-08-26T20:08:25.9145585Z * [new branch] gh/c00w/38/orig -> origin/gh/c00w/38/orig 2025-08-26T20:08:25.9145727Z * [new branch] gh/c00w/48/base -> origin/gh/c00w/48/base 2025-08-26T20:08:25.9145869Z * [new branch] gh/c00w/48/head -> origin/gh/c00w/48/head 2025-08-26T20:08:25.9146003Z * [new branch] gh/c00w/48/orig -> origin/gh/c00w/48/orig 2025-08-26T20:08:25.9146144Z * [new branch] gh/c00w/51/base -> origin/gh/c00w/51/base 2025-08-26T20:08:25.9148616Z * [new branch] gh/c00w/51/head -> origin/gh/c00w/51/head 2025-08-26T20:08:25.9149415Z * [new branch] gh/c00w/51/orig -> origin/gh/c00w/51/orig 2025-08-26T20:08:25.9149589Z * [new branch] gh/c00w/52/base -> origin/gh/c00w/52/base 2025-08-26T20:08:25.9149892Z * [new branch] gh/c00w/52/head -> origin/gh/c00w/52/head 2025-08-26T20:08:25.9150054Z * [new branch] gh/c00w/52/orig -> origin/gh/c00w/52/orig 2025-08-26T20:08:25.9150190Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-08-26T20:08:25.9150338Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-08-26T20:08:25.9153175Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-08-26T20:08:25.9153461Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-08-26T20:08:25.9153605Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-08-26T20:08:25.9153786Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-08-26T20:08:25.9153941Z * [new branch] gh/c00w/55/base -> origin/gh/c00w/55/base 2025-08-26T20:08:25.9154085Z * [new branch] gh/c00w/55/head -> origin/gh/c00w/55/head 2025-08-26T20:08:25.9154232Z * [new branch] gh/c00w/55/orig -> origin/gh/c00w/55/orig 2025-08-26T20:08:25.9155409Z * [new branch] gh/chenmillie/1/base -> origin/gh/chenmillie/1/base 2025-08-26T20:08:25.9155817Z * [new branch] gh/chenmillie/1/head -> origin/gh/chenmillie/1/head 2025-08-26T20:08:25.9157098Z * [new branch] gh/chenmillie/1/orig -> origin/gh/chenmillie/1/orig 2025-08-26T20:08:25.9157980Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-08-26T20:08:25.9158688Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-08-26T20:08:25.9159495Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-08-26T20:08:25.9160870Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-08-26T20:08:25.9161591Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-08-26T20:08:25.9162914Z * [new branch] gh/coconutruben/11/base -> origin/gh/coconutruben/11/base 2025-08-26T20:08:25.9163807Z * [new branch] gh/coconutruben/11/head -> origin/gh/coconutruben/11/head 2025-08-26T20:08:25.9164326Z * [new branch] gh/coconutruben/11/orig -> origin/gh/coconutruben/11/orig 2025-08-26T20:08:25.9166282Z * [new branch] gh/coconutruben/12/base -> origin/gh/coconutruben/12/base 2025-08-26T20:08:25.9167327Z * [new branch] gh/coconutruben/12/head -> origin/gh/coconutruben/12/head 2025-08-26T20:08:25.9168461Z * [new branch] gh/coconutruben/12/orig -> origin/gh/coconutruben/12/orig 2025-08-26T20:08:25.9169356Z * [new branch] gh/coconutruben/13/base -> origin/gh/coconutruben/13/base 2025-08-26T20:08:25.9169821Z * [new branch] gh/coconutruben/13/head -> origin/gh/coconutruben/13/head 2025-08-26T20:08:25.9172857Z * [new branch] gh/coconutruben/13/orig -> origin/gh/coconutruben/13/orig 2025-08-26T20:08:25.9173086Z * [new branch] gh/coconutruben/14/base -> origin/gh/coconutruben/14/base 2025-08-26T20:08:25.9173249Z * [new branch] gh/coconutruben/14/head -> origin/gh/coconutruben/14/head 2025-08-26T20:08:25.9173431Z * [new branch] gh/coconutruben/14/orig -> origin/gh/coconutruben/14/orig 2025-08-26T20:08:25.9174778Z * [new branch] gh/coconutruben/15/base -> origin/gh/coconutruben/15/base 2025-08-26T20:08:25.9175325Z * [new branch] gh/coconutruben/15/head -> origin/gh/coconutruben/15/head 2025-08-26T20:08:25.9176294Z * [new branch] gh/coconutruben/15/orig -> origin/gh/coconutruben/15/orig 2025-08-26T20:08:25.9177331Z * [new branch] gh/coconutruben/16/base -> origin/gh/coconutruben/16/base 2025-08-26T20:08:25.9177592Z * [new branch] gh/coconutruben/16/head -> origin/gh/coconutruben/16/head 2025-08-26T20:08:25.9181875Z * [new branch] gh/coconutruben/16/orig -> origin/gh/coconutruben/16/orig 2025-08-26T20:08:25.9182130Z * [new branch] gh/coconutruben/17/base -> origin/gh/coconutruben/17/base 2025-08-26T20:08:25.9182309Z * [new branch] gh/coconutruben/17/head -> origin/gh/coconutruben/17/head 2025-08-26T20:08:25.9182471Z * [new branch] gh/coconutruben/17/orig -> origin/gh/coconutruben/17/orig 2025-08-26T20:08:25.9182783Z * [new branch] gh/coconutruben/18/base -> origin/gh/coconutruben/18/base 2025-08-26T20:08:25.9183367Z * [new branch] gh/coconutruben/18/head -> origin/gh/coconutruben/18/head 2025-08-26T20:08:25.9183637Z * [new branch] gh/coconutruben/18/orig -> origin/gh/coconutruben/18/orig 2025-08-26T20:08:25.9185158Z * [new branch] gh/coconutruben/19/base -> origin/gh/coconutruben/19/base 2025-08-26T20:08:25.9185652Z * [new branch] gh/coconutruben/19/head -> origin/gh/coconutruben/19/head 2025-08-26T20:08:25.9186601Z * [new branch] gh/coconutruben/19/orig -> origin/gh/coconutruben/19/orig 2025-08-26T20:08:25.9187691Z * [new branch] gh/coconutruben/20/base -> origin/gh/coconutruben/20/base 2025-08-26T20:08:25.9188508Z * [new branch] gh/coconutruben/20/head -> origin/gh/coconutruben/20/head 2025-08-26T20:08:25.9188995Z * [new branch] gh/coconutruben/20/orig -> origin/gh/coconutruben/20/orig 2025-08-26T20:08:25.9190361Z * [new branch] gh/coconutruben/21/base -> origin/gh/coconutruben/21/base 2025-08-26T20:08:25.9190639Z * [new branch] gh/coconutruben/21/head -> origin/gh/coconutruben/21/head 2025-08-26T20:08:25.9191691Z * [new branch] gh/coconutruben/21/orig -> origin/gh/coconutruben/21/orig 2025-08-26T20:08:25.9192993Z * [new branch] gh/coconutruben/22/base -> origin/gh/coconutruben/22/base 2025-08-26T20:08:25.9194309Z * [new branch] gh/coconutruben/22/head -> origin/gh/coconutruben/22/head 2025-08-26T20:08:25.9194510Z * [new branch] gh/coconutruben/22/orig -> origin/gh/coconutruben/22/orig 2025-08-26T20:08:25.9195683Z * [new branch] gh/coconutruben/23/base -> origin/gh/coconutruben/23/base 2025-08-26T20:08:25.9195843Z * [new branch] gh/coconutruben/23/head -> origin/gh/coconutruben/23/head 2025-08-26T20:08:25.9196697Z * [new branch] gh/coconutruben/23/orig -> origin/gh/coconutruben/23/orig 2025-08-26T20:08:25.9197861Z * [new branch] gh/coconutruben/24/base -> origin/gh/coconutruben/24/base 2025-08-26T20:08:25.9198446Z * [new branch] gh/coconutruben/24/head -> origin/gh/coconutruben/24/head 2025-08-26T20:08:25.9199944Z * [new branch] gh/coconutruben/24/orig -> origin/gh/coconutruben/24/orig 2025-08-26T20:08:25.9200906Z * [new branch] gh/coconutruben/25/base -> origin/gh/coconutruben/25/base 2025-08-26T20:08:25.9204472Z * [new branch] gh/coconutruben/25/head -> origin/gh/coconutruben/25/head 2025-08-26T20:08:25.9204681Z * [new branch] gh/coconutruben/25/orig -> origin/gh/coconutruben/25/orig 2025-08-26T20:08:25.9204853Z * [new branch] gh/coconutruben/26/base -> origin/gh/coconutruben/26/base 2025-08-26T20:08:25.9205009Z * [new branch] gh/coconutruben/26/head -> origin/gh/coconutruben/26/head 2025-08-26T20:08:25.9205557Z * [new branch] gh/coconutruben/26/orig -> origin/gh/coconutruben/26/orig 2025-08-26T20:08:25.9206870Z * [new branch] gh/coconutruben/27/base -> origin/gh/coconutruben/27/base 2025-08-26T20:08:25.9207043Z * [new branch] gh/coconutruben/27/head -> origin/gh/coconutruben/27/head 2025-08-26T20:08:25.9211371Z * [new branch] gh/coconutruben/27/orig -> origin/gh/coconutruben/27/orig 2025-08-26T20:08:25.9211594Z * [new branch] gh/coconutruben/28/base -> origin/gh/coconutruben/28/base 2025-08-26T20:08:25.9212090Z * [new branch] gh/coconutruben/28/head -> origin/gh/coconutruben/28/head 2025-08-26T20:08:25.9212254Z * [new branch] gh/coconutruben/28/orig -> origin/gh/coconutruben/28/orig 2025-08-26T20:08:25.9212414Z * [new branch] gh/coconutruben/29/base -> origin/gh/coconutruben/29/base 2025-08-26T20:08:25.9212792Z * [new branch] gh/coconutruben/29/head -> origin/gh/coconutruben/29/head 2025-08-26T20:08:25.9213768Z * [new branch] gh/coconutruben/29/orig -> origin/gh/coconutruben/29/orig 2025-08-26T20:08:25.9215133Z * [new branch] gh/coconutruben/30/base -> origin/gh/coconutruben/30/base 2025-08-26T20:08:25.9215733Z * [new branch] gh/coconutruben/30/head -> origin/gh/coconutruben/30/head 2025-08-26T20:08:25.9216628Z * [new branch] gh/coconutruben/30/orig -> origin/gh/coconutruben/30/orig 2025-08-26T20:08:25.9217754Z * [new branch] gh/coconutruben/31/base -> origin/gh/coconutruben/31/base 2025-08-26T20:08:25.9218202Z * [new branch] gh/coconutruben/31/head -> origin/gh/coconutruben/31/head 2025-08-26T20:08:25.9219263Z * [new branch] gh/coconutruben/31/orig -> origin/gh/coconutruben/31/orig 2025-08-26T20:08:25.9220664Z * [new branch] gh/coconutruben/32/base -> origin/gh/coconutruben/32/base 2025-08-26T20:08:25.9221498Z * [new branch] gh/coconutruben/32/head -> origin/gh/coconutruben/32/head 2025-08-26T20:08:25.9223157Z * [new branch] gh/coconutruben/32/orig -> origin/gh/coconutruben/32/orig 2025-08-26T20:08:25.9223731Z * [new branch] gh/coconutruben/33/base -> origin/gh/coconutruben/33/base 2025-08-26T20:08:25.9223935Z * [new branch] gh/coconutruben/33/head -> origin/gh/coconutruben/33/head 2025-08-26T20:08:25.9226374Z * [new branch] gh/coconutruben/33/orig -> origin/gh/coconutruben/33/orig 2025-08-26T20:08:25.9226610Z * [new branch] gh/coconutruben/34/base -> origin/gh/coconutruben/34/base 2025-08-26T20:08:25.9226785Z * [new branch] gh/coconutruben/34/head -> origin/gh/coconutruben/34/head 2025-08-26T20:08:25.9226950Z * [new branch] gh/coconutruben/34/orig -> origin/gh/coconutruben/34/orig 2025-08-26T20:08:25.9227741Z * [new branch] gh/coconutruben/35/base -> origin/gh/coconutruben/35/base 2025-08-26T20:08:25.9228394Z * [new branch] gh/coconutruben/35/head -> origin/gh/coconutruben/35/head 2025-08-26T20:08:25.9228984Z * [new branch] gh/coconutruben/35/orig -> origin/gh/coconutruben/35/orig 2025-08-26T20:08:25.9234849Z * [new branch] gh/coconutruben/36/base -> origin/gh/coconutruben/36/base 2025-08-26T20:08:25.9235029Z * [new branch] gh/coconutruben/36/head -> origin/gh/coconutruben/36/head 2025-08-26T20:08:25.9235217Z * [new branch] gh/coconutruben/36/orig -> origin/gh/coconutruben/36/orig 2025-08-26T20:08:25.9235386Z * [new branch] gh/coconutruben/37/base -> origin/gh/coconutruben/37/base 2025-08-26T20:08:25.9235851Z * [new branch] gh/coconutruben/37/head -> origin/gh/coconutruben/37/head 2025-08-26T20:08:25.9237161Z * [new branch] gh/coconutruben/37/orig -> origin/gh/coconutruben/37/orig 2025-08-26T20:08:25.9237694Z * [new branch] gh/coconutruben/38/base -> origin/gh/coconutruben/38/base 2025-08-26T20:08:25.9238661Z * [new branch] gh/coconutruben/38/head -> origin/gh/coconutruben/38/head 2025-08-26T20:08:25.9239243Z * [new branch] gh/coconutruben/38/orig -> origin/gh/coconutruben/38/orig 2025-08-26T20:08:25.9242951Z * [new branch] gh/coconutruben/39/base -> origin/gh/coconutruben/39/base 2025-08-26T20:08:25.9243122Z * [new branch] gh/coconutruben/39/head -> origin/gh/coconutruben/39/head 2025-08-26T20:08:25.9243281Z * [new branch] gh/coconutruben/39/orig -> origin/gh/coconutruben/39/orig 2025-08-26T20:08:25.9249695Z * [new branch] gh/coconutruben/40/base -> origin/gh/coconutruben/40/base 2025-08-26T20:08:25.9254665Z * [new branch] gh/coconutruben/40/head -> origin/gh/coconutruben/40/head 2025-08-26T20:08:25.9259742Z * [new branch] gh/coconutruben/40/orig -> origin/gh/coconutruben/40/orig 2025-08-26T20:08:25.9259963Z * [new branch] gh/coconutruben/41/base -> origin/gh/coconutruben/41/base 2025-08-26T20:08:25.9260136Z * [new branch] gh/coconutruben/41/head -> origin/gh/coconutruben/41/head 2025-08-26T20:08:25.9260297Z * [new branch] gh/coconutruben/41/orig -> origin/gh/coconutruben/41/orig 2025-08-26T20:08:25.9260485Z * [new branch] gh/coconutruben/42/base -> origin/gh/coconutruben/42/base 2025-08-26T20:08:25.9260680Z * [new branch] gh/coconutruben/42/head -> origin/gh/coconutruben/42/head 2025-08-26T20:08:25.9260850Z * [new branch] gh/coconutruben/42/orig -> origin/gh/coconutruben/42/orig 2025-08-26T20:08:25.9261009Z * [new branch] gh/coconutruben/43/base -> origin/gh/coconutruben/43/base 2025-08-26T20:08:25.9261173Z * [new branch] gh/coconutruben/43/head -> origin/gh/coconutruben/43/head 2025-08-26T20:08:25.9261338Z * [new branch] gh/coconutruben/43/orig -> origin/gh/coconutruben/43/orig 2025-08-26T20:08:25.9261491Z * [new branch] gh/coconutruben/44/base -> origin/gh/coconutruben/44/base 2025-08-26T20:08:25.9261662Z * [new branch] gh/coconutruben/44/head -> origin/gh/coconutruben/44/head 2025-08-26T20:08:25.9261825Z * [new branch] gh/coconutruben/44/orig -> origin/gh/coconutruben/44/orig 2025-08-26T20:08:25.9261992Z * [new branch] gh/coconutruben/45/base -> origin/gh/coconutruben/45/base 2025-08-26T20:08:25.9262314Z * [new branch] gh/coconutruben/45/head -> origin/gh/coconutruben/45/head 2025-08-26T20:08:25.9262490Z * [new branch] gh/coconutruben/45/orig -> origin/gh/coconutruben/45/orig 2025-08-26T20:08:25.9262652Z * [new branch] gh/coconutruben/46/base -> origin/gh/coconutruben/46/base 2025-08-26T20:08:25.9262806Z * [new branch] gh/coconutruben/46/head -> origin/gh/coconutruben/46/head 2025-08-26T20:08:25.9262977Z * [new branch] gh/coconutruben/46/orig -> origin/gh/coconutruben/46/orig 2025-08-26T20:08:25.9263137Z * [new branch] gh/coconutruben/47/base -> origin/gh/coconutruben/47/base 2025-08-26T20:08:25.9263304Z * [new branch] gh/coconutruben/47/head -> origin/gh/coconutruben/47/head 2025-08-26T20:08:25.9263464Z * [new branch] gh/coconutruben/47/orig -> origin/gh/coconutruben/47/orig 2025-08-26T20:08:25.9263665Z * [new branch] gh/coconutruben/48/base -> origin/gh/coconutruben/48/base 2025-08-26T20:08:25.9263819Z * [new branch] gh/coconutruben/48/head -> origin/gh/coconutruben/48/head 2025-08-26T20:08:25.9264030Z * [new branch] gh/coconutruben/48/orig -> origin/gh/coconutruben/48/orig 2025-08-26T20:08:25.9266179Z * [new branch] gh/coconutruben/49/base -> origin/gh/coconutruben/49/base 2025-08-26T20:08:25.9266580Z * [new branch] gh/coconutruben/49/head -> origin/gh/coconutruben/49/head 2025-08-26T20:08:25.9266901Z * [new branch] gh/coconutruben/49/orig -> origin/gh/coconutruben/49/orig 2025-08-26T20:08:25.9267423Z * [new branch] gh/coconutruben/50/base -> origin/gh/coconutruben/50/base 2025-08-26T20:08:25.9269189Z * [new branch] gh/coconutruben/50/head -> origin/gh/coconutruben/50/head 2025-08-26T20:08:25.9269389Z * [new branch] gh/coconutruben/50/orig -> origin/gh/coconutruben/50/orig 2025-08-26T20:08:25.9270474Z * [new branch] gh/coconutruben/51/base -> origin/gh/coconutruben/51/base 2025-08-26T20:08:25.9271881Z * [new branch] gh/coconutruben/51/head -> origin/gh/coconutruben/51/head 2025-08-26T20:08:25.9272078Z * [new branch] gh/coconutruben/51/orig -> origin/gh/coconutruben/51/orig 2025-08-26T20:08:25.9272719Z * [new branch] gh/coconutruben/52/base -> origin/gh/coconutruben/52/base 2025-08-26T20:08:25.9273705Z * [new branch] gh/coconutruben/52/head -> origin/gh/coconutruben/52/head 2025-08-26T20:08:25.9274043Z * [new branch] gh/coconutruben/52/orig -> origin/gh/coconutruben/52/orig 2025-08-26T20:08:25.9275268Z * [new branch] gh/coconutruben/53/base -> origin/gh/coconutruben/53/base 2025-08-26T20:08:25.9275981Z * [new branch] gh/coconutruben/53/head -> origin/gh/coconutruben/53/head 2025-08-26T20:08:25.9276593Z * [new branch] gh/coconutruben/53/orig -> origin/gh/coconutruben/53/orig 2025-08-26T20:08:25.9277806Z * [new branch] gh/coconutruben/54/base -> origin/gh/coconutruben/54/base 2025-08-26T20:08:25.9278305Z * [new branch] gh/coconutruben/54/head -> origin/gh/coconutruben/54/head 2025-08-26T20:08:25.9279477Z * [new branch] gh/coconutruben/54/orig -> origin/gh/coconutruben/54/orig 2025-08-26T20:08:25.9280082Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-08-26T20:08:25.9281090Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-08-26T20:08:25.9281486Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-08-26T20:08:25.9283543Z * [new branch] gh/codingwithsurya/12/base -> origin/gh/codingwithsurya/12/base 2025-08-26T20:08:25.9283900Z * [new branch] gh/codingwithsurya/12/head -> origin/gh/codingwithsurya/12/head 2025-08-26T20:08:25.9288854Z * [new branch] gh/codingwithsurya/12/orig -> origin/gh/codingwithsurya/12/orig 2025-08-26T20:08:25.9289071Z * [new branch] gh/codingwithsurya/13/base -> origin/gh/codingwithsurya/13/base 2025-08-26T20:08:25.9289241Z * [new branch] gh/codingwithsurya/13/head -> origin/gh/codingwithsurya/13/head 2025-08-26T20:08:25.9289427Z * [new branch] gh/codingwithsurya/13/orig -> origin/gh/codingwithsurya/13/orig 2025-08-26T20:08:25.9289589Z * [new branch] gh/codingwithsurya/14/base -> origin/gh/codingwithsurya/14/base 2025-08-26T20:08:25.9289772Z * [new branch] gh/codingwithsurya/14/head -> origin/gh/codingwithsurya/14/head 2025-08-26T20:08:25.9290005Z * [new branch] gh/codingwithsurya/14/orig -> origin/gh/codingwithsurya/14/orig 2025-08-26T20:08:25.9294537Z * [new branch] gh/codingwithsurya/15/base -> origin/gh/codingwithsurya/15/base 2025-08-26T20:08:25.9294755Z * [new branch] gh/codingwithsurya/15/head -> origin/gh/codingwithsurya/15/head 2025-08-26T20:08:25.9294931Z * [new branch] gh/codingwithsurya/15/orig -> origin/gh/codingwithsurya/15/orig 2025-08-26T20:08:25.9295095Z * [new branch] gh/codingwithsurya/16/base -> origin/gh/codingwithsurya/16/base 2025-08-26T20:08:25.9295255Z * [new branch] gh/codingwithsurya/16/head -> origin/gh/codingwithsurya/16/head 2025-08-26T20:08:25.9295895Z * [new branch] gh/codingwithsurya/16/orig -> origin/gh/codingwithsurya/16/orig 2025-08-26T20:08:25.9297176Z * [new branch] gh/codingwithsurya/17/base -> origin/gh/codingwithsurya/17/base 2025-08-26T20:08:25.9297661Z * [new branch] gh/codingwithsurya/17/head -> origin/gh/codingwithsurya/17/head 2025-08-26T20:08:25.9299803Z * [new branch] gh/codingwithsurya/17/orig -> origin/gh/codingwithsurya/17/orig 2025-08-26T20:08:25.9300027Z * [new branch] gh/codingwithsurya/18/base -> origin/gh/codingwithsurya/18/base 2025-08-26T20:08:25.9300383Z * [new branch] gh/codingwithsurya/18/head -> origin/gh/codingwithsurya/18/head 2025-08-26T20:08:25.9304238Z * [new branch] gh/codingwithsurya/18/orig -> origin/gh/codingwithsurya/18/orig 2025-08-26T20:08:25.9304637Z * [new branch] gh/codingwithsurya/19/base -> origin/gh/codingwithsurya/19/base 2025-08-26T20:08:25.9305265Z * [new branch] gh/codingwithsurya/19/head -> origin/gh/codingwithsurya/19/head 2025-08-26T20:08:25.9306034Z * [new branch] gh/codingwithsurya/19/orig -> origin/gh/codingwithsurya/19/orig 2025-08-26T20:08:25.9306257Z * [new branch] gh/codingwithsurya/20/base -> origin/gh/codingwithsurya/20/base 2025-08-26T20:08:25.9306621Z * [new branch] gh/codingwithsurya/20/head -> origin/gh/codingwithsurya/20/head 2025-08-26T20:08:25.9306798Z * [new branch] gh/codingwithsurya/20/orig -> origin/gh/codingwithsurya/20/orig 2025-08-26T20:08:25.9307651Z * [new branch] gh/codingwithsurya/21/base -> origin/gh/codingwithsurya/21/base 2025-08-26T20:08:25.9312220Z * [new branch] gh/codingwithsurya/21/head -> origin/gh/codingwithsurya/21/head 2025-08-26T20:08:25.9314652Z * [new branch] gh/codingwithsurya/21/orig -> origin/gh/codingwithsurya/21/orig 2025-08-26T20:08:25.9315260Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-08-26T20:08:25.9315724Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-08-26T20:08:25.9315900Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-08-26T20:08:25.9316332Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-08-26T20:08:25.9316522Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-08-26T20:08:25.9316927Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-08-26T20:08:25.9317097Z * [new branch] gh/colinchan15/4/base -> origin/gh/colinchan15/4/base 2025-08-26T20:08:25.9317260Z * [new branch] gh/colinchan15/4/head -> origin/gh/colinchan15/4/head 2025-08-26T20:08:25.9317429Z * [new branch] gh/colinchan15/5/base -> origin/gh/colinchan15/5/base 2025-08-26T20:08:25.9317611Z * [new branch] gh/colinchan15/5/head -> origin/gh/colinchan15/5/head 2025-08-26T20:08:25.9319363Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-08-26T20:08:25.9320000Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-08-26T20:08:25.9323920Z * [new branch] gh/davidberard98/382/base -> origin/gh/davidberard98/382/base 2025-08-26T20:08:25.9324128Z * [new branch] gh/davidberard98/382/head -> origin/gh/davidberard98/382/head 2025-08-26T20:08:25.9324368Z * [new branch] gh/davidberard98/382/orig -> origin/gh/davidberard98/382/orig 2025-08-26T20:08:25.9324633Z * [new branch] gh/davidberard98/386/base -> origin/gh/davidberard98/386/base 2025-08-26T20:08:25.9324988Z * [new branch] gh/davidberard98/386/head -> origin/gh/davidberard98/386/head 2025-08-26T20:08:25.9325314Z * [new branch] gh/davidberard98/386/orig -> origin/gh/davidberard98/386/orig 2025-08-26T20:08:25.9325554Z * [new branch] gh/davidberard98/391/base -> origin/gh/davidberard98/391/base 2025-08-26T20:08:25.9325805Z * [new branch] gh/davidberard98/391/head -> origin/gh/davidberard98/391/head 2025-08-26T20:08:25.9326549Z * [new branch] gh/davidberard98/391/orig -> origin/gh/davidberard98/391/orig 2025-08-26T20:08:25.9327233Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-08-26T20:08:25.9327855Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-08-26T20:08:25.9328833Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-08-26T20:08:25.9333207Z * [new branch] gh/davidberard98/393/base -> origin/gh/davidberard98/393/base 2025-08-26T20:08:25.9333421Z * [new branch] gh/davidberard98/393/head -> origin/gh/davidberard98/393/head 2025-08-26T20:08:25.9333823Z * [new branch] gh/davidberard98/393/orig -> origin/gh/davidberard98/393/orig 2025-08-26T20:08:25.9334000Z * [new branch] gh/davidberard98/394/base -> origin/gh/davidberard98/394/base 2025-08-26T20:08:25.9334177Z * [new branch] gh/davidberard98/394/head -> origin/gh/davidberard98/394/head 2025-08-26T20:08:25.9334345Z * [new branch] gh/davidberard98/394/orig -> origin/gh/davidberard98/394/orig 2025-08-26T20:08:25.9334861Z * [new branch] gh/davidberard98/395/base -> origin/gh/davidberard98/395/base 2025-08-26T20:08:25.9335188Z * [new branch] gh/davidberard98/395/head -> origin/gh/davidberard98/395/head 2025-08-26T20:08:25.9336524Z * [new branch] gh/davidberard98/395/orig -> origin/gh/davidberard98/395/orig 2025-08-26T20:08:25.9336860Z * [new branch] gh/davidberard98/396/base -> origin/gh/davidberard98/396/base 2025-08-26T20:08:25.9337712Z * [new branch] gh/davidberard98/396/head -> origin/gh/davidberard98/396/head 2025-08-26T20:08:25.9338775Z * [new branch] gh/davidberard98/396/orig -> origin/gh/davidberard98/396/orig 2025-08-26T20:08:25.9339711Z * [new branch] gh/davidberard98/397/base -> origin/gh/davidberard98/397/base 2025-08-26T20:08:25.9340014Z * [new branch] gh/davidberard98/397/head -> origin/gh/davidberard98/397/head 2025-08-26T20:08:25.9341118Z * [new branch] gh/davidberard98/397/orig -> origin/gh/davidberard98/397/orig 2025-08-26T20:08:25.9341753Z * [new branch] gh/davidberard98/398/base -> origin/gh/davidberard98/398/base 2025-08-26T20:08:25.9342367Z * [new branch] gh/davidberard98/398/head -> origin/gh/davidberard98/398/head 2025-08-26T20:08:25.9343045Z * [new branch] gh/davidberard98/398/orig -> origin/gh/davidberard98/398/orig 2025-08-26T20:08:25.9344188Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-08-26T20:08:25.9344654Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-08-26T20:08:25.9346112Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-08-26T20:08:25.9346291Z * [new branch] gh/davidberard98/400/base -> origin/gh/davidberard98/400/base 2025-08-26T20:08:25.9347393Z * [new branch] gh/davidberard98/400/head -> origin/gh/davidberard98/400/head 2025-08-26T20:08:25.9347862Z * [new branch] gh/davidberard98/400/orig -> origin/gh/davidberard98/400/orig 2025-08-26T20:08:25.9349086Z * [new branch] gh/davidberard98/401/base -> origin/gh/davidberard98/401/base 2025-08-26T20:08:25.9349362Z * [new branch] gh/davidberard98/401/head -> origin/gh/davidberard98/401/head 2025-08-26T20:08:25.9350417Z * [new branch] gh/davidberard98/401/orig -> origin/gh/davidberard98/401/orig 2025-08-26T20:08:25.9351326Z * [new branch] gh/davidberard98/402/base -> origin/gh/davidberard98/402/base 2025-08-26T20:08:25.9351760Z * [new branch] gh/davidberard98/402/head -> origin/gh/davidberard98/402/head 2025-08-26T20:08:25.9352718Z * [new branch] gh/davidberard98/402/orig -> origin/gh/davidberard98/402/orig 2025-08-26T20:08:25.9353980Z * [new branch] gh/desertfire/589/base -> origin/gh/desertfire/589/base 2025-08-26T20:08:25.9354538Z * [new branch] gh/desertfire/589/head -> origin/gh/desertfire/589/head 2025-08-26T20:08:25.9355666Z * [new branch] gh/desertfire/589/orig -> origin/gh/desertfire/589/orig 2025-08-26T20:08:25.9356357Z * [new branch] gh/desertfire/591/base -> origin/gh/desertfire/591/base 2025-08-26T20:08:25.9357036Z * [new branch] gh/desertfire/591/head -> origin/gh/desertfire/591/head 2025-08-26T20:08:25.9358021Z * [new branch] gh/desertfire/591/orig -> origin/gh/desertfire/591/orig 2025-08-26T20:08:25.9359046Z * [new branch] gh/desertfire/592/base -> origin/gh/desertfire/592/base 2025-08-26T20:08:25.9359636Z * [new branch] gh/desertfire/592/head -> origin/gh/desertfire/592/head 2025-08-26T20:08:25.9360796Z * [new branch] gh/desertfire/592/orig -> origin/gh/desertfire/592/orig 2025-08-26T20:08:25.9361767Z * [new branch] gh/desertfire/593/base -> origin/gh/desertfire/593/base 2025-08-26T20:08:25.9361991Z * [new branch] gh/desertfire/593/head -> origin/gh/desertfire/593/head 2025-08-26T20:08:25.9363104Z * [new branch] gh/desertfire/593/orig -> origin/gh/desertfire/593/orig 2025-08-26T20:08:25.9365035Z * [new branch] gh/desertfire/594/base -> origin/gh/desertfire/594/base 2025-08-26T20:08:25.9365277Z * [new branch] gh/desertfire/594/head -> origin/gh/desertfire/594/head 2025-08-26T20:08:25.9365553Z * [new branch] gh/desertfire/594/orig -> origin/gh/desertfire/594/orig 2025-08-26T20:08:25.9367505Z * [new branch] gh/desertfire/595/base -> origin/gh/desertfire/595/base 2025-08-26T20:08:25.9367715Z * [new branch] gh/desertfire/595/head -> origin/gh/desertfire/595/head 2025-08-26T20:08:25.9367906Z * [new branch] gh/desertfire/595/orig -> origin/gh/desertfire/595/orig 2025-08-26T20:08:25.9369438Z * [new branch] gh/desertfire/596/base -> origin/gh/desertfire/596/base 2025-08-26T20:08:25.9369804Z * [new branch] gh/desertfire/596/head -> origin/gh/desertfire/596/head 2025-08-26T20:08:25.9370331Z * [new branch] gh/desertfire/596/orig -> origin/gh/desertfire/596/orig 2025-08-26T20:08:25.9371651Z * [new branch] gh/desertfire/597/base -> origin/gh/desertfire/597/base 2025-08-26T20:08:25.9371818Z * [new branch] gh/desertfire/597/head -> origin/gh/desertfire/597/head 2025-08-26T20:08:25.9372645Z * [new branch] gh/desertfire/597/orig -> origin/gh/desertfire/597/orig 2025-08-26T20:08:25.9379510Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-08-26T20:08:25.9379768Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-08-26T20:08:25.9379914Z * [new branch] gh/dharakk/4/base -> origin/gh/dharakk/4/base 2025-08-26T20:08:25.9380081Z * [new branch] gh/dharakk/4/head -> origin/gh/dharakk/4/head 2025-08-26T20:08:25.9380240Z * [new branch] gh/dharakk/4/orig -> origin/gh/dharakk/4/orig 2025-08-26T20:08:25.9380413Z * [new branch] gh/drisspg/149/base -> origin/gh/drisspg/149/base 2025-08-26T20:08:25.9380566Z * [new branch] gh/drisspg/149/head -> origin/gh/drisspg/149/head 2025-08-26T20:08:25.9380720Z * [new branch] gh/drisspg/149/orig -> origin/gh/drisspg/149/orig 2025-08-26T20:08:25.9380878Z * [new branch] gh/drisspg/150/base -> origin/gh/drisspg/150/base 2025-08-26T20:08:25.9381035Z * [new branch] gh/drisspg/150/head -> origin/gh/drisspg/150/head 2025-08-26T20:08:25.9381512Z * [new branch] gh/drisspg/150/orig -> origin/gh/drisspg/150/orig 2025-08-26T20:08:25.9382650Z * [new branch] gh/drisspg/151/base -> origin/gh/drisspg/151/base 2025-08-26T20:08:25.9383042Z * [new branch] gh/drisspg/151/head -> origin/gh/drisspg/151/head 2025-08-26T20:08:25.9384021Z * [new branch] gh/drisspg/151/orig -> origin/gh/drisspg/151/orig 2025-08-26T20:08:25.9384959Z * [new branch] gh/drisspg/159/base -> origin/gh/drisspg/159/base 2025-08-26T20:08:25.9385431Z * [new branch] gh/drisspg/159/head -> origin/gh/drisspg/159/head 2025-08-26T20:08:25.9386103Z * [new branch] gh/drisspg/159/orig -> origin/gh/drisspg/159/orig 2025-08-26T20:08:25.9387308Z * [new branch] gh/drisspg/166/base -> origin/gh/drisspg/166/base 2025-08-26T20:08:25.9387597Z * [new branch] gh/drisspg/166/head -> origin/gh/drisspg/166/head 2025-08-26T20:08:25.9389377Z * [new branch] gh/drisspg/166/orig -> origin/gh/drisspg/166/orig 2025-08-26T20:08:25.9389790Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-08-26T20:08:25.9390563Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-08-26T20:08:25.9391161Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-08-26T20:08:25.9392395Z * [new branch] gh/drisspg/172/base -> origin/gh/drisspg/172/base 2025-08-26T20:08:25.9392673Z * [new branch] gh/drisspg/172/head -> origin/gh/drisspg/172/head 2025-08-26T20:08:25.9393732Z * [new branch] gh/drisspg/172/orig -> origin/gh/drisspg/172/orig 2025-08-26T20:08:25.9395316Z * [new branch] gh/drisspg/173/base -> origin/gh/drisspg/173/base 2025-08-26T20:08:25.9395561Z * [new branch] gh/drisspg/173/head -> origin/gh/drisspg/173/head 2025-08-26T20:08:25.9395718Z * [new branch] gh/drisspg/173/orig -> origin/gh/drisspg/173/orig 2025-08-26T20:08:25.9397417Z * [new branch] gh/drisspg/175/base -> origin/gh/drisspg/175/base 2025-08-26T20:08:25.9397804Z * [new branch] gh/drisspg/175/head -> origin/gh/drisspg/175/head 2025-08-26T20:08:25.9398713Z * [new branch] gh/drisspg/175/orig -> origin/gh/drisspg/175/orig 2025-08-26T20:08:25.9399744Z * [new branch] gh/drisspg/176/base -> origin/gh/drisspg/176/base 2025-08-26T20:08:25.9400071Z * [new branch] gh/drisspg/176/head -> origin/gh/drisspg/176/head 2025-08-26T20:08:25.9401265Z * [new branch] gh/drisspg/176/orig -> origin/gh/drisspg/176/orig 2025-08-26T20:08:25.9402137Z * [new branch] gh/drisspg/177/base -> origin/gh/drisspg/177/base 2025-08-26T20:08:25.9402380Z * [new branch] gh/drisspg/177/head -> origin/gh/drisspg/177/head 2025-08-26T20:08:25.9403458Z * [new branch] gh/drisspg/177/orig -> origin/gh/drisspg/177/orig 2025-08-26T20:08:25.9404112Z * [new branch] gh/drisspg/178/base -> origin/gh/drisspg/178/base 2025-08-26T20:08:25.9404978Z * [new branch] gh/drisspg/178/head -> origin/gh/drisspg/178/head 2025-08-26T20:08:25.9405141Z * [new branch] gh/drisspg/178/orig -> origin/gh/drisspg/178/orig 2025-08-26T20:08:25.9406506Z * [new branch] gh/drisspg/179/base -> origin/gh/drisspg/179/base 2025-08-26T20:08:25.9406802Z * [new branch] gh/drisspg/179/head -> origin/gh/drisspg/179/head 2025-08-26T20:08:25.9408008Z * [new branch] gh/drisspg/179/orig -> origin/gh/drisspg/179/orig 2025-08-26T20:08:25.9408569Z * [new branch] gh/drisspg/180/base -> origin/gh/drisspg/180/base 2025-08-26T20:08:25.9413422Z * [new branch] gh/drisspg/180/head -> origin/gh/drisspg/180/head 2025-08-26T20:08:25.9413962Z * [new branch] gh/drisspg/180/orig -> origin/gh/drisspg/180/orig 2025-08-26T20:08:25.9414137Z * [new branch] gh/drisspg/181/base -> origin/gh/drisspg/181/base 2025-08-26T20:08:25.9422192Z * [new branch] gh/drisspg/181/head -> origin/gh/drisspg/181/head 2025-08-26T20:08:25.9422386Z * [new branch] gh/drisspg/181/orig -> origin/gh/drisspg/181/orig 2025-08-26T20:08:25.9422590Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-08-26T20:08:25.9422767Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-08-26T20:08:25.9423137Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-08-26T20:08:25.9423297Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-08-26T20:08:25.9423444Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-08-26T20:08:25.9423598Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-08-26T20:08:25.9423742Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-08-26T20:08:25.9423924Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-08-26T20:08:25.9424067Z * [new branch] gh/drisspg/186/base -> origin/gh/drisspg/186/base 2025-08-26T20:08:25.9424220Z * [new branch] gh/drisspg/186/head -> origin/gh/drisspg/186/head 2025-08-26T20:08:25.9424361Z * [new branch] gh/drisspg/186/orig -> origin/gh/drisspg/186/orig 2025-08-26T20:08:25.9424994Z * [new branch] gh/drisspg/187/base -> origin/gh/drisspg/187/base 2025-08-26T20:08:25.9425169Z * [new branch] gh/drisspg/187/head -> origin/gh/drisspg/187/head 2025-08-26T20:08:25.9425315Z * [new branch] gh/drisspg/187/orig -> origin/gh/drisspg/187/orig 2025-08-26T20:08:25.9425449Z * [new branch] gh/drisspg/188/base -> origin/gh/drisspg/188/base 2025-08-26T20:08:25.9425589Z * [new branch] gh/drisspg/188/head -> origin/gh/drisspg/188/head 2025-08-26T20:08:25.9425885Z * [new branch] gh/drisspg/188/orig -> origin/gh/drisspg/188/orig 2025-08-26T20:08:25.9426040Z * [new branch] gh/drisspg/189/base -> origin/gh/drisspg/189/base 2025-08-26T20:08:25.9426179Z * [new branch] gh/drisspg/189/head -> origin/gh/drisspg/189/head 2025-08-26T20:08:25.9429690Z * [new branch] gh/drisspg/189/orig -> origin/gh/drisspg/189/orig 2025-08-26T20:08:25.9429949Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-08-26T20:08:25.9430562Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-08-26T20:08:25.9430763Z * [new branch] gh/eellison/784/base -> origin/gh/eellison/784/base 2025-08-26T20:08:25.9435448Z * [new branch] gh/eellison/784/head -> origin/gh/eellison/784/head 2025-08-26T20:08:25.9435667Z * [new branch] gh/eellison/784/orig -> origin/gh/eellison/784/orig 2025-08-26T20:08:25.9436059Z * [new branch] gh/eellison/785/base -> origin/gh/eellison/785/base 2025-08-26T20:08:25.9436218Z * [new branch] gh/eellison/785/head -> origin/gh/eellison/785/head 2025-08-26T20:08:25.9436420Z * [new branch] gh/eellison/785/orig -> origin/gh/eellison/785/orig 2025-08-26T20:08:25.9436598Z * [new branch] gh/eellison/789/base -> origin/gh/eellison/789/base 2025-08-26T20:08:25.9436754Z * [new branch] gh/eellison/789/head -> origin/gh/eellison/789/head 2025-08-26T20:08:25.9436908Z * [new branch] gh/eellison/789/orig -> origin/gh/eellison/789/orig 2025-08-26T20:08:25.9437068Z * [new branch] gh/eellison/800/base -> origin/gh/eellison/800/base 2025-08-26T20:08:25.9437385Z * [new branch] gh/eellison/800/head -> origin/gh/eellison/800/head 2025-08-26T20:08:25.9437765Z * [new branch] gh/eellison/800/orig -> origin/gh/eellison/800/orig 2025-08-26T20:08:25.9439020Z * [new branch] gh/eellison/801/base -> origin/gh/eellison/801/base 2025-08-26T20:08:25.9439873Z * [new branch] gh/eellison/801/head -> origin/gh/eellison/801/head 2025-08-26T20:08:25.9440025Z * [new branch] gh/eellison/801/orig -> origin/gh/eellison/801/orig 2025-08-26T20:08:25.9444505Z * [new branch] gh/eellison/802/base -> origin/gh/eellison/802/base 2025-08-26T20:08:25.9444830Z * [new branch] gh/eellison/802/head -> origin/gh/eellison/802/head 2025-08-26T20:08:25.9445002Z * [new branch] gh/eellison/802/orig -> origin/gh/eellison/802/orig 2025-08-26T20:08:25.9445145Z * [new branch] gh/eellison/805/base -> origin/gh/eellison/805/base 2025-08-26T20:08:25.9445298Z * [new branch] gh/eellison/805/head -> origin/gh/eellison/805/head 2025-08-26T20:08:25.9445456Z * [new branch] gh/eellison/805/orig -> origin/gh/eellison/805/orig 2025-08-26T20:08:25.9445607Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-08-26T20:08:25.9446163Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-08-26T20:08:25.9446756Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-08-26T20:08:25.9447888Z * [new branch] gh/eellison/809/base -> origin/gh/eellison/809/base 2025-08-26T20:08:25.9448361Z * [new branch] gh/eellison/809/head -> origin/gh/eellison/809/head 2025-08-26T20:08:25.9449027Z * [new branch] gh/eellison/809/orig -> origin/gh/eellison/809/orig 2025-08-26T20:08:25.9450183Z * [new branch] gh/eellison/810/base -> origin/gh/eellison/810/base 2025-08-26T20:08:25.9450540Z * [new branch] gh/eellison/810/head -> origin/gh/eellison/810/head 2025-08-26T20:08:25.9451688Z * [new branch] gh/eellison/810/orig -> origin/gh/eellison/810/orig 2025-08-26T20:08:25.9452188Z * [new branch] gh/eellison/811/base -> origin/gh/eellison/811/base 2025-08-26T20:08:25.9452938Z * [new branch] gh/eellison/811/head -> origin/gh/eellison/811/head 2025-08-26T20:08:25.9453517Z * [new branch] gh/eellison/811/orig -> origin/gh/eellison/811/orig 2025-08-26T20:08:25.9454712Z * [new branch] gh/eellison/812/base -> origin/gh/eellison/812/base 2025-08-26T20:08:25.9459338Z * [new branch] gh/eellison/812/head -> origin/gh/eellison/812/head 2025-08-26T20:08:25.9459548Z * [new branch] gh/eellison/812/orig -> origin/gh/eellison/812/orig 2025-08-26T20:08:25.9459705Z * [new branch] gh/eellison/813/base -> origin/gh/eellison/813/base 2025-08-26T20:08:25.9459894Z * [new branch] gh/eellison/813/head -> origin/gh/eellison/813/head 2025-08-26T20:08:25.9460059Z * [new branch] gh/eellison/813/orig -> origin/gh/eellison/813/orig 2025-08-26T20:08:25.9460212Z * [new branch] gh/eellison/814/base -> origin/gh/eellison/814/base 2025-08-26T20:08:25.9460371Z * [new branch] gh/eellison/814/head -> origin/gh/eellison/814/head 2025-08-26T20:08:25.9460543Z * [new branch] gh/eellison/814/orig -> origin/gh/eellison/814/orig 2025-08-26T20:08:25.9462072Z * [new branch] gh/eellison/815/base -> origin/gh/eellison/815/base 2025-08-26T20:08:25.9462469Z * [new branch] gh/eellison/815/head -> origin/gh/eellison/815/head 2025-08-26T20:08:25.9463247Z * [new branch] gh/eellison/815/orig -> origin/gh/eellison/815/orig 2025-08-26T20:08:25.9464324Z * [new branch] gh/eellison/816/base -> origin/gh/eellison/816/base 2025-08-26T20:08:25.9464667Z * [new branch] gh/eellison/816/head -> origin/gh/eellison/816/head 2025-08-26T20:08:25.9466058Z * [new branch] gh/eellison/816/orig -> origin/gh/eellison/816/orig 2025-08-26T20:08:25.9468534Z * [new branch] gh/eellison/817/base -> origin/gh/eellison/817/base 2025-08-26T20:08:25.9468815Z * [new branch] gh/eellison/817/head -> origin/gh/eellison/817/head 2025-08-26T20:08:25.9469166Z * [new branch] gh/eellison/817/orig -> origin/gh/eellison/817/orig 2025-08-26T20:08:25.9469318Z * [new branch] gh/eellison/818/base -> origin/gh/eellison/818/base 2025-08-26T20:08:25.9470047Z * [new branch] gh/eellison/818/head -> origin/gh/eellison/818/head 2025-08-26T20:08:25.9470599Z * [new branch] gh/eellison/818/orig -> origin/gh/eellison/818/orig 2025-08-26T20:08:25.9472026Z * [new branch] gh/eellison/819/base -> origin/gh/eellison/819/base 2025-08-26T20:08:25.9472337Z * [new branch] gh/eellison/819/head -> origin/gh/eellison/819/head 2025-08-26T20:08:25.9473294Z * [new branch] gh/eellison/819/orig -> origin/gh/eellison/819/orig 2025-08-26T20:08:25.9474702Z * [new branch] gh/eellison/820/base -> origin/gh/eellison/820/base 2025-08-26T20:08:25.9475028Z * [new branch] gh/eellison/820/head -> origin/gh/eellison/820/head 2025-08-26T20:08:25.9476586Z * [new branch] gh/eellison/820/orig -> origin/gh/eellison/820/orig 2025-08-26T20:08:25.9476798Z * [new branch] gh/eellison/821/base -> origin/gh/eellison/821/base 2025-08-26T20:08:25.9477291Z * [new branch] gh/eellison/821/head -> origin/gh/eellison/821/head 2025-08-26T20:08:25.9478298Z * [new branch] gh/eellison/821/orig -> origin/gh/eellison/821/orig 2025-08-26T20:08:25.9479342Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-08-26T20:08:25.9479908Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-08-26T20:08:25.9480481Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-08-26T20:08:25.9481895Z * [new branch] gh/etaf/132/base -> origin/gh/etaf/132/base 2025-08-26T20:08:25.9482320Z * [new branch] gh/etaf/132/head -> origin/gh/etaf/132/head 2025-08-26T20:08:25.9483428Z * [new branch] gh/etaf/132/orig -> origin/gh/etaf/132/orig 2025-08-26T20:08:25.9484438Z * [new branch] gh/etaf/138/base -> origin/gh/etaf/138/base 2025-08-26T20:08:25.9485446Z * [new branch] gh/etaf/138/head -> origin/gh/etaf/138/head 2025-08-26T20:08:25.9485968Z * [new branch] gh/etaf/138/orig -> origin/gh/etaf/138/orig 2025-08-26T20:08:25.9487138Z * [new branch] gh/etaf/140/base -> origin/gh/etaf/140/base 2025-08-26T20:08:25.9487562Z * [new branch] gh/etaf/140/head -> origin/gh/etaf/140/head 2025-08-26T20:08:25.9488528Z * [new branch] gh/etaf/140/orig -> origin/gh/etaf/140/orig 2025-08-26T20:08:25.9489584Z * [new branch] gh/etaf/143/base -> origin/gh/etaf/143/base 2025-08-26T20:08:25.9489961Z * [new branch] gh/etaf/143/head -> origin/gh/etaf/143/head 2025-08-26T20:08:25.9490922Z * [new branch] gh/etaf/143/orig -> origin/gh/etaf/143/orig 2025-08-26T20:08:25.9491904Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-08-26T20:08:25.9492263Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-08-26T20:08:25.9493709Z * [new branch] gh/etaf/149/base -> origin/gh/etaf/149/base 2025-08-26T20:08:25.9493950Z * [new branch] gh/etaf/149/head -> origin/gh/etaf/149/head 2025-08-26T20:08:25.9495356Z * [new branch] gh/etaf/149/orig -> origin/gh/etaf/149/orig 2025-08-26T20:08:25.9495816Z * [new branch] gh/etaf/150/base -> origin/gh/etaf/150/base 2025-08-26T20:08:25.9497037Z * [new branch] gh/etaf/150/head -> origin/gh/etaf/150/head 2025-08-26T20:08:25.9499960Z * [new branch] gh/etaf/150/orig -> origin/gh/etaf/150/orig 2025-08-26T20:08:25.9505659Z * [new branch] gh/etaf/151/base -> origin/gh/etaf/151/base 2025-08-26T20:08:25.9506000Z * [new branch] gh/etaf/151/head -> origin/gh/etaf/151/head 2025-08-26T20:08:25.9506176Z * [new branch] gh/etaf/151/orig -> origin/gh/etaf/151/orig 2025-08-26T20:08:25.9506391Z * [new branch] gh/etaf/152/base -> origin/gh/etaf/152/base 2025-08-26T20:08:25.9506548Z * [new branch] gh/etaf/152/head -> origin/gh/etaf/152/head 2025-08-26T20:08:25.9506729Z * [new branch] gh/etaf/152/orig -> origin/gh/etaf/152/orig 2025-08-26T20:08:25.9507193Z * [new branch] gh/etaf/153/base -> origin/gh/etaf/153/base 2025-08-26T20:08:25.9508565Z * [new branch] gh/etaf/153/head -> origin/gh/etaf/153/head 2025-08-26T20:08:25.9508739Z * [new branch] gh/etaf/153/orig -> origin/gh/etaf/153/orig 2025-08-26T20:08:25.9511227Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-08-26T20:08:25.9511596Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-08-26T20:08:25.9511756Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-08-26T20:08:25.9512174Z * [new branch] gh/etaf/155/base -> origin/gh/etaf/155/base 2025-08-26T20:08:25.9513345Z * [new branch] gh/etaf/155/head -> origin/gh/etaf/155/head 2025-08-26T20:08:25.9514053Z * [new branch] gh/etaf/155/orig -> origin/gh/etaf/155/orig 2025-08-26T20:08:25.9515299Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-08-26T20:08:25.9515592Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-08-26T20:08:25.9516177Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-08-26T20:08:25.9517515Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-08-26T20:08:25.9517720Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-08-26T20:08:25.9518795Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-08-26T20:08:25.9523446Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-08-26T20:08:25.9523767Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-08-26T20:08:25.9524164Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-08-26T20:08:25.9524302Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-08-26T20:08:25.9524436Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-08-26T20:08:25.9524660Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-08-26T20:08:25.9530844Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-08-26T20:08:25.9535813Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-08-26T20:08:25.9541080Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-08-26T20:08:25.9543189Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-08-26T20:08:25.9543481Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-08-26T20:08:25.9549376Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-08-26T20:08:25.9555617Z * [new branch] gh/etaf/162/base -> origin/gh/etaf/162/base 2025-08-26T20:08:25.9555972Z * [new branch] gh/etaf/162/head -> origin/gh/etaf/162/head 2025-08-26T20:08:25.9556188Z * [new branch] gh/etaf/162/orig -> origin/gh/etaf/162/orig 2025-08-26T20:08:25.9556629Z * [new branch] gh/etaf/163/base -> origin/gh/etaf/163/base 2025-08-26T20:08:25.9556925Z * [new branch] gh/etaf/163/head -> origin/gh/etaf/163/head 2025-08-26T20:08:25.9557461Z * [new branch] gh/etaf/163/orig -> origin/gh/etaf/163/orig 2025-08-26T20:08:25.9557633Z * [new branch] gh/etaf/164/base -> origin/gh/etaf/164/base 2025-08-26T20:08:25.9557781Z * [new branch] gh/etaf/164/head -> origin/gh/etaf/164/head 2025-08-26T20:08:25.9557939Z * [new branch] gh/etaf/164/orig -> origin/gh/etaf/164/orig 2025-08-26T20:08:25.9558078Z * [new branch] gh/etaf/165/base -> origin/gh/etaf/165/base 2025-08-26T20:08:25.9558223Z * [new branch] gh/etaf/165/head -> origin/gh/etaf/165/head 2025-08-26T20:08:25.9558357Z * [new branch] gh/etaf/165/orig -> origin/gh/etaf/165/orig 2025-08-26T20:08:25.9558556Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-08-26T20:08:25.9558709Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-08-26T20:08:25.9558864Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-08-26T20:08:25.9559007Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-08-26T20:08:25.9559381Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-08-26T20:08:25.9559772Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-08-26T20:08:25.9559925Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-08-26T20:08:25.9560079Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-08-26T20:08:25.9560225Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-08-26T20:08:25.9560371Z * [new branch] gh/ezyang/3068/base -> origin/gh/ezyang/3068/base 2025-08-26T20:08:25.9560526Z * [new branch] gh/ezyang/3068/head -> origin/gh/ezyang/3068/head 2025-08-26T20:08:25.9560668Z * [new branch] gh/ezyang/3068/orig -> origin/gh/ezyang/3068/orig 2025-08-26T20:08:25.9560819Z * [new branch] gh/ezyang/3071/base -> origin/gh/ezyang/3071/base 2025-08-26T20:08:25.9560959Z * [new branch] gh/ezyang/3071/head -> origin/gh/ezyang/3071/head 2025-08-26T20:08:25.9561143Z * [new branch] gh/ezyang/3071/orig -> origin/gh/ezyang/3071/orig 2025-08-26T20:08:25.9561285Z * [new branch] gh/ezyang/3074/base -> origin/gh/ezyang/3074/base 2025-08-26T20:08:25.9561434Z * [new branch] gh/ezyang/3074/head -> origin/gh/ezyang/3074/head 2025-08-26T20:08:25.9561577Z * [new branch] gh/ezyang/3074/orig -> origin/gh/ezyang/3074/orig 2025-08-26T20:08:25.9561718Z * [new branch] gh/ezyang/3088/base -> origin/gh/ezyang/3088/base 2025-08-26T20:08:25.9561864Z * [new branch] gh/ezyang/3088/head -> origin/gh/ezyang/3088/head 2025-08-26T20:08:25.9562005Z * [new branch] gh/ezyang/3088/orig -> origin/gh/ezyang/3088/orig 2025-08-26T20:08:25.9562150Z * [new branch] gh/ezyang/3092/base -> origin/gh/ezyang/3092/base 2025-08-26T20:08:25.9562287Z * [new branch] gh/ezyang/3092/head -> origin/gh/ezyang/3092/head 2025-08-26T20:08:25.9562427Z * [new branch] gh/ezyang/3092/orig -> origin/gh/ezyang/3092/orig 2025-08-26T20:08:25.9562570Z * [new branch] gh/ezyang/3103/base -> origin/gh/ezyang/3103/base 2025-08-26T20:08:25.9562706Z * [new branch] gh/ezyang/3103/head -> origin/gh/ezyang/3103/head 2025-08-26T20:08:25.9563161Z * [new branch] gh/ezyang/3103/orig -> origin/gh/ezyang/3103/orig 2025-08-26T20:08:25.9563324Z * [new branch] gh/ezyang/3105/base -> origin/gh/ezyang/3105/base 2025-08-26T20:08:25.9563474Z * [new branch] gh/ezyang/3105/head -> origin/gh/ezyang/3105/head 2025-08-26T20:08:25.9563815Z * [new branch] gh/ezyang/3105/orig -> origin/gh/ezyang/3105/orig 2025-08-26T20:08:25.9568696Z * [new branch] gh/ezyang/3114/base -> origin/gh/ezyang/3114/base 2025-08-26T20:08:25.9569016Z * [new branch] gh/ezyang/3114/head -> origin/gh/ezyang/3114/head 2025-08-26T20:08:25.9576832Z * [new branch] gh/ezyang/3114/orig -> origin/gh/ezyang/3114/orig 2025-08-26T20:08:25.9580188Z * [new branch] gh/ezyang/3116/base -> origin/gh/ezyang/3116/base 2025-08-26T20:08:25.9580608Z * [new branch] gh/ezyang/3116/head -> origin/gh/ezyang/3116/head 2025-08-26T20:08:25.9580768Z * [new branch] gh/ezyang/3116/orig -> origin/gh/ezyang/3116/orig 2025-08-26T20:08:25.9581057Z * [new branch] gh/ezyang/3117/base -> origin/gh/ezyang/3117/base 2025-08-26T20:08:25.9581562Z * [new branch] gh/ezyang/3117/head -> origin/gh/ezyang/3117/head 2025-08-26T20:08:25.9581748Z * [new branch] gh/ezyang/3117/orig -> origin/gh/ezyang/3117/orig 2025-08-26T20:08:25.9581896Z * [new branch] gh/ezyang/3118/base -> origin/gh/ezyang/3118/base 2025-08-26T20:08:25.9582249Z * [new branch] gh/ezyang/3118/head -> origin/gh/ezyang/3118/head 2025-08-26T20:08:25.9582438Z * [new branch] gh/ezyang/3118/orig -> origin/gh/ezyang/3118/orig 2025-08-26T20:08:25.9582598Z * [new branch] gh/ezyang/3119/base -> origin/gh/ezyang/3119/base 2025-08-26T20:08:25.9582746Z * [new branch] gh/ezyang/3119/head -> origin/gh/ezyang/3119/head 2025-08-26T20:08:25.9582899Z * [new branch] gh/ezyang/3119/orig -> origin/gh/ezyang/3119/orig 2025-08-26T20:08:25.9583047Z * [new branch] gh/ezyang/3120/base -> origin/gh/ezyang/3120/base 2025-08-26T20:08:25.9583195Z * [new branch] gh/ezyang/3120/head -> origin/gh/ezyang/3120/head 2025-08-26T20:08:25.9583622Z * [new branch] gh/ezyang/3120/orig -> origin/gh/ezyang/3120/orig 2025-08-26T20:08:25.9584146Z * [new branch] gh/ezyang/3121/base -> origin/gh/ezyang/3121/base 2025-08-26T20:08:25.9584349Z * [new branch] gh/ezyang/3121/head -> origin/gh/ezyang/3121/head 2025-08-26T20:08:25.9584514Z * [new branch] gh/ezyang/3121/orig -> origin/gh/ezyang/3121/orig 2025-08-26T20:08:25.9584686Z * [new branch] gh/ezyang/3122/base -> origin/gh/ezyang/3122/base 2025-08-26T20:08:25.9584836Z * [new branch] gh/ezyang/3122/head -> origin/gh/ezyang/3122/head 2025-08-26T20:08:25.9587745Z * [new branch] gh/ezyang/3122/orig -> origin/gh/ezyang/3122/orig 2025-08-26T20:08:25.9587923Z * [new branch] gh/ezyang/3123/base -> origin/gh/ezyang/3123/base 2025-08-26T20:08:25.9588526Z * [new branch] gh/ezyang/3123/head -> origin/gh/ezyang/3123/head 2025-08-26T20:08:25.9588710Z * [new branch] gh/ezyang/3123/orig -> origin/gh/ezyang/3123/orig 2025-08-26T20:08:25.9588858Z * [new branch] gh/ezyang/3124/base -> origin/gh/ezyang/3124/base 2025-08-26T20:08:25.9589026Z * [new branch] gh/ezyang/3124/head -> origin/gh/ezyang/3124/head 2025-08-26T20:08:25.9593919Z * [new branch] gh/ezyang/3124/orig -> origin/gh/ezyang/3124/orig 2025-08-26T20:08:25.9594115Z * [new branch] gh/ezyang/3125/base -> origin/gh/ezyang/3125/base 2025-08-26T20:08:25.9594267Z * [new branch] gh/ezyang/3125/head -> origin/gh/ezyang/3125/head 2025-08-26T20:08:25.9594572Z * [new branch] gh/ezyang/3125/orig -> origin/gh/ezyang/3125/orig 2025-08-26T20:08:25.9594718Z * [new branch] gh/ezyang/3126/base -> origin/gh/ezyang/3126/base 2025-08-26T20:08:25.9594880Z * [new branch] gh/ezyang/3126/head -> origin/gh/ezyang/3126/head 2025-08-26T20:08:25.9595032Z * [new branch] gh/ezyang/3126/orig -> origin/gh/ezyang/3126/orig 2025-08-26T20:08:25.9595185Z * [new branch] gh/ezyang/3127/base -> origin/gh/ezyang/3127/base 2025-08-26T20:08:25.9595347Z * [new branch] gh/ezyang/3127/head -> origin/gh/ezyang/3127/head 2025-08-26T20:08:25.9595495Z * [new branch] gh/ezyang/3127/orig -> origin/gh/ezyang/3127/orig 2025-08-26T20:08:25.9596042Z * [new branch] gh/ezyang/3128/base -> origin/gh/ezyang/3128/base 2025-08-26T20:08:25.9597035Z * [new branch] gh/ezyang/3128/head -> origin/gh/ezyang/3128/head 2025-08-26T20:08:25.9597667Z * [new branch] gh/ezyang/3128/orig -> origin/gh/ezyang/3128/orig 2025-08-26T20:08:25.9599333Z * [new branch] gh/ezyang/3129/base -> origin/gh/ezyang/3129/base 2025-08-26T20:08:25.9599555Z * [new branch] gh/ezyang/3129/head -> origin/gh/ezyang/3129/head 2025-08-26T20:08:25.9600023Z * [new branch] gh/ezyang/3129/orig -> origin/gh/ezyang/3129/orig 2025-08-26T20:08:25.9603944Z * [new branch] gh/ezyang/3130/base -> origin/gh/ezyang/3130/base 2025-08-26T20:08:25.9604363Z * [new branch] gh/ezyang/3130/head -> origin/gh/ezyang/3130/head 2025-08-26T20:08:25.9604526Z * [new branch] gh/ezyang/3130/orig -> origin/gh/ezyang/3130/orig 2025-08-26T20:08:25.9609059Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-08-26T20:08:25.9609276Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-08-26T20:08:25.9609422Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-08-26T20:08:25.9609561Z * [new branch] gh/ezyang/3132/base -> origin/gh/ezyang/3132/base 2025-08-26T20:08:25.9609712Z * [new branch] gh/ezyang/3132/head -> origin/gh/ezyang/3132/head 2025-08-26T20:08:25.9609854Z * [new branch] gh/ezyang/3132/orig -> origin/gh/ezyang/3132/orig 2025-08-26T20:08:25.9610007Z * [new branch] gh/ezyang/3133/base -> origin/gh/ezyang/3133/base 2025-08-26T20:08:25.9614669Z * [new branch] gh/ezyang/3133/head -> origin/gh/ezyang/3133/head 2025-08-26T20:08:25.9615205Z * [new branch] gh/ezyang/3133/orig -> origin/gh/ezyang/3133/orig 2025-08-26T20:08:25.9615379Z * [new branch] gh/ezyang/3134/base -> origin/gh/ezyang/3134/base 2025-08-26T20:08:25.9615540Z * [new branch] gh/ezyang/3134/head -> origin/gh/ezyang/3134/head 2025-08-26T20:08:25.9615710Z * [new branch] gh/ezyang/3134/orig -> origin/gh/ezyang/3134/orig 2025-08-26T20:08:25.9615858Z * [new branch] gh/ezyang/3135/base -> origin/gh/ezyang/3135/base 2025-08-26T20:08:25.9616007Z * [new branch] gh/ezyang/3135/head -> origin/gh/ezyang/3135/head 2025-08-26T20:08:25.9616151Z * [new branch] gh/ezyang/3135/orig -> origin/gh/ezyang/3135/orig 2025-08-26T20:08:25.9616307Z * [new branch] gh/ezyang/3136/base -> origin/gh/ezyang/3136/base 2025-08-26T20:08:25.9616447Z * [new branch] gh/ezyang/3136/head -> origin/gh/ezyang/3136/head 2025-08-26T20:08:25.9616693Z * [new branch] gh/ezyang/3136/orig -> origin/gh/ezyang/3136/orig 2025-08-26T20:08:25.9618032Z * [new branch] gh/ezyang/3137/base -> origin/gh/ezyang/3137/base 2025-08-26T20:08:25.9618597Z * [new branch] gh/ezyang/3137/head -> origin/gh/ezyang/3137/head 2025-08-26T20:08:25.9618993Z * [new branch] gh/ezyang/3137/orig -> origin/gh/ezyang/3137/orig 2025-08-26T20:08:25.9619457Z * [new branch] gh/fadara01/1/base -> origin/gh/fadara01/1/base 2025-08-26T20:08:25.9623026Z * [new branch] gh/fadara01/1/head -> origin/gh/fadara01/1/head 2025-08-26T20:08:25.9623749Z * [new branch] gh/fadara01/1/orig -> origin/gh/fadara01/1/orig 2025-08-26T20:08:25.9623954Z * [new branch] gh/fduwjj/169/base -> origin/gh/fduwjj/169/base 2025-08-26T20:08:25.9624108Z * [new branch] gh/fduwjj/169/head -> origin/gh/fduwjj/169/head 2025-08-26T20:08:25.9624249Z * [new branch] gh/fduwjj/169/orig -> origin/gh/fduwjj/169/orig 2025-08-26T20:08:25.9624600Z * [new branch] gh/fduwjj/171/base -> origin/gh/fduwjj/171/base 2025-08-26T20:08:25.9628279Z * [new branch] gh/fduwjj/171/head -> origin/gh/fduwjj/171/head 2025-08-26T20:08:25.9628470Z * [new branch] gh/fduwjj/171/orig -> origin/gh/fduwjj/171/orig 2025-08-26T20:08:25.9628620Z * [new branch] gh/fduwjj/175/base -> origin/gh/fduwjj/175/base 2025-08-26T20:08:25.9628764Z * [new branch] gh/fduwjj/175/head -> origin/gh/fduwjj/175/head 2025-08-26T20:08:25.9629462Z * [new branch] gh/fduwjj/175/orig -> origin/gh/fduwjj/175/orig 2025-08-26T20:08:25.9630007Z * [new branch] gh/fduwjj/176/base -> origin/gh/fduwjj/176/base 2025-08-26T20:08:25.9630192Z * [new branch] gh/fduwjj/176/head -> origin/gh/fduwjj/176/head 2025-08-26T20:08:25.9631638Z * [new branch] gh/fduwjj/176/orig -> origin/gh/fduwjj/176/orig 2025-08-26T20:08:25.9632152Z * [new branch] gh/fduwjj/177/base -> origin/gh/fduwjj/177/base 2025-08-26T20:08:25.9636646Z * [new branch] gh/fduwjj/177/head -> origin/gh/fduwjj/177/head 2025-08-26T20:08:25.9637022Z * [new branch] gh/fduwjj/177/orig -> origin/gh/fduwjj/177/orig 2025-08-26T20:08:25.9637199Z * [new branch] gh/fduwjj/178/base -> origin/gh/fduwjj/178/base 2025-08-26T20:08:25.9637350Z * [new branch] gh/fduwjj/178/head -> origin/gh/fduwjj/178/head 2025-08-26T20:08:25.9637510Z * [new branch] gh/fduwjj/178/orig -> origin/gh/fduwjj/178/orig 2025-08-26T20:08:25.9637816Z * [new branch] gh/fduwjj/179/base -> origin/gh/fduwjj/179/base 2025-08-26T20:08:25.9637991Z * [new branch] gh/fduwjj/179/head -> origin/gh/fduwjj/179/head 2025-08-26T20:08:25.9638441Z * [new branch] gh/fduwjj/179/orig -> origin/gh/fduwjj/179/orig 2025-08-26T20:08:25.9640545Z * [new branch] gh/fduwjj/180/base -> origin/gh/fduwjj/180/base 2025-08-26T20:08:25.9640861Z * [new branch] gh/fduwjj/180/head -> origin/gh/fduwjj/180/head 2025-08-26T20:08:25.9646030Z * [new branch] gh/fduwjj/180/orig -> origin/gh/fduwjj/180/orig 2025-08-26T20:08:25.9648157Z * [new branch] gh/fduwjj/181/base -> origin/gh/fduwjj/181/base 2025-08-26T20:08:25.9653043Z * [new branch] gh/fduwjj/181/head -> origin/gh/fduwjj/181/head 2025-08-26T20:08:25.9653232Z * [new branch] gh/fduwjj/181/orig -> origin/gh/fduwjj/181/orig 2025-08-26T20:08:25.9653399Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-08-26T20:08:25.9653559Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-08-26T20:08:25.9653703Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-08-26T20:08:25.9653853Z * [new branch] gh/fduwjj/183/base -> origin/gh/fduwjj/183/base 2025-08-26T20:08:25.9654135Z * [new branch] gh/fduwjj/183/head -> origin/gh/fduwjj/183/head 2025-08-26T20:08:25.9654279Z * [new branch] gh/fduwjj/183/orig -> origin/gh/fduwjj/183/orig 2025-08-26T20:08:25.9654428Z * [new branch] gh/fduwjj/184/base -> origin/gh/fduwjj/184/base 2025-08-26T20:08:25.9654567Z * [new branch] gh/fduwjj/184/head -> origin/gh/fduwjj/184/head 2025-08-26T20:08:25.9654712Z * [new branch] gh/fduwjj/184/orig -> origin/gh/fduwjj/184/orig 2025-08-26T20:08:25.9654857Z * [new branch] gh/fduwjj/185/base -> origin/gh/fduwjj/185/base 2025-08-26T20:08:25.9655005Z * [new branch] gh/fduwjj/185/head -> origin/gh/fduwjj/185/head 2025-08-26T20:08:25.9655145Z * [new branch] gh/fduwjj/185/orig -> origin/gh/fduwjj/185/orig 2025-08-26T20:08:25.9657128Z * [new branch] gh/fduwjj/186/base -> origin/gh/fduwjj/186/base 2025-08-26T20:08:25.9657562Z * [new branch] gh/fduwjj/186/head -> origin/gh/fduwjj/186/head 2025-08-26T20:08:25.9657734Z * [new branch] gh/fduwjj/186/orig -> origin/gh/fduwjj/186/orig 2025-08-26T20:08:25.9657884Z * [new branch] gh/fduwjj/187/base -> origin/gh/fduwjj/187/base 2025-08-26T20:08:25.9658028Z * [new branch] gh/fduwjj/187/head -> origin/gh/fduwjj/187/head 2025-08-26T20:08:25.9658164Z * [new branch] gh/fduwjj/187/orig -> origin/gh/fduwjj/187/orig 2025-08-26T20:08:25.9661735Z * [new branch] gh/fduwjj/188/base -> origin/gh/fduwjj/188/base 2025-08-26T20:08:25.9661983Z * [new branch] gh/fduwjj/188/head -> origin/gh/fduwjj/188/head 2025-08-26T20:08:25.9666801Z * [new branch] gh/fduwjj/188/orig -> origin/gh/fduwjj/188/orig 2025-08-26T20:08:25.9670959Z * [new branch] gh/fduwjj/189/base -> origin/gh/fduwjj/189/base 2025-08-26T20:08:25.9671152Z * [new branch] gh/fduwjj/189/head -> origin/gh/fduwjj/189/head 2025-08-26T20:08:25.9671313Z * [new branch] gh/fduwjj/189/orig -> origin/gh/fduwjj/189/orig 2025-08-26T20:08:25.9671454Z * [new branch] gh/fduwjj/190/base -> origin/gh/fduwjj/190/base 2025-08-26T20:08:25.9671606Z * [new branch] gh/fduwjj/190/head -> origin/gh/fduwjj/190/head 2025-08-26T20:08:25.9671749Z * [new branch] gh/fduwjj/190/orig -> origin/gh/fduwjj/190/orig 2025-08-26T20:08:25.9671914Z * [new branch] gh/fduwjj/191/base -> origin/gh/fduwjj/191/base 2025-08-26T20:08:25.9672055Z * [new branch] gh/fduwjj/191/head -> origin/gh/fduwjj/191/head 2025-08-26T20:08:25.9672200Z * [new branch] gh/fduwjj/191/orig -> origin/gh/fduwjj/191/orig 2025-08-26T20:08:25.9672351Z * [new branch] gh/fegin/306/base -> origin/gh/fegin/306/base 2025-08-26T20:08:25.9672489Z * [new branch] gh/fegin/306/head -> origin/gh/fegin/306/head 2025-08-26T20:08:25.9672629Z * [new branch] gh/fegin/306/orig -> origin/gh/fegin/306/orig 2025-08-26T20:08:25.9672761Z * [new branch] gh/fegin/307/base -> origin/gh/fegin/307/base 2025-08-26T20:08:25.9672901Z * [new branch] gh/fegin/307/head -> origin/gh/fegin/307/head 2025-08-26T20:08:25.9673040Z * [new branch] gh/fegin/307/orig -> origin/gh/fegin/307/orig 2025-08-26T20:08:25.9673192Z * [new branch] gh/fffrog/124/base -> origin/gh/fffrog/124/base 2025-08-26T20:08:25.9674417Z * [new branch] gh/fffrog/124/head -> origin/gh/fffrog/124/head 2025-08-26T20:08:25.9674596Z * [new branch] gh/fffrog/124/orig -> origin/gh/fffrog/124/orig 2025-08-26T20:08:25.9675830Z * [new branch] gh/fffrog/128/base -> origin/gh/fffrog/128/base 2025-08-26T20:08:25.9676293Z * [new branch] gh/fffrog/128/head -> origin/gh/fffrog/128/head 2025-08-26T20:08:25.9677248Z * [new branch] gh/fffrog/128/orig -> origin/gh/fffrog/128/orig 2025-08-26T20:08:25.9678170Z * [new branch] gh/fffrog/129/base -> origin/gh/fffrog/129/base 2025-08-26T20:08:25.9678567Z * [new branch] gh/fffrog/129/head -> origin/gh/fffrog/129/head 2025-08-26T20:08:25.9679775Z * [new branch] gh/fffrog/129/orig -> origin/gh/fffrog/129/orig 2025-08-26T20:08:25.9682762Z * [new branch] gh/fffrog/130/base -> origin/gh/fffrog/130/base 2025-08-26T20:08:25.9682943Z * [new branch] gh/fffrog/130/head -> origin/gh/fffrog/130/head 2025-08-26T20:08:25.9683245Z * [new branch] gh/fffrog/130/orig -> origin/gh/fffrog/130/orig 2025-08-26T20:08:25.9683424Z * [new branch] gh/fffrog/131/base -> origin/gh/fffrog/131/base 2025-08-26T20:08:25.9683579Z * [new branch] gh/fffrog/131/head -> origin/gh/fffrog/131/head 2025-08-26T20:08:25.9686339Z * [new branch] gh/fffrog/131/orig -> origin/gh/fffrog/131/orig 2025-08-26T20:08:25.9686525Z * [new branch] gh/fffrog/132/base -> origin/gh/fffrog/132/base 2025-08-26T20:08:25.9686697Z * [new branch] gh/fffrog/132/head -> origin/gh/fffrog/132/head 2025-08-26T20:08:25.9690073Z * [new branch] gh/fffrog/132/orig -> origin/gh/fffrog/132/orig 2025-08-26T20:08:25.9690225Z * [new branch] gh/fffrog/133/base -> origin/gh/fffrog/133/base 2025-08-26T20:08:25.9690368Z * [new branch] gh/fffrog/133/head -> origin/gh/fffrog/133/head 2025-08-26T20:08:25.9690503Z * [new branch] gh/fffrog/133/orig -> origin/gh/fffrog/133/orig 2025-08-26T20:08:25.9690653Z * [new branch] gh/fffrog/134/base -> origin/gh/fffrog/134/base 2025-08-26T20:08:25.9697543Z * [new branch] gh/fffrog/134/head -> origin/gh/fffrog/134/head 2025-08-26T20:08:25.9702526Z * [new branch] gh/fffrog/134/orig -> origin/gh/fffrog/134/orig 2025-08-26T20:08:25.9707540Z * [new branch] gh/fffrog/135/base -> origin/gh/fffrog/135/base 2025-08-26T20:08:25.9711927Z * [new branch] gh/fffrog/135/head -> origin/gh/fffrog/135/head 2025-08-26T20:08:25.9714120Z * [new branch] gh/fffrog/135/orig -> origin/gh/fffrog/135/orig 2025-08-26T20:08:25.9714295Z * [new branch] gh/fffrog/136/base -> origin/gh/fffrog/136/base 2025-08-26T20:08:25.9714551Z * [new branch] gh/fffrog/136/head -> origin/gh/fffrog/136/head 2025-08-26T20:08:25.9715156Z * [new branch] gh/fffrog/136/orig -> origin/gh/fffrog/136/orig 2025-08-26T20:08:25.9715521Z * [new branch] gh/fffrog/137/base -> origin/gh/fffrog/137/base 2025-08-26T20:08:25.9715899Z * [new branch] gh/fffrog/137/head -> origin/gh/fffrog/137/head 2025-08-26T20:08:25.9716117Z * [new branch] gh/fffrog/137/orig -> origin/gh/fffrog/137/orig 2025-08-26T20:08:25.9716335Z * [new branch] gh/fffrog/138/base -> origin/gh/fffrog/138/base 2025-08-26T20:08:25.9716542Z * [new branch] gh/fffrog/138/head -> origin/gh/fffrog/138/head 2025-08-26T20:08:25.9716741Z * [new branch] gh/fffrog/138/orig -> origin/gh/fffrog/138/orig 2025-08-26T20:08:25.9716907Z * [new branch] gh/fffrog/139/base -> origin/gh/fffrog/139/base 2025-08-26T20:08:25.9717051Z * [new branch] gh/fffrog/139/head -> origin/gh/fffrog/139/head 2025-08-26T20:08:25.9717207Z * [new branch] gh/fffrog/139/orig -> origin/gh/fffrog/139/orig 2025-08-26T20:08:25.9717571Z * [new branch] gh/fffrog/140/base -> origin/gh/fffrog/140/base 2025-08-26T20:08:25.9717721Z * [new branch] gh/fffrog/140/head -> origin/gh/fffrog/140/head 2025-08-26T20:08:25.9717872Z * [new branch] gh/fffrog/140/orig -> origin/gh/fffrog/140/orig 2025-08-26T20:08:25.9718021Z * [new branch] gh/fffrog/141/base -> origin/gh/fffrog/141/base 2025-08-26T20:08:25.9718169Z * [new branch] gh/fffrog/141/head -> origin/gh/fffrog/141/head 2025-08-26T20:08:25.9718324Z * [new branch] gh/fffrog/141/orig -> origin/gh/fffrog/141/orig 2025-08-26T20:08:25.9718487Z * [new branch] gh/fffrog/142/base -> origin/gh/fffrog/142/base 2025-08-26T20:08:25.9718632Z * [new branch] gh/fffrog/142/head -> origin/gh/fffrog/142/head 2025-08-26T20:08:25.9718782Z * [new branch] gh/fffrog/142/orig -> origin/gh/fffrog/142/orig 2025-08-26T20:08:25.9718924Z * [new branch] gh/fffrog/143/base -> origin/gh/fffrog/143/base 2025-08-26T20:08:25.9719071Z * [new branch] gh/fffrog/143/head -> origin/gh/fffrog/143/head 2025-08-26T20:08:25.9719582Z * [new branch] gh/fffrog/143/orig -> origin/gh/fffrog/143/orig 2025-08-26T20:08:25.9719767Z * [new branch] gh/fffrog/144/base -> origin/gh/fffrog/144/base 2025-08-26T20:08:25.9719976Z * [new branch] gh/fffrog/144/head -> origin/gh/fffrog/144/head 2025-08-26T20:08:25.9720275Z * [new branch] gh/fffrog/144/orig -> origin/gh/fffrog/144/orig 2025-08-26T20:08:25.9720833Z * [new branch] gh/gmagogsfm/1/base -> origin/gh/gmagogsfm/1/base 2025-08-26T20:08:25.9721376Z * [new branch] gh/gmagogsfm/1/head -> origin/gh/gmagogsfm/1/head 2025-08-26T20:08:25.9721878Z * [new branch] gh/gmagogsfm/1/orig -> origin/gh/gmagogsfm/1/orig 2025-08-26T20:08:25.9722387Z * [new branch] gh/gmagogsfm/2/base -> origin/gh/gmagogsfm/2/base 2025-08-26T20:08:25.9722981Z * [new branch] gh/gmagogsfm/2/head -> origin/gh/gmagogsfm/2/head 2025-08-26T20:08:25.9723496Z * [new branch] gh/gmagogsfm/2/orig -> origin/gh/gmagogsfm/2/orig 2025-08-26T20:08:25.9724004Z * [new branch] gh/gmagogsfm/3/base -> origin/gh/gmagogsfm/3/base 2025-08-26T20:08:25.9724547Z * [new branch] gh/gmagogsfm/3/head -> origin/gh/gmagogsfm/3/head 2025-08-26T20:08:25.9725040Z * [new branch] gh/gmagogsfm/3/orig -> origin/gh/gmagogsfm/3/orig 2025-08-26T20:08:25.9726092Z * [new branch] gh/guangyey/130/base -> origin/gh/guangyey/130/base 2025-08-26T20:08:25.9726706Z * [new branch] gh/guangyey/130/head -> origin/gh/guangyey/130/head 2025-08-26T20:08:25.9727278Z * [new branch] gh/guangyey/130/orig -> origin/gh/guangyey/130/orig 2025-08-26T20:08:25.9727867Z * [new branch] gh/guangyey/133/base -> origin/gh/guangyey/133/base 2025-08-26T20:08:25.9728429Z * [new branch] gh/guangyey/133/head -> origin/gh/guangyey/133/head 2025-08-26T20:08:25.9729026Z * [new branch] gh/guangyey/133/orig -> origin/gh/guangyey/133/orig 2025-08-26T20:08:25.9729630Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-08-26T20:08:25.9730245Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-08-26T20:08:25.9730868Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-08-26T20:08:25.9731453Z * [new branch] gh/guangyey/135/base -> origin/gh/guangyey/135/base 2025-08-26T20:08:25.9732036Z * [new branch] gh/guangyey/135/head -> origin/gh/guangyey/135/head 2025-08-26T20:08:25.9732712Z * [new branch] gh/guangyey/135/orig -> origin/gh/guangyey/135/orig 2025-08-26T20:08:25.9733284Z * [new branch] gh/guangyey/139/base -> origin/gh/guangyey/139/base 2025-08-26T20:08:25.9733801Z * [new branch] gh/guangyey/139/head -> origin/gh/guangyey/139/head 2025-08-26T20:08:25.9734658Z * [new branch] gh/guangyey/139/orig -> origin/gh/guangyey/139/orig 2025-08-26T20:08:25.9735189Z * [new branch] gh/guangyey/140/base -> origin/gh/guangyey/140/base 2025-08-26T20:08:25.9735658Z * [new branch] gh/guangyey/140/head -> origin/gh/guangyey/140/head 2025-08-26T20:08:25.9736193Z * [new branch] gh/guangyey/140/orig -> origin/gh/guangyey/140/orig 2025-08-26T20:08:25.9736637Z * [new branch] gh/guangyey/142/base -> origin/gh/guangyey/142/base 2025-08-26T20:08:25.9736985Z * [new branch] gh/guangyey/142/head -> origin/gh/guangyey/142/head 2025-08-26T20:08:25.9737336Z * [new branch] gh/guangyey/142/orig -> origin/gh/guangyey/142/orig 2025-08-26T20:08:25.9740301Z * [new branch] gh/guangyey/145/base -> origin/gh/guangyey/145/base 2025-08-26T20:08:25.9740755Z * [new branch] gh/guangyey/145/head -> origin/gh/guangyey/145/head 2025-08-26T20:08:25.9741121Z * [new branch] gh/guangyey/145/orig -> origin/gh/guangyey/145/orig 2025-08-26T20:08:25.9741482Z * [new branch] gh/guangyey/153/base -> origin/gh/guangyey/153/base 2025-08-26T20:08:25.9741990Z * [new branch] gh/guangyey/153/head -> origin/gh/guangyey/153/head 2025-08-26T20:08:25.9742360Z * [new branch] gh/guangyey/153/orig -> origin/gh/guangyey/153/orig 2025-08-26T20:08:25.9747042Z * [new branch] gh/guangyey/158/base -> origin/gh/guangyey/158/base 2025-08-26T20:08:25.9747512Z * [new branch] gh/guangyey/158/head -> origin/gh/guangyey/158/head 2025-08-26T20:08:25.9747926Z * [new branch] gh/guangyey/158/orig -> origin/gh/guangyey/158/orig 2025-08-26T20:08:25.9748340Z * [new branch] gh/guangyey/159/base -> origin/gh/guangyey/159/base 2025-08-26T20:08:25.9748714Z * [new branch] gh/guangyey/159/head -> origin/gh/guangyey/159/head 2025-08-26T20:08:25.9749082Z * [new branch] gh/guangyey/159/orig -> origin/gh/guangyey/159/orig 2025-08-26T20:08:25.9749454Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-08-26T20:08:25.9749860Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-08-26T20:08:25.9750236Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-08-26T20:08:25.9750602Z * [new branch] gh/guangyey/165/base -> origin/gh/guangyey/165/base 2025-08-26T20:08:25.9750967Z * [new branch] gh/guangyey/165/head -> origin/gh/guangyey/165/head 2025-08-26T20:08:25.9751337Z * [new branch] gh/guangyey/165/orig -> origin/gh/guangyey/165/orig 2025-08-26T20:08:25.9751697Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-08-26T20:08:25.9753568Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-08-26T20:08:25.9753961Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-08-26T20:08:25.9754336Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-08-26T20:08:25.9754726Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-08-26T20:08:25.9755080Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-08-26T20:08:25.9755442Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-08-26T20:08:25.9757720Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-08-26T20:08:25.9758110Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-08-26T20:08:25.9758481Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-08-26T20:08:25.9758848Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-08-26T20:08:25.9759332Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-08-26T20:08:25.9759714Z * [new branch] gh/guangyey/173/base -> origin/gh/guangyey/173/base 2025-08-26T20:08:25.9760179Z * [new branch] gh/guangyey/173/head -> origin/gh/guangyey/173/head 2025-08-26T20:08:25.9760583Z * [new branch] gh/guangyey/173/orig -> origin/gh/guangyey/173/orig 2025-08-26T20:08:25.9761079Z * [new branch] gh/guangyey/174/base -> origin/gh/guangyey/174/base 2025-08-26T20:08:25.9767448Z * [new branch] gh/guangyey/174/head -> origin/gh/guangyey/174/head 2025-08-26T20:08:25.9768052Z * [new branch] gh/guangyey/174/orig -> origin/gh/guangyey/174/orig 2025-08-26T20:08:25.9772624Z * [new branch] gh/guangyey/175/base -> origin/gh/guangyey/175/base 2025-08-26T20:08:25.9778477Z * [new branch] gh/guangyey/175/head -> origin/gh/guangyey/175/head 2025-08-26T20:08:25.9782981Z * [new branch] gh/guangyey/175/orig -> origin/gh/guangyey/175/orig 2025-08-26T20:08:25.9788172Z * [new branch] gh/guangyey/176/base -> origin/gh/guangyey/176/base 2025-08-26T20:08:25.9793848Z * [new branch] gh/guangyey/176/head -> origin/gh/guangyey/176/head 2025-08-26T20:08:25.9796597Z * [new branch] gh/guangyey/176/orig -> origin/gh/guangyey/176/orig 2025-08-26T20:08:25.9797050Z * [new branch] gh/guangyey/177/base -> origin/gh/guangyey/177/base 2025-08-26T20:08:25.9797536Z * [new branch] gh/guangyey/177/head -> origin/gh/guangyey/177/head 2025-08-26T20:08:25.9797914Z * [new branch] gh/guangyey/177/orig -> origin/gh/guangyey/177/orig 2025-08-26T20:08:25.9798364Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-08-26T20:08:25.9798729Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-08-26T20:08:25.9799455Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-08-26T20:08:25.9799848Z * [new branch] gh/guangyey/179/base -> origin/gh/guangyey/179/base 2025-08-26T20:08:25.9800215Z * [new branch] gh/guangyey/179/head -> origin/gh/guangyey/179/head 2025-08-26T20:08:25.9800582Z * [new branch] gh/guangyey/179/orig -> origin/gh/guangyey/179/orig 2025-08-26T20:08:25.9800948Z * [new branch] gh/guangyey/180/base -> origin/gh/guangyey/180/base 2025-08-26T20:08:25.9801317Z * [new branch] gh/guangyey/180/head -> origin/gh/guangyey/180/head 2025-08-26T20:08:25.9801733Z * [new branch] gh/guangyey/180/orig -> origin/gh/guangyey/180/orig 2025-08-26T20:08:25.9802100Z * [new branch] gh/guangyey/181/base -> origin/gh/guangyey/181/base 2025-08-26T20:08:25.9802472Z * [new branch] gh/guangyey/181/head -> origin/gh/guangyey/181/head 2025-08-26T20:08:25.9802859Z * [new branch] gh/guangyey/181/orig -> origin/gh/guangyey/181/orig 2025-08-26T20:08:25.9803228Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-08-26T20:08:25.9803603Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-08-26T20:08:25.9804055Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-08-26T20:08:25.9804640Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-08-26T20:08:25.9805010Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-08-26T20:08:25.9805474Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-08-26T20:08:25.9805856Z * [new branch] gh/guangyey/184/base -> origin/gh/guangyey/184/base 2025-08-26T20:08:25.9806220Z * [new branch] gh/guangyey/184/head -> origin/gh/guangyey/184/head 2025-08-26T20:08:25.9806592Z * [new branch] gh/guangyey/184/orig -> origin/gh/guangyey/184/orig 2025-08-26T20:08:25.9806961Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-08-26T20:08:25.9807345Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-08-26T20:08:25.9807722Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-08-26T20:08:25.9808089Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-08-26T20:08:25.9808459Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-08-26T20:08:25.9808821Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-08-26T20:08:25.9809183Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-08-26T20:08:25.9809549Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-08-26T20:08:25.9810031Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-08-26T20:08:25.9810401Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-08-26T20:08:25.9810766Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-08-26T20:08:25.9811160Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-08-26T20:08:25.9811513Z * [new branch] gh/guangyey/189/base -> origin/gh/guangyey/189/base 2025-08-26T20:08:25.9811873Z * [new branch] gh/guangyey/189/head -> origin/gh/guangyey/189/head 2025-08-26T20:08:25.9812227Z * [new branch] gh/guangyey/189/orig -> origin/gh/guangyey/189/orig 2025-08-26T20:08:25.9812579Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-08-26T20:08:25.9812950Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-08-26T20:08:25.9813312Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-08-26T20:08:25.9813667Z * [new branch] gh/guangyey/191/base -> origin/gh/guangyey/191/base 2025-08-26T20:08:25.9814017Z * [new branch] gh/guangyey/191/head -> origin/gh/guangyey/191/head 2025-08-26T20:08:25.9814370Z * [new branch] gh/guangyey/191/orig -> origin/gh/guangyey/191/orig 2025-08-26T20:08:25.9814725Z * [new branch] gh/guangyey/79/base -> origin/gh/guangyey/79/base 2025-08-26T20:08:25.9815073Z * [new branch] gh/guangyey/79/head -> origin/gh/guangyey/79/head 2025-08-26T20:08:25.9815424Z * [new branch] gh/guangyey/79/orig -> origin/gh/guangyey/79/orig 2025-08-26T20:08:25.9815782Z * [new branch] gh/guangyey/89/base -> origin/gh/guangyey/89/base 2025-08-26T20:08:25.9816136Z * [new branch] gh/guangyey/89/head -> origin/gh/guangyey/89/head 2025-08-26T20:08:25.9816485Z * [new branch] gh/guangyey/89/orig -> origin/gh/guangyey/89/orig 2025-08-26T20:08:25.9816855Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-08-26T20:08:25.9817246Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-08-26T20:08:25.9817908Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-08-26T20:08:25.9818312Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-08-26T20:08:25.9819984Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-08-26T20:08:25.9820385Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-08-26T20:08:25.9820787Z * [new branch] gh/guilhermeleobas/124/base -> origin/gh/guilhermeleobas/124/base 2025-08-26T20:08:25.9821196Z * [new branch] gh/guilhermeleobas/124/head -> origin/gh/guilhermeleobas/124/head 2025-08-26T20:08:25.9821592Z * [new branch] gh/guilhermeleobas/124/orig -> origin/gh/guilhermeleobas/124/orig 2025-08-26T20:08:25.9821996Z * [new branch] gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base 2025-08-26T20:08:25.9825040Z * [new branch] gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-08-26T20:08:25.9825534Z * [new branch] gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig 2025-08-26T20:08:25.9831748Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-08-26T20:08:25.9833860Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-08-26T20:08:25.9834299Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-08-26T20:08:25.9834946Z * [new branch] gh/guilhermeleobas/163/base -> origin/gh/guilhermeleobas/163/base 2025-08-26T20:08:25.9835378Z * [new branch] gh/guilhermeleobas/163/head -> origin/gh/guilhermeleobas/163/head 2025-08-26T20:08:25.9835797Z * [new branch] gh/guilhermeleobas/163/orig -> origin/gh/guilhermeleobas/163/orig 2025-08-26T20:08:25.9836244Z * [new branch] gh/guilhermeleobas/164/base -> origin/gh/guilhermeleobas/164/base 2025-08-26T20:08:25.9836702Z * [new branch] gh/guilhermeleobas/164/head -> origin/gh/guilhermeleobas/164/head 2025-08-26T20:08:25.9837122Z * [new branch] gh/guilhermeleobas/164/orig -> origin/gh/guilhermeleobas/164/orig 2025-08-26T20:08:25.9837570Z * [new branch] gh/guilhermeleobas/165/base -> origin/gh/guilhermeleobas/165/base 2025-08-26T20:08:25.9837998Z * [new branch] gh/guilhermeleobas/165/head -> origin/gh/guilhermeleobas/165/head 2025-08-26T20:08:25.9838423Z * [new branch] gh/guilhermeleobas/165/orig -> origin/gh/guilhermeleobas/165/orig 2025-08-26T20:08:25.9838852Z * [new branch] gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base 2025-08-26T20:08:25.9839501Z * [new branch] gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head 2025-08-26T20:08:25.9839927Z * [new branch] gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig 2025-08-26T20:08:25.9840356Z * [new branch] gh/guilhermeleobas/167/base -> origin/gh/guilhermeleobas/167/base 2025-08-26T20:08:25.9840836Z * [new branch] gh/guilhermeleobas/167/head -> origin/gh/guilhermeleobas/167/head 2025-08-26T20:08:25.9841240Z * [new branch] gh/guilhermeleobas/167/orig -> origin/gh/guilhermeleobas/167/orig 2025-08-26T20:08:25.9841647Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-08-26T20:08:25.9842052Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-08-26T20:08:25.9842466Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-08-26T20:08:25.9842879Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-08-26T20:08:25.9843284Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-08-26T20:08:25.9843799Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-08-26T20:08:25.9844198Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-08-26T20:08:25.9844611Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-08-26T20:08:25.9845024Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-08-26T20:08:25.9845448Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-08-26T20:08:25.9845858Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-08-26T20:08:25.9846263Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-08-26T20:08:25.9846676Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-08-26T20:08:25.9847079Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-08-26T20:08:25.9847490Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-08-26T20:08:25.9847894Z * [new branch] gh/guilhermeleobas/183/base -> origin/gh/guilhermeleobas/183/base 2025-08-26T20:08:25.9848296Z * [new branch] gh/guilhermeleobas/183/head -> origin/gh/guilhermeleobas/183/head 2025-08-26T20:08:25.9848704Z * [new branch] gh/guilhermeleobas/183/orig -> origin/gh/guilhermeleobas/183/orig 2025-08-26T20:08:25.9849144Z * [new branch] gh/guilhermeleobas/184/base -> origin/gh/guilhermeleobas/184/base 2025-08-26T20:08:25.9853184Z * [new branch] gh/guilhermeleobas/184/head -> origin/gh/guilhermeleobas/184/head 2025-08-26T20:08:25.9858760Z * [new branch] gh/guilhermeleobas/184/orig -> origin/gh/guilhermeleobas/184/orig 2025-08-26T20:08:25.9859362Z * [new branch] gh/guilhermeleobas/185/base -> origin/gh/guilhermeleobas/185/base 2025-08-26T20:08:25.9859791Z * [new branch] gh/guilhermeleobas/185/head -> origin/gh/guilhermeleobas/185/head 2025-08-26T20:08:25.9860218Z * [new branch] gh/guilhermeleobas/185/orig -> origin/gh/guilhermeleobas/185/orig 2025-08-26T20:08:25.9860633Z * [new branch] gh/guilhermeleobas/192/base -> origin/gh/guilhermeleobas/192/base 2025-08-26T20:08:25.9861097Z * [new branch] gh/guilhermeleobas/192/head -> origin/gh/guilhermeleobas/192/head 2025-08-26T20:08:25.9861521Z * [new branch] gh/guilhermeleobas/192/orig -> origin/gh/guilhermeleobas/192/orig 2025-08-26T20:08:25.9861914Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-08-26T20:08:25.9862313Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-08-26T20:08:25.9862712Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-08-26T20:08:25.9863115Z * [new branch] gh/guilhermeleobas/194/base -> origin/gh/guilhermeleobas/194/base 2025-08-26T20:08:25.9863523Z * [new branch] gh/guilhermeleobas/194/head -> origin/gh/guilhermeleobas/194/head 2025-08-26T20:08:25.9863927Z * [new branch] gh/guilhermeleobas/194/orig -> origin/gh/guilhermeleobas/194/orig 2025-08-26T20:08:25.9864326Z * [new branch] gh/guilhermeleobas/203/base -> origin/gh/guilhermeleobas/203/base 2025-08-26T20:08:25.9864727Z * [new branch] gh/guilhermeleobas/203/head -> origin/gh/guilhermeleobas/203/head 2025-08-26T20:08:25.9865138Z * [new branch] gh/guilhermeleobas/203/orig -> origin/gh/guilhermeleobas/203/orig 2025-08-26T20:08:25.9865532Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-08-26T20:08:25.9865931Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-08-26T20:08:25.9866492Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-08-26T20:08:25.9866913Z * [new branch] gh/guilhermeleobas/205/base -> origin/gh/guilhermeleobas/205/base 2025-08-26T20:08:25.9867312Z * [new branch] gh/guilhermeleobas/205/head -> origin/gh/guilhermeleobas/205/head 2025-08-26T20:08:25.9867709Z * [new branch] gh/guilhermeleobas/205/orig -> origin/gh/guilhermeleobas/205/orig 2025-08-26T20:08:25.9868104Z * [new branch] gh/guilhermeleobas/206/base -> origin/gh/guilhermeleobas/206/base 2025-08-26T20:08:25.9868507Z * [new branch] gh/guilhermeleobas/206/head -> origin/gh/guilhermeleobas/206/head 2025-08-26T20:08:25.9868909Z * [new branch] gh/guilhermeleobas/206/orig -> origin/gh/guilhermeleobas/206/orig 2025-08-26T20:08:25.9869310Z * [new branch] gh/guilhermeleobas/209/base -> origin/gh/guilhermeleobas/209/base 2025-08-26T20:08:25.9869708Z * [new branch] gh/guilhermeleobas/209/head -> origin/gh/guilhermeleobas/209/head 2025-08-26T20:08:25.9870754Z * [new branch] gh/guilhermeleobas/209/orig -> origin/gh/guilhermeleobas/209/orig 2025-08-26T20:08:25.9871239Z * [new branch] gh/guilhermeleobas/210/base -> origin/gh/guilhermeleobas/210/base 2025-08-26T20:08:25.9871680Z * [new branch] gh/guilhermeleobas/210/head -> origin/gh/guilhermeleobas/210/head 2025-08-26T20:08:25.9872096Z * [new branch] gh/guilhermeleobas/210/orig -> origin/gh/guilhermeleobas/210/orig 2025-08-26T20:08:25.9875073Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-08-26T20:08:25.9875535Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-08-26T20:08:25.9875956Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-08-26T20:08:25.9876377Z * [new branch] gh/guilhermeleobas/213/base -> origin/gh/guilhermeleobas/213/base 2025-08-26T20:08:25.9876825Z * [new branch] gh/guilhermeleobas/213/head -> origin/gh/guilhermeleobas/213/head 2025-08-26T20:08:25.9877259Z * [new branch] gh/guilhermeleobas/213/orig -> origin/gh/guilhermeleobas/213/orig 2025-08-26T20:08:25.9877687Z * [new branch] gh/guilhermeleobas/214/base -> origin/gh/guilhermeleobas/214/base 2025-08-26T20:08:25.9878127Z * [new branch] gh/guilhermeleobas/214/head -> origin/gh/guilhermeleobas/214/head 2025-08-26T20:08:25.9878552Z * [new branch] gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig 2025-08-26T20:08:25.9878990Z * [new branch] gh/guilhermeleobas/215/base -> origin/gh/guilhermeleobas/215/base 2025-08-26T20:08:25.9879789Z * [new branch] gh/guilhermeleobas/215/head -> origin/gh/guilhermeleobas/215/head 2025-08-26T20:08:25.9880280Z * [new branch] gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig 2025-08-26T20:08:25.9883305Z * [new branch] gh/guilhermeleobas/216/base -> origin/gh/guilhermeleobas/216/base 2025-08-26T20:08:25.9883835Z * [new branch] gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head 2025-08-26T20:08:25.9884240Z * [new branch] gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig 2025-08-26T20:08:25.9884638Z * [new branch] gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base 2025-08-26T20:08:25.9889325Z * [new branch] gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head 2025-08-26T20:08:25.9891307Z * [new branch] gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig 2025-08-26T20:08:25.9891740Z * [new branch] gh/guilhermeleobas/218/base -> origin/gh/guilhermeleobas/218/base 2025-08-26T20:08:25.9892160Z * [new branch] gh/guilhermeleobas/218/head -> origin/gh/guilhermeleobas/218/head 2025-08-26T20:08:25.9892717Z * [new branch] gh/guilhermeleobas/218/orig -> origin/gh/guilhermeleobas/218/orig 2025-08-26T20:08:25.9893139Z * [new branch] gh/guilhermeleobas/219/base -> origin/gh/guilhermeleobas/219/base 2025-08-26T20:08:25.9893567Z * [new branch] gh/guilhermeleobas/219/head -> origin/gh/guilhermeleobas/219/head 2025-08-26T20:08:25.9894015Z * [new branch] gh/guilhermeleobas/219/orig -> origin/gh/guilhermeleobas/219/orig 2025-08-26T20:08:25.9894437Z * [new branch] gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base 2025-08-26T20:08:25.9894849Z * [new branch] gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head 2025-08-26T20:08:25.9895280Z * [new branch] gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig 2025-08-26T20:08:25.9897605Z * [new branch] gh/guilhermeleobas/221/base -> origin/gh/guilhermeleobas/221/base 2025-08-26T20:08:25.9898053Z * [new branch] gh/guilhermeleobas/221/head -> origin/gh/guilhermeleobas/221/head 2025-08-26T20:08:25.9898495Z * [new branch] gh/guilhermeleobas/221/orig -> origin/gh/guilhermeleobas/221/orig 2025-08-26T20:08:25.9898888Z * [new branch] gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base 2025-08-26T20:08:25.9899291Z * [new branch] gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head 2025-08-26T20:08:25.9899707Z * [new branch] gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig 2025-08-26T20:08:25.9900239Z * [new branch] gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base 2025-08-26T20:08:25.9900652Z * [new branch] gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head 2025-08-26T20:08:25.9901056Z * [new branch] gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig 2025-08-26T20:08:25.9901456Z * [new branch] gh/guilhermeleobas/224/base -> origin/gh/guilhermeleobas/224/base 2025-08-26T20:08:25.9901861Z * [new branch] gh/guilhermeleobas/224/head -> origin/gh/guilhermeleobas/224/head 2025-08-26T20:08:25.9902269Z * [new branch] gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig 2025-08-26T20:08:25.9903798Z * [new branch] gh/guilhermeleobas/225/base -> origin/gh/guilhermeleobas/225/base 2025-08-26T20:08:25.9904219Z * [new branch] gh/guilhermeleobas/225/head -> origin/gh/guilhermeleobas/225/head 2025-08-26T20:08:25.9904639Z * [new branch] gh/guilhermeleobas/225/orig -> origin/gh/guilhermeleobas/225/orig 2025-08-26T20:08:25.9905565Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-08-26T20:08:25.9906051Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-08-26T20:08:25.9906677Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-08-26T20:08:25.9907365Z * [new branch] gh/guilhermeleobas/227/base -> origin/gh/guilhermeleobas/227/base 2025-08-26T20:08:25.9908034Z * [new branch] gh/guilhermeleobas/227/head -> origin/gh/guilhermeleobas/227/head 2025-08-26T20:08:25.9909704Z * [new branch] gh/guilhermeleobas/227/orig -> origin/gh/guilhermeleobas/227/orig 2025-08-26T20:08:25.9910155Z * [new branch] gh/guilhermeleobas/228/base -> origin/gh/guilhermeleobas/228/base 2025-08-26T20:08:25.9910559Z * [new branch] gh/guilhermeleobas/228/head -> origin/gh/guilhermeleobas/228/head 2025-08-26T20:08:25.9911005Z * [new branch] gh/guilhermeleobas/228/orig -> origin/gh/guilhermeleobas/228/orig 2025-08-26T20:08:25.9911838Z * [new branch] gh/guilhermeleobas/229/base -> origin/gh/guilhermeleobas/229/base 2025-08-26T20:08:25.9912595Z * [new branch] gh/guilhermeleobas/229/head -> origin/gh/guilhermeleobas/229/head 2025-08-26T20:08:25.9913548Z * [new branch] gh/guilhermeleobas/229/orig -> origin/gh/guilhermeleobas/229/orig 2025-08-26T20:08:25.9914240Z * [new branch] gh/guilhermeleobas/230/base -> origin/gh/guilhermeleobas/230/base 2025-08-26T20:08:25.9914866Z * [new branch] gh/guilhermeleobas/230/head -> origin/gh/guilhermeleobas/230/head 2025-08-26T20:08:25.9915642Z * [new branch] gh/guilhermeleobas/230/orig -> origin/gh/guilhermeleobas/230/orig 2025-08-26T20:08:25.9916634Z * [new branch] gh/guilhermeleobas/231/base -> origin/gh/guilhermeleobas/231/base 2025-08-26T20:08:25.9917181Z * [new branch] gh/guilhermeleobas/231/head -> origin/gh/guilhermeleobas/231/head 2025-08-26T20:08:25.9917952Z * [new branch] gh/guilhermeleobas/231/orig -> origin/gh/guilhermeleobas/231/orig 2025-08-26T20:08:25.9919280Z * [new branch] gh/guilhermeleobas/232/base -> origin/gh/guilhermeleobas/232/base 2025-08-26T20:08:25.9919707Z * [new branch] gh/guilhermeleobas/232/head -> origin/gh/guilhermeleobas/232/head 2025-08-26T20:08:25.9920545Z * [new branch] gh/guilhermeleobas/232/orig -> origin/gh/guilhermeleobas/232/orig 2025-08-26T20:08:25.9921465Z * [new branch] gh/guilhermeleobas/233/base -> origin/gh/guilhermeleobas/233/base 2025-08-26T20:08:25.9922096Z * [new branch] gh/guilhermeleobas/233/head -> origin/gh/guilhermeleobas/233/head 2025-08-26T20:08:25.9922774Z * [new branch] gh/guilhermeleobas/233/orig -> origin/gh/guilhermeleobas/233/orig 2025-08-26T20:08:25.9924015Z * [new branch] gh/guilhermeleobas/234/base -> origin/gh/guilhermeleobas/234/base 2025-08-26T20:08:25.9924431Z * [new branch] gh/guilhermeleobas/234/head -> origin/gh/guilhermeleobas/234/head 2025-08-26T20:08:25.9925058Z * [new branch] gh/guilhermeleobas/234/orig -> origin/gh/guilhermeleobas/234/orig 2025-08-26T20:08:25.9926288Z * [new branch] gh/guilhermeleobas/235/base -> origin/gh/guilhermeleobas/235/base 2025-08-26T20:08:25.9928239Z * [new branch] gh/guilhermeleobas/235/head -> origin/gh/guilhermeleobas/235/head 2025-08-26T20:08:25.9928642Z * [new branch] gh/guilhermeleobas/235/orig -> origin/gh/guilhermeleobas/235/orig 2025-08-26T20:08:25.9929039Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-08-26T20:08:25.9929580Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-08-26T20:08:25.9931199Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-08-26T20:08:25.9931713Z * [new branch] gh/guilhermeleobas/237/base -> origin/gh/guilhermeleobas/237/base 2025-08-26T20:08:25.9932156Z * [new branch] gh/guilhermeleobas/237/head -> origin/gh/guilhermeleobas/237/head 2025-08-26T20:08:25.9932648Z * [new branch] gh/guilhermeleobas/237/orig -> origin/gh/guilhermeleobas/237/orig 2025-08-26T20:08:25.9933277Z * [new branch] gh/guilhermeleobas/238/base -> origin/gh/guilhermeleobas/238/base 2025-08-26T20:08:25.9933912Z * [new branch] gh/guilhermeleobas/238/head -> origin/gh/guilhermeleobas/238/head 2025-08-26T20:08:25.9934576Z * [new branch] gh/guilhermeleobas/238/orig -> origin/gh/guilhermeleobas/238/orig 2025-08-26T20:08:25.9936032Z * [new branch] gh/guilhermeleobas/239/base -> origin/gh/guilhermeleobas/239/base 2025-08-26T20:08:25.9936446Z * [new branch] gh/guilhermeleobas/239/head -> origin/gh/guilhermeleobas/239/head 2025-08-26T20:08:25.9936883Z * [new branch] gh/guilhermeleobas/239/orig -> origin/gh/guilhermeleobas/239/orig 2025-08-26T20:08:25.9939078Z * [new branch] gh/guilhermeleobas/73/base -> origin/gh/guilhermeleobas/73/base 2025-08-26T20:08:25.9939509Z * [new branch] gh/guilhermeleobas/73/head -> origin/gh/guilhermeleobas/73/head 2025-08-26T20:08:25.9940071Z * [new branch] gh/guilhermeleobas/73/orig -> origin/gh/guilhermeleobas/73/orig 2025-08-26T20:08:25.9940505Z * [new branch] gh/henrylhtsang/103/base -> origin/gh/henrylhtsang/103/base 2025-08-26T20:08:25.9941289Z * [new branch] gh/henrylhtsang/103/head -> origin/gh/henrylhtsang/103/head 2025-08-26T20:08:25.9942027Z * [new branch] gh/henrylhtsang/103/orig -> origin/gh/henrylhtsang/103/orig 2025-08-26T20:08:25.9943134Z * [new branch] gh/henrylhtsang/132/base -> origin/gh/henrylhtsang/132/base 2025-08-26T20:08:25.9943656Z * [new branch] gh/henrylhtsang/132/head -> origin/gh/henrylhtsang/132/head 2025-08-26T20:08:25.9945074Z * [new branch] gh/henrylhtsang/132/orig -> origin/gh/henrylhtsang/132/orig 2025-08-26T20:08:25.9945816Z * [new branch] gh/henrylhtsang/133/base -> origin/gh/henrylhtsang/133/base 2025-08-26T20:08:25.9947312Z * [new branch] gh/henrylhtsang/133/head -> origin/gh/henrylhtsang/133/head 2025-08-26T20:08:25.9947724Z * [new branch] gh/henrylhtsang/133/orig -> origin/gh/henrylhtsang/133/orig 2025-08-26T20:08:25.9948123Z * [new branch] gh/henrylhtsang/134/base -> origin/gh/henrylhtsang/134/base 2025-08-26T20:08:25.9948867Z * [new branch] gh/henrylhtsang/134/head -> origin/gh/henrylhtsang/134/head 2025-08-26T20:08:25.9950217Z * [new branch] gh/henrylhtsang/134/orig -> origin/gh/henrylhtsang/134/orig 2025-08-26T20:08:25.9951028Z * [new branch] gh/henrylhtsang/135/base -> origin/gh/henrylhtsang/135/base 2025-08-26T20:08:25.9951740Z * [new branch] gh/henrylhtsang/135/head -> origin/gh/henrylhtsang/135/head 2025-08-26T20:08:25.9952330Z * [new branch] gh/henrylhtsang/135/orig -> origin/gh/henrylhtsang/135/orig 2025-08-26T20:08:25.9953441Z * [new branch] gh/henrylhtsang/136/base -> origin/gh/henrylhtsang/136/base 2025-08-26T20:08:25.9953837Z * [new branch] gh/henrylhtsang/136/head -> origin/gh/henrylhtsang/136/head 2025-08-26T20:08:25.9954796Z * [new branch] gh/henrylhtsang/136/orig -> origin/gh/henrylhtsang/136/orig 2025-08-26T20:08:25.9955512Z * [new branch] gh/henrylhtsang/137/base -> origin/gh/henrylhtsang/137/base 2025-08-26T20:08:25.9956197Z * [new branch] gh/henrylhtsang/137/head -> origin/gh/henrylhtsang/137/head 2025-08-26T20:08:25.9956837Z * [new branch] gh/henrylhtsang/137/orig -> origin/gh/henrylhtsang/137/orig 2025-08-26T20:08:25.9957990Z * [new branch] gh/henrylhtsang/138/base -> origin/gh/henrylhtsang/138/base 2025-08-26T20:08:25.9958498Z * [new branch] gh/henrylhtsang/138/head -> origin/gh/henrylhtsang/138/head 2025-08-26T20:08:25.9959483Z * [new branch] gh/henrylhtsang/138/orig -> origin/gh/henrylhtsang/138/orig 2025-08-26T20:08:25.9960325Z * [new branch] gh/henrylhtsang/139/base -> origin/gh/henrylhtsang/139/base 2025-08-26T20:08:25.9961025Z * [new branch] gh/henrylhtsang/139/head -> origin/gh/henrylhtsang/139/head 2025-08-26T20:08:25.9961835Z * [new branch] gh/henrylhtsang/139/orig -> origin/gh/henrylhtsang/139/orig 2025-08-26T20:08:25.9962941Z * [new branch] gh/henrylhtsang/140/base -> origin/gh/henrylhtsang/140/base 2025-08-26T20:08:25.9963551Z * [new branch] gh/henrylhtsang/140/head -> origin/gh/henrylhtsang/140/head 2025-08-26T20:08:25.9964216Z * [new branch] gh/henrylhtsang/140/orig -> origin/gh/henrylhtsang/140/orig 2025-08-26T20:08:25.9965556Z * [new branch] gh/henrylhtsang/141/base -> origin/gh/henrylhtsang/141/base 2025-08-26T20:08:25.9965938Z * [new branch] gh/henrylhtsang/141/head -> origin/gh/henrylhtsang/141/head 2025-08-26T20:08:25.9966487Z * [new branch] gh/henrylhtsang/141/orig -> origin/gh/henrylhtsang/141/orig 2025-08-26T20:08:25.9971399Z * [new branch] gh/henrylhtsang/142/base -> origin/gh/henrylhtsang/142/base 2025-08-26T20:08:25.9971870Z * [new branch] gh/henrylhtsang/142/head -> origin/gh/henrylhtsang/142/head 2025-08-26T20:08:25.9972276Z * [new branch] gh/henrylhtsang/142/orig -> origin/gh/henrylhtsang/142/orig 2025-08-26T20:08:25.9972676Z * [new branch] gh/henrylhtsang/143/base -> origin/gh/henrylhtsang/143/base 2025-08-26T20:08:25.9973066Z * [new branch] gh/henrylhtsang/143/head -> origin/gh/henrylhtsang/143/head 2025-08-26T20:08:25.9973484Z * [new branch] gh/henrylhtsang/143/orig -> origin/gh/henrylhtsang/143/orig 2025-08-26T20:08:25.9973942Z * [new branch] gh/henrylhtsang/144/base -> origin/gh/henrylhtsang/144/base 2025-08-26T20:08:25.9974335Z * [new branch] gh/henrylhtsang/144/head -> origin/gh/henrylhtsang/144/head 2025-08-26T20:08:25.9975046Z * [new branch] gh/henrylhtsang/144/orig -> origin/gh/henrylhtsang/144/orig 2025-08-26T20:08:25.9975983Z * [new branch] gh/henrylhtsang/145/base -> origin/gh/henrylhtsang/145/base 2025-08-26T20:08:25.9976632Z * [new branch] gh/henrylhtsang/145/head -> origin/gh/henrylhtsang/145/head 2025-08-26T20:08:25.9977328Z * [new branch] gh/henrylhtsang/145/orig -> origin/gh/henrylhtsang/145/orig 2025-08-26T20:08:25.9978966Z * [new branch] gh/henrylhtsang/146/base -> origin/gh/henrylhtsang/146/base 2025-08-26T20:08:25.9979538Z * [new branch] gh/henrylhtsang/146/head -> origin/gh/henrylhtsang/146/head 2025-08-26T20:08:25.9980555Z * [new branch] gh/henrylhtsang/146/orig -> origin/gh/henrylhtsang/146/orig 2025-08-26T20:08:25.9981141Z * [new branch] gh/henrylhtsang/147/base -> origin/gh/henrylhtsang/147/base 2025-08-26T20:08:25.9981882Z * [new branch] gh/henrylhtsang/147/head -> origin/gh/henrylhtsang/147/head 2025-08-26T20:08:25.9983456Z * [new branch] gh/henrylhtsang/147/orig -> origin/gh/henrylhtsang/147/orig 2025-08-26T20:08:25.9983861Z * [new branch] gh/henrylhtsang/148/base -> origin/gh/henrylhtsang/148/base 2025-08-26T20:08:25.9984535Z * [new branch] gh/henrylhtsang/148/head -> origin/gh/henrylhtsang/148/head 2025-08-26T20:08:25.9985203Z * [new branch] gh/henrylhtsang/148/orig -> origin/gh/henrylhtsang/148/orig 2025-08-26T20:08:25.9986133Z * [new branch] gh/henrylhtsang/149/base -> origin/gh/henrylhtsang/149/base 2025-08-26T20:08:25.9986738Z * [new branch] gh/henrylhtsang/149/head -> origin/gh/henrylhtsang/149/head 2025-08-26T20:08:25.9987341Z * [new branch] gh/henrylhtsang/149/orig -> origin/gh/henrylhtsang/149/orig 2025-08-26T20:08:25.9988783Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-08-26T20:08:25.9989363Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-08-26T20:08:25.9990494Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-08-26T20:08:25.9992171Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-08-26T20:08:25.9993006Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-08-26T20:08:25.9993811Z * [new branch] gh/huydhn/6/head -> origin/gh/huydhn/6/head 2025-08-26T20:08:25.9994420Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-08-26T20:08:25.9995043Z * [new branch] gh/huydhn/6/orig -> origin/gh/huydhn/6/orig 2025-08-26T20:08:25.9996666Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-08-26T20:08:25.9997249Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-08-26T20:08:25.9998398Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-08-26T20:08:25.9999006Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-08-26T20:08:26.0000748Z * [new branch] gh/isuruf/116/base -> origin/gh/isuruf/116/base 2025-08-26T20:08:26.0001126Z * [new branch] gh/isuruf/116/head -> origin/gh/isuruf/116/head 2025-08-26T20:08:26.0001515Z * [new branch] gh/isuruf/116/orig -> origin/gh/isuruf/116/orig 2025-08-26T20:08:26.0003539Z * [new branch] gh/isuruf/141/base -> origin/gh/isuruf/141/base 2025-08-26T20:08:26.0004110Z * [new branch] gh/isuruf/141/head -> origin/gh/isuruf/141/head 2025-08-26T20:08:26.0004619Z * [new branch] gh/isuruf/141/orig -> origin/gh/isuruf/141/orig 2025-08-26T20:08:26.0005118Z * [new branch] gh/isuruf/142/base -> origin/gh/isuruf/142/base 2025-08-26T20:08:26.0005620Z * [new branch] gh/isuruf/142/head -> origin/gh/isuruf/142/head 2025-08-26T20:08:26.0006143Z * [new branch] gh/isuruf/142/orig -> origin/gh/isuruf/142/orig 2025-08-26T20:08:26.0007345Z * [new branch] gh/isuruf/143/base -> origin/gh/isuruf/143/base 2025-08-26T20:08:26.0007874Z * [new branch] gh/isuruf/143/head -> origin/gh/isuruf/143/head 2025-08-26T20:08:26.0008543Z * [new branch] gh/isuruf/143/orig -> origin/gh/isuruf/143/orig 2025-08-26T20:08:26.0009804Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-08-26T20:08:26.0010342Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-08-26T20:08:26.0010796Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-08-26T20:08:26.0014489Z * [new branch] gh/jamesjwu/140/base -> origin/gh/jamesjwu/140/base 2025-08-26T20:08:26.0014938Z * [new branch] gh/jamesjwu/140/head -> origin/gh/jamesjwu/140/head 2025-08-26T20:08:26.0015322Z * [new branch] gh/jamesjwu/140/orig -> origin/gh/jamesjwu/140/orig 2025-08-26T20:08:26.0015684Z * [new branch] gh/jamesjwu/150/base -> origin/gh/jamesjwu/150/base 2025-08-26T20:08:26.0016042Z * [new branch] gh/jamesjwu/150/head -> origin/gh/jamesjwu/150/head 2025-08-26T20:08:26.0016390Z * [new branch] gh/jamesjwu/150/orig -> origin/gh/jamesjwu/150/orig 2025-08-26T20:08:26.0016794Z * [new branch] gh/jamesjwu/154/base -> origin/gh/jamesjwu/154/base 2025-08-26T20:08:26.0017497Z * [new branch] gh/jamesjwu/154/head -> origin/gh/jamesjwu/154/head 2025-08-26T20:08:26.0018092Z * [new branch] gh/jamesjwu/154/orig -> origin/gh/jamesjwu/154/orig 2025-08-26T20:08:26.0019087Z * [new branch] gh/jamesjwu/155/base -> origin/gh/jamesjwu/155/base 2025-08-26T20:08:26.0025107Z * [new branch] gh/jamesjwu/155/head -> origin/gh/jamesjwu/155/head 2025-08-26T20:08:26.0025535Z * [new branch] gh/jamesjwu/155/orig -> origin/gh/jamesjwu/155/orig 2025-08-26T20:08:26.0025909Z * [new branch] gh/jamesjwu/159/base -> origin/gh/jamesjwu/159/base 2025-08-26T20:08:26.0026264Z * [new branch] gh/jamesjwu/159/head -> origin/gh/jamesjwu/159/head 2025-08-26T20:08:26.0026612Z * [new branch] gh/jamesjwu/159/orig -> origin/gh/jamesjwu/159/orig 2025-08-26T20:08:26.0026987Z * [new branch] gh/jamesjwu/163/base -> origin/gh/jamesjwu/163/base 2025-08-26T20:08:26.0027341Z * [new branch] gh/jamesjwu/163/head -> origin/gh/jamesjwu/163/head 2025-08-26T20:08:26.0027707Z * [new branch] gh/jamesjwu/163/orig -> origin/gh/jamesjwu/163/orig 2025-08-26T20:08:26.0028057Z * [new branch] gh/jamesjwu/171/base -> origin/gh/jamesjwu/171/base 2025-08-26T20:08:26.0028601Z * [new branch] gh/jamesjwu/171/head -> origin/gh/jamesjwu/171/head 2025-08-26T20:08:26.0028961Z * [new branch] gh/jamesjwu/171/orig -> origin/gh/jamesjwu/171/orig 2025-08-26T20:08:26.0029299Z * [new branch] gh/jamesjwu/175/base -> origin/gh/jamesjwu/175/base 2025-08-26T20:08:26.0029643Z * [new branch] gh/jamesjwu/175/head -> origin/gh/jamesjwu/175/head 2025-08-26T20:08:26.0029993Z * [new branch] gh/jamesjwu/175/orig -> origin/gh/jamesjwu/175/orig 2025-08-26T20:08:26.0030834Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-08-26T20:08:26.0031200Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-08-26T20:08:26.0031596Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-08-26T20:08:26.0033217Z * [new branch] gh/jamesjwu/180/base -> origin/gh/jamesjwu/180/base 2025-08-26T20:08:26.0033582Z * [new branch] gh/jamesjwu/180/head -> origin/gh/jamesjwu/180/head 2025-08-26T20:08:26.0033931Z * [new branch] gh/jamesjwu/180/orig -> origin/gh/jamesjwu/180/orig 2025-08-26T20:08:26.0035052Z * [new branch] gh/jamesjwu/181/base -> origin/gh/jamesjwu/181/base 2025-08-26T20:08:26.0036129Z * [new branch] gh/jamesjwu/181/head -> origin/gh/jamesjwu/181/head 2025-08-26T20:08:26.0036496Z * [new branch] gh/jamesjwu/181/orig -> origin/gh/jamesjwu/181/orig 2025-08-26T20:08:26.0036911Z * [new branch] gh/jamesjwu/182/base -> origin/gh/jamesjwu/182/base 2025-08-26T20:08:26.0037601Z * [new branch] gh/jamesjwu/182/head -> origin/gh/jamesjwu/182/head 2025-08-26T20:08:26.0044445Z * [new branch] gh/jamesjwu/182/orig -> origin/gh/jamesjwu/182/orig 2025-08-26T20:08:26.0045566Z * [new branch] gh/jamesjwu/183/base -> origin/gh/jamesjwu/183/base 2025-08-26T20:08:26.0046099Z * [new branch] gh/jamesjwu/183/head -> origin/gh/jamesjwu/183/head 2025-08-26T20:08:26.0046600Z * [new branch] gh/jamesjwu/183/orig -> origin/gh/jamesjwu/183/orig 2025-08-26T20:08:26.0047094Z * [new branch] gh/jamesjwu/184/base -> origin/gh/jamesjwu/184/base 2025-08-26T20:08:26.0047591Z * [new branch] gh/jamesjwu/184/head -> origin/gh/jamesjwu/184/head 2025-08-26T20:08:26.0048083Z * [new branch] gh/jamesjwu/184/orig -> origin/gh/jamesjwu/184/orig 2025-08-26T20:08:26.0048942Z * [new branch] gh/jamesjwu/185/base -> origin/gh/jamesjwu/185/base 2025-08-26T20:08:26.0049555Z * [new branch] gh/jamesjwu/185/head -> origin/gh/jamesjwu/185/head 2025-08-26T20:08:26.0050104Z * [new branch] gh/jamesjwu/185/orig -> origin/gh/jamesjwu/185/orig 2025-08-26T20:08:26.0051059Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-08-26T20:08:26.0051653Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-08-26T20:08:26.0052139Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-08-26T20:08:26.0052518Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-08-26T20:08:26.0052878Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-08-26T20:08:26.0053432Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-08-26T20:08:26.0053796Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-08-26T20:08:26.0054167Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-08-26T20:08:26.0088502Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-08-26T20:08:26.0089174Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-08-26T20:08:26.0089760Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-08-26T20:08:26.0090277Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-08-26T20:08:26.0090740Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-08-26T20:08:26.0091184Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-08-26T20:08:26.0091665Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-08-26T20:08:26.0092143Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-08-26T20:08:26.0093002Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-08-26T20:08:26.0093526Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-08-26T20:08:26.0093957Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-08-26T20:08:26.0094358Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-08-26T20:08:26.0094780Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-08-26T20:08:26.0095195Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-08-26T20:08:26.0095579Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-08-26T20:08:26.0096170Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-08-26T20:08:26.0096770Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-08-26T20:08:26.0097193Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-08-26T20:08:26.0097599Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-08-26T20:08:26.0097952Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-08-26T20:08:26.0098345Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-08-26T20:08:26.0098715Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-08-26T20:08:26.0099133Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-08-26T20:08:26.0099520Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-08-26T20:08:26.0099862Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-08-26T20:08:26.0100186Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-08-26T20:08:26.0100530Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-08-26T20:08:26.0100883Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-08-26T20:08:26.0101238Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-08-26T20:08:26.0101589Z * [new branch] gh/janeyx99/282/base -> origin/gh/janeyx99/282/base 2025-08-26T20:08:26.0101938Z * [new branch] gh/janeyx99/282/head -> origin/gh/janeyx99/282/head 2025-08-26T20:08:26.0102283Z * [new branch] gh/janeyx99/282/orig -> origin/gh/janeyx99/282/orig 2025-08-26T20:08:26.0102636Z * [new branch] gh/janeyx99/283/base -> origin/gh/janeyx99/283/base 2025-08-26T20:08:26.0102980Z * [new branch] gh/janeyx99/283/head -> origin/gh/janeyx99/283/head 2025-08-26T20:08:26.0103306Z * [new branch] gh/janeyx99/283/orig -> origin/gh/janeyx99/283/orig 2025-08-26T20:08:26.0103624Z * [new branch] gh/janeyx99/284/base -> origin/gh/janeyx99/284/base 2025-08-26T20:08:26.0104072Z * [new branch] gh/janeyx99/284/head -> origin/gh/janeyx99/284/head 2025-08-26T20:08:26.0104429Z * [new branch] gh/janeyx99/284/orig -> origin/gh/janeyx99/284/orig 2025-08-26T20:08:26.0104785Z * [new branch] gh/janeyx99/285/base -> origin/gh/janeyx99/285/base 2025-08-26T20:08:26.0105131Z * [new branch] gh/janeyx99/285/head -> origin/gh/janeyx99/285/head 2025-08-26T20:08:26.0105467Z * [new branch] gh/janeyx99/285/orig -> origin/gh/janeyx99/285/orig 2025-08-26T20:08:26.0105818Z * [new branch] gh/janeyx99/286/base -> origin/gh/janeyx99/286/base 2025-08-26T20:08:26.0106158Z * [new branch] gh/janeyx99/286/head -> origin/gh/janeyx99/286/head 2025-08-26T20:08:26.0106489Z * [new branch] gh/janeyx99/286/orig -> origin/gh/janeyx99/286/orig 2025-08-26T20:08:26.0106840Z * [new branch] gh/janeyx99/287/base -> origin/gh/janeyx99/287/base 2025-08-26T20:08:26.0107180Z * [new branch] gh/janeyx99/287/head -> origin/gh/janeyx99/287/head 2025-08-26T20:08:26.0107528Z * [new branch] gh/janeyx99/287/orig -> origin/gh/janeyx99/287/orig 2025-08-26T20:08:26.0107871Z * [new branch] gh/janeyx99/288/base -> origin/gh/janeyx99/288/base 2025-08-26T20:08:26.0108212Z * [new branch] gh/janeyx99/288/head -> origin/gh/janeyx99/288/head 2025-08-26T20:08:26.0108574Z * [new branch] gh/janeyx99/288/orig -> origin/gh/janeyx99/288/orig 2025-08-26T20:08:26.0108977Z * [new branch] gh/janeyx99/289/base -> origin/gh/janeyx99/289/base 2025-08-26T20:08:26.0109322Z * [new branch] gh/janeyx99/289/head -> origin/gh/janeyx99/289/head 2025-08-26T20:08:26.0109666Z * [new branch] gh/janeyx99/289/orig -> origin/gh/janeyx99/289/orig 2025-08-26T20:08:26.0110009Z * [new branch] gh/janeyx99/290/base -> origin/gh/janeyx99/290/base 2025-08-26T20:08:26.0110361Z * [new branch] gh/janeyx99/290/head -> origin/gh/janeyx99/290/head 2025-08-26T20:08:26.0110706Z * [new branch] gh/janeyx99/290/orig -> origin/gh/janeyx99/290/orig 2025-08-26T20:08:26.0111062Z * [new branch] gh/janeyx99/291/base -> origin/gh/janeyx99/291/base 2025-08-26T20:08:26.0111414Z * [new branch] gh/janeyx99/291/head -> origin/gh/janeyx99/291/head 2025-08-26T20:08:26.0111766Z * [new branch] gh/janeyx99/291/orig -> origin/gh/janeyx99/291/orig 2025-08-26T20:08:26.0112116Z * [new branch] gh/janeyx99/292/base -> origin/gh/janeyx99/292/base 2025-08-26T20:08:26.0112462Z * [new branch] gh/janeyx99/292/head -> origin/gh/janeyx99/292/head 2025-08-26T20:08:26.0112803Z * [new branch] gh/janeyx99/292/orig -> origin/gh/janeyx99/292/orig 2025-08-26T20:08:26.0113141Z * [new branch] gh/janeyx99/293/base -> origin/gh/janeyx99/293/base 2025-08-26T20:08:26.0113485Z * [new branch] gh/janeyx99/293/head -> origin/gh/janeyx99/293/head 2025-08-26T20:08:26.0113831Z * [new branch] gh/janeyx99/293/orig -> origin/gh/janeyx99/293/orig 2025-08-26T20:08:26.0114162Z * [new branch] gh/janeyx99/294/base -> origin/gh/janeyx99/294/base 2025-08-26T20:08:26.0114500Z * [new branch] gh/janeyx99/294/head -> origin/gh/janeyx99/294/head 2025-08-26T20:08:26.0114836Z * [new branch] gh/janeyx99/294/orig -> origin/gh/janeyx99/294/orig 2025-08-26T20:08:26.0115174Z * [new branch] gh/janeyx99/295/base -> origin/gh/janeyx99/295/base 2025-08-26T20:08:26.0115513Z * [new branch] gh/janeyx99/295/head -> origin/gh/janeyx99/295/head 2025-08-26T20:08:26.0115846Z * [new branch] gh/janeyx99/295/orig -> origin/gh/janeyx99/295/orig 2025-08-26T20:08:26.0116239Z * [new branch] gh/janeyx99/296/base -> origin/gh/janeyx99/296/base 2025-08-26T20:08:26.0116588Z * [new branch] gh/janeyx99/296/head -> origin/gh/janeyx99/296/head 2025-08-26T20:08:26.0116722Z * [new branch] gh/janeyx99/296/orig -> origin/gh/janeyx99/296/orig 2025-08-26T20:08:26.0116865Z * [new branch] gh/janeyx99/297/base -> origin/gh/janeyx99/297/base 2025-08-26T20:08:26.0117003Z * [new branch] gh/janeyx99/297/head -> origin/gh/janeyx99/297/head 2025-08-26T20:08:26.0117147Z * [new branch] gh/janeyx99/297/orig -> origin/gh/janeyx99/297/orig 2025-08-26T20:08:26.0117294Z * [new branch] gh/janeyx99/298/base -> origin/gh/janeyx99/298/base 2025-08-26T20:08:26.0117432Z * [new branch] gh/janeyx99/298/head -> origin/gh/janeyx99/298/head 2025-08-26T20:08:26.0117571Z * [new branch] gh/janeyx99/298/orig -> origin/gh/janeyx99/298/orig 2025-08-26T20:08:26.0117714Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-08-26T20:08:26.0117859Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-08-26T20:08:26.0117997Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-08-26T20:08:26.0118139Z * [new branch] gh/janeyx99/300/base -> origin/gh/janeyx99/300/base 2025-08-26T20:08:26.0118275Z * [new branch] gh/janeyx99/300/head -> origin/gh/janeyx99/300/head 2025-08-26T20:08:26.0118466Z * [new branch] gh/janeyx99/300/orig -> origin/gh/janeyx99/300/orig 2025-08-26T20:08:26.0118625Z * [new branch] gh/janeyx99/301/base -> origin/gh/janeyx99/301/base 2025-08-26T20:08:26.0118770Z * [new branch] gh/janeyx99/301/head -> origin/gh/janeyx99/301/head 2025-08-26T20:08:26.0119737Z * [new branch] gh/janeyx99/301/orig -> origin/gh/janeyx99/301/orig 2025-08-26T20:08:26.0126294Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-08-26T20:08:26.0126916Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-08-26T20:08:26.0127098Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-08-26T20:08:26.0127257Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-08-26T20:08:26.0127403Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-08-26T20:08:26.0127569Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-08-26T20:08:26.0127713Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-08-26T20:08:26.0130523Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-08-26T20:08:26.0130760Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-08-26T20:08:26.0130943Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-08-26T20:08:26.0131499Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-08-26T20:08:26.0131667Z * [new branch] gh/jansel/531/base -> origin/gh/jansel/531/base 2025-08-26T20:08:26.0131813Z * [new branch] gh/jansel/531/head -> origin/gh/jansel/531/head 2025-08-26T20:08:26.0131955Z * [new branch] gh/jansel/531/orig -> origin/gh/jansel/531/orig 2025-08-26T20:08:26.0134731Z * [new branch] gh/jansel/534/base -> origin/gh/jansel/534/base 2025-08-26T20:08:26.0135094Z * [new branch] gh/jansel/534/head -> origin/gh/jansel/534/head 2025-08-26T20:08:26.0135282Z * [new branch] gh/jansel/534/orig -> origin/gh/jansel/534/orig 2025-08-26T20:08:26.0135460Z * [new branch] gh/jbschlosser/208/head -> origin/gh/jbschlosser/208/head 2025-08-26T20:08:26.0135901Z * [new branch] gh/jbschlosser/239/base -> origin/gh/jbschlosser/239/base 2025-08-26T20:08:26.0136091Z * [new branch] gh/jbschlosser/239/head -> origin/gh/jbschlosser/239/head 2025-08-26T20:08:26.0136259Z * [new branch] gh/jbschlosser/239/orig -> origin/gh/jbschlosser/239/orig 2025-08-26T20:08:26.0139272Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-08-26T20:08:26.0139531Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-08-26T20:08:26.0139780Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-08-26T20:08:26.0139965Z * [new branch] gh/jbschlosser/248/base -> origin/gh/jbschlosser/248/base 2025-08-26T20:08:26.0140168Z * [new branch] gh/jbschlosser/248/head -> origin/gh/jbschlosser/248/head 2025-08-26T20:08:26.0140463Z * [new branch] gh/jbschlosser/248/orig -> origin/gh/jbschlosser/248/orig 2025-08-26T20:08:26.0148118Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-08-26T20:08:26.0148348Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-08-26T20:08:26.0148536Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-08-26T20:08:26.0148794Z * [new branch] gh/jiayisunx/57/base -> origin/gh/jiayisunx/57/base 2025-08-26T20:08:26.0149347Z * [new branch] gh/jiayisunx/57/head -> origin/gh/jiayisunx/57/head 2025-08-26T20:08:26.0149570Z * [new branch] gh/jiayisunx/57/orig -> origin/gh/jiayisunx/57/orig 2025-08-26T20:08:26.0153032Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-08-26T20:08:26.0153230Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-08-26T20:08:26.0153404Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-08-26T20:08:26.0153566Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-08-26T20:08:26.0153719Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-08-26T20:08:26.0153870Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-08-26T20:08:26.0154026Z * [new branch] gh/jiayisunx/64/base -> origin/gh/jiayisunx/64/base 2025-08-26T20:08:26.0154183Z * [new branch] gh/jiayisunx/64/head -> origin/gh/jiayisunx/64/head 2025-08-26T20:08:26.0154333Z * [new branch] gh/jiayisunx/64/orig -> origin/gh/jiayisunx/64/orig 2025-08-26T20:08:26.0154482Z * [new branch] gh/jiayisunx/65/base -> origin/gh/jiayisunx/65/base 2025-08-26T20:08:26.0154639Z * [new branch] gh/jiayisunx/65/head -> origin/gh/jiayisunx/65/head 2025-08-26T20:08:26.0154787Z * [new branch] gh/jiayisunx/65/orig -> origin/gh/jiayisunx/65/orig 2025-08-26T20:08:26.0154939Z * [new branch] gh/jiayisunx/66/base -> origin/gh/jiayisunx/66/base 2025-08-26T20:08:26.0155088Z * [new branch] gh/jiayisunx/66/head -> origin/gh/jiayisunx/66/head 2025-08-26T20:08:26.0155242Z * [new branch] gh/jiayisunx/66/orig -> origin/gh/jiayisunx/66/orig 2025-08-26T20:08:26.0155397Z * [new branch] gh/jiayisunx/67/base -> origin/gh/jiayisunx/67/base 2025-08-26T20:08:26.0155772Z * [new branch] gh/jiayisunx/67/head -> origin/gh/jiayisunx/67/head 2025-08-26T20:08:26.0156382Z * [new branch] gh/jiayisunx/67/orig -> origin/gh/jiayisunx/67/orig 2025-08-26T20:08:26.0157555Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-08-26T20:08:26.0158004Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-08-26T20:08:26.0158924Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-08-26T20:08:26.0160118Z * [new branch] gh/jiayisunx/69/base -> origin/gh/jiayisunx/69/base 2025-08-26T20:08:26.0160400Z * [new branch] gh/jiayisunx/69/head -> origin/gh/jiayisunx/69/head 2025-08-26T20:08:26.0163751Z * [new branch] gh/jiayisunx/69/orig -> origin/gh/jiayisunx/69/orig 2025-08-26T20:08:26.0163958Z * [new branch] gh/jiayisunx/70/base -> origin/gh/jiayisunx/70/base 2025-08-26T20:08:26.0164107Z * [new branch] gh/jiayisunx/70/head -> origin/gh/jiayisunx/70/head 2025-08-26T20:08:26.0164259Z * [new branch] gh/jiayisunx/70/orig -> origin/gh/jiayisunx/70/orig 2025-08-26T20:08:26.0164472Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-08-26T20:08:26.0165292Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-08-26T20:08:26.0166390Z * [new branch] gh/justinchuby/111/base -> origin/gh/justinchuby/111/base 2025-08-26T20:08:26.0166797Z * [new branch] gh/justinchuby/111/head -> origin/gh/justinchuby/111/head 2025-08-26T20:08:26.0169891Z * [new branch] gh/justinchuby/111/orig -> origin/gh/justinchuby/111/orig 2025-08-26T20:08:26.0175298Z * [new branch] gh/justinchuby/112/base -> origin/gh/justinchuby/112/base 2025-08-26T20:08:26.0177557Z * [new branch] gh/justinchuby/112/head -> origin/gh/justinchuby/112/head 2025-08-26T20:08:26.0177737Z * [new branch] gh/justinchuby/112/orig -> origin/gh/justinchuby/112/orig 2025-08-26T20:08:26.0177928Z * [new branch] gh/justinchuby/113/base -> origin/gh/justinchuby/113/base 2025-08-26T20:08:26.0178163Z * [new branch] gh/justinchuby/113/head -> origin/gh/justinchuby/113/head 2025-08-26T20:08:26.0178343Z * [new branch] gh/justinchuby/113/orig -> origin/gh/justinchuby/113/orig 2025-08-26T20:08:26.0178511Z * [new branch] gh/justinchuby/114/base -> origin/gh/justinchuby/114/base 2025-08-26T20:08:26.0178675Z * [new branch] gh/justinchuby/114/head -> origin/gh/justinchuby/114/head 2025-08-26T20:08:26.0178845Z * [new branch] gh/justinchuby/114/orig -> origin/gh/justinchuby/114/orig 2025-08-26T20:08:26.0179006Z * [new branch] gh/karthickai/1/base -> origin/gh/karthickai/1/base 2025-08-26T20:08:26.0179157Z * [new branch] gh/karthickai/1/head -> origin/gh/karthickai/1/head 2025-08-26T20:08:26.0179315Z * [new branch] gh/karthickai/1/orig -> origin/gh/karthickai/1/orig 2025-08-26T20:08:26.0184288Z * [new branch] gh/karthickai/2/base -> origin/gh/karthickai/2/base 2025-08-26T20:08:26.0187203Z * [new branch] gh/karthickai/2/head -> origin/gh/karthickai/2/head 2025-08-26T20:08:26.0187372Z * [new branch] gh/karthickai/2/orig -> origin/gh/karthickai/2/orig 2025-08-26T20:08:26.0187775Z * [new branch] gh/kurtamohler/32/base -> origin/gh/kurtamohler/32/base 2025-08-26T20:08:26.0187974Z * [new branch] gh/kurtamohler/32/head -> origin/gh/kurtamohler/32/head 2025-08-26T20:08:26.0188165Z * [new branch] gh/kurtamohler/32/orig -> origin/gh/kurtamohler/32/orig 2025-08-26T20:08:26.0188348Z * [new branch] gh/kurtamohler/33/base -> origin/gh/kurtamohler/33/base 2025-08-26T20:08:26.0188506Z * [new branch] gh/kurtamohler/33/head -> origin/gh/kurtamohler/33/head 2025-08-26T20:08:26.0188672Z * [new branch] gh/kurtamohler/33/orig -> origin/gh/kurtamohler/33/orig 2025-08-26T20:08:26.0188817Z * [new branch] gh/kurtamohler/34/base -> origin/gh/kurtamohler/34/base 2025-08-26T20:08:26.0189123Z * [new branch] gh/kurtamohler/34/head -> origin/gh/kurtamohler/34/head 2025-08-26T20:08:26.0189291Z * [new branch] gh/kurtamohler/34/orig -> origin/gh/kurtamohler/34/orig 2025-08-26T20:08:26.0189694Z * [new branch] gh/kurtamohler/41/base -> origin/gh/kurtamohler/41/base 2025-08-26T20:08:26.0190722Z * [new branch] gh/kurtamohler/41/head -> origin/gh/kurtamohler/41/head 2025-08-26T20:08:26.0191380Z * [new branch] gh/kurtamohler/41/orig -> origin/gh/kurtamohler/41/orig 2025-08-26T20:08:26.0191763Z * [new branch] gh/kurtamohler/42/base -> origin/gh/kurtamohler/42/base 2025-08-26T20:08:26.0191954Z * [new branch] gh/kurtamohler/42/head -> origin/gh/kurtamohler/42/head 2025-08-26T20:08:26.0192123Z * [new branch] gh/kurtamohler/42/orig -> origin/gh/kurtamohler/42/orig 2025-08-26T20:08:26.0192301Z * [new branch] gh/kurtamohler/43/base -> origin/gh/kurtamohler/43/base 2025-08-26T20:08:26.0192536Z * [new branch] gh/kurtamohler/43/head -> origin/gh/kurtamohler/43/head 2025-08-26T20:08:26.0195634Z * [new branch] gh/kurtamohler/43/orig -> origin/gh/kurtamohler/43/orig 2025-08-26T20:08:26.0195823Z * [new branch] gh/kurtamohler/44/base -> origin/gh/kurtamohler/44/base 2025-08-26T20:08:26.0195992Z * [new branch] gh/kurtamohler/44/head -> origin/gh/kurtamohler/44/head 2025-08-26T20:08:26.0196481Z * [new branch] gh/kurtamohler/44/orig -> origin/gh/kurtamohler/44/orig 2025-08-26T20:08:26.0196654Z * [new branch] gh/kurtamohler/45/base -> origin/gh/kurtamohler/45/base 2025-08-26T20:08:26.0196815Z * [new branch] gh/kurtamohler/45/head -> origin/gh/kurtamohler/45/head 2025-08-26T20:08:26.0196974Z * [new branch] gh/kurtamohler/45/orig -> origin/gh/kurtamohler/45/orig 2025-08-26T20:08:26.0197138Z * [new branch] gh/kurtamohler/46/base -> origin/gh/kurtamohler/46/base 2025-08-26T20:08:26.0197304Z * [new branch] gh/kurtamohler/46/head -> origin/gh/kurtamohler/46/head 2025-08-26T20:08:26.0198093Z * [new branch] gh/kurtamohler/46/orig -> origin/gh/kurtamohler/46/orig 2025-08-26T20:08:26.0198637Z * [new branch] gh/kurtamohler/47/base -> origin/gh/kurtamohler/47/base 2025-08-26T20:08:26.0199767Z * [new branch] gh/kurtamohler/47/head -> origin/gh/kurtamohler/47/head 2025-08-26T20:08:26.0200169Z * [new branch] gh/kurtamohler/47/orig -> origin/gh/kurtamohler/47/orig 2025-08-26T20:08:26.0204427Z * [new branch] gh/kurtamohler/48/base -> origin/gh/kurtamohler/48/base 2025-08-26T20:08:26.0204661Z * [new branch] gh/kurtamohler/48/head -> origin/gh/kurtamohler/48/head 2025-08-26T20:08:26.0204898Z * [new branch] gh/kurtamohler/48/orig -> origin/gh/kurtamohler/48/orig 2025-08-26T20:08:26.0205138Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-08-26T20:08:26.0205378Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-08-26T20:08:26.0212097Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-08-26T20:08:26.0212313Z * [new branch] gh/kwen2501/142/base -> origin/gh/kwen2501/142/base 2025-08-26T20:08:26.0212477Z * [new branch] gh/kwen2501/142/head -> origin/gh/kwen2501/142/head 2025-08-26T20:08:26.0212664Z * [new branch] gh/kwen2501/142/orig -> origin/gh/kwen2501/142/orig 2025-08-26T20:08:26.0212820Z * [new branch] gh/kwen2501/15/base -> origin/gh/kwen2501/15/base 2025-08-26T20:08:26.0212972Z * [new branch] gh/kwen2501/15/head -> origin/gh/kwen2501/15/head 2025-08-26T20:08:26.0213348Z * [new branch] gh/kwen2501/156/base -> origin/gh/kwen2501/156/base 2025-08-26T20:08:26.0213500Z * [new branch] gh/kwen2501/156/head -> origin/gh/kwen2501/156/head 2025-08-26T20:08:26.0217967Z * [new branch] gh/kwen2501/156/orig -> origin/gh/kwen2501/156/orig 2025-08-26T20:08:26.0218147Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-08-26T20:08:26.0218286Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-08-26T20:08:26.0218437Z * [new branch] gh/kwen2501/186/base -> origin/gh/kwen2501/186/base 2025-08-26T20:08:26.0218584Z * [new branch] gh/kwen2501/186/head -> origin/gh/kwen2501/186/head 2025-08-26T20:08:26.0218717Z * [new branch] gh/kwen2501/186/orig -> origin/gh/kwen2501/186/orig 2025-08-26T20:08:26.0218854Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-08-26T20:08:26.0219567Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-08-26T20:08:26.0219727Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-08-26T20:08:26.0220073Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-08-26T20:08:26.0220228Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-08-26T20:08:26.0220362Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-08-26T20:08:26.0220662Z * [new branch] gh/kwen2501/194/base -> origin/gh/kwen2501/194/base 2025-08-26T20:08:26.0220808Z * [new branch] gh/kwen2501/194/head -> origin/gh/kwen2501/194/head 2025-08-26T20:08:26.0220941Z * [new branch] gh/kwen2501/194/orig -> origin/gh/kwen2501/194/orig 2025-08-26T20:08:26.0221221Z * [new branch] gh/kwen2501/199/base -> origin/gh/kwen2501/199/base 2025-08-26T20:08:26.0221708Z * [new branch] gh/kwen2501/199/head -> origin/gh/kwen2501/199/head 2025-08-26T20:08:26.0222645Z * [new branch] gh/kwen2501/199/orig -> origin/gh/kwen2501/199/orig 2025-08-26T20:08:26.0223241Z * [new branch] gh/kwen2501/200/base -> origin/gh/kwen2501/200/base 2025-08-26T20:08:26.0224556Z * [new branch] gh/kwen2501/200/head -> origin/gh/kwen2501/200/head 2025-08-26T20:08:26.0224706Z * [new branch] gh/kwen2501/200/orig -> origin/gh/kwen2501/200/orig 2025-08-26T20:08:26.0226174Z * [new branch] gh/kwen2501/201/base -> origin/gh/kwen2501/201/base 2025-08-26T20:08:26.0226865Z * [new branch] gh/kwen2501/201/head -> origin/gh/kwen2501/201/head 2025-08-26T20:08:26.0227268Z * [new branch] gh/kwen2501/201/orig -> origin/gh/kwen2501/201/orig 2025-08-26T20:08:26.0228385Z * [new branch] gh/kwen2501/202/base -> origin/gh/kwen2501/202/base 2025-08-26T20:08:26.0228561Z * [new branch] gh/kwen2501/202/head -> origin/gh/kwen2501/202/head 2025-08-26T20:08:26.0229692Z * [new branch] gh/kwen2501/202/orig -> origin/gh/kwen2501/202/orig 2025-08-26T20:08:26.0230679Z * [new branch] gh/kwen2501/203/base -> origin/gh/kwen2501/203/base 2025-08-26T20:08:26.0231305Z * [new branch] gh/kwen2501/203/head -> origin/gh/kwen2501/203/head 2025-08-26T20:08:26.0232008Z * [new branch] gh/kwen2501/203/orig -> origin/gh/kwen2501/203/orig 2025-08-26T20:08:26.0235176Z * [new branch] gh/kwen2501/204/base -> origin/gh/kwen2501/204/base 2025-08-26T20:08:26.0235369Z * [new branch] gh/kwen2501/204/head -> origin/gh/kwen2501/204/head 2025-08-26T20:08:26.0235534Z * [new branch] gh/kwen2501/204/orig -> origin/gh/kwen2501/204/orig 2025-08-26T20:08:26.0235880Z * [new branch] gh/kwen2501/205/base -> origin/gh/kwen2501/205/base 2025-08-26T20:08:26.0236033Z * [new branch] gh/kwen2501/205/head -> origin/gh/kwen2501/205/head 2025-08-26T20:08:26.0236214Z * [new branch] gh/kwen2501/205/orig -> origin/gh/kwen2501/205/orig 2025-08-26T20:08:26.0237729Z * [new branch] gh/kwen2501/206/base -> origin/gh/kwen2501/206/base 2025-08-26T20:08:26.0238728Z * [new branch] gh/kwen2501/206/head -> origin/gh/kwen2501/206/head 2025-08-26T20:08:26.0242825Z * [new branch] gh/kwen2501/206/orig -> origin/gh/kwen2501/206/orig 2025-08-26T20:08:26.0243199Z * [new branch] gh/kwen2501/207/base -> origin/gh/kwen2501/207/base 2025-08-26T20:08:26.0243454Z * [new branch] gh/kwen2501/207/head -> origin/gh/kwen2501/207/head 2025-08-26T20:08:26.0243643Z * [new branch] gh/kwen2501/207/orig -> origin/gh/kwen2501/207/orig 2025-08-26T20:08:26.0248305Z * [new branch] gh/kwen2501/208/base -> origin/gh/kwen2501/208/base 2025-08-26T20:08:26.0248686Z * [new branch] gh/kwen2501/208/head -> origin/gh/kwen2501/208/head 2025-08-26T20:08:26.0248943Z * [new branch] gh/kwen2501/208/orig -> origin/gh/kwen2501/208/orig 2025-08-26T20:08:26.0249126Z * [new branch] gh/kwen2501/209/base -> origin/gh/kwen2501/209/base 2025-08-26T20:08:26.0249284Z * [new branch] gh/kwen2501/209/head -> origin/gh/kwen2501/209/head 2025-08-26T20:08:26.0249780Z * [new branch] gh/kwen2501/209/orig -> origin/gh/kwen2501/209/orig 2025-08-26T20:08:26.0254444Z * [new branch] gh/kwen2501/210/base -> origin/gh/kwen2501/210/base 2025-08-26T20:08:26.0254805Z * [new branch] gh/kwen2501/210/head -> origin/gh/kwen2501/210/head 2025-08-26T20:08:26.0255070Z * [new branch] gh/kwen2501/210/orig -> origin/gh/kwen2501/210/orig 2025-08-26T20:08:26.0255344Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-08-26T20:08:26.0255562Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-08-26T20:08:26.0255765Z * [new branch] gh/kwen2501/212/base -> origin/gh/kwen2501/212/base 2025-08-26T20:08:26.0256457Z * [new branch] gh/kwen2501/212/head -> origin/gh/kwen2501/212/head 2025-08-26T20:08:26.0256659Z * [new branch] gh/kwen2501/212/orig -> origin/gh/kwen2501/212/orig 2025-08-26T20:08:26.0256830Z * [new branch] gh/kwen2501/213/base -> origin/gh/kwen2501/213/base 2025-08-26T20:08:26.0256983Z * [new branch] gh/kwen2501/213/head -> origin/gh/kwen2501/213/head 2025-08-26T20:08:26.0257137Z * [new branch] gh/kwen2501/213/orig -> origin/gh/kwen2501/213/orig 2025-08-26T20:08:26.0257288Z * [new branch] gh/kwen2501/214/base -> origin/gh/kwen2501/214/base 2025-08-26T20:08:26.0257456Z * [new branch] gh/kwen2501/214/head -> origin/gh/kwen2501/214/head 2025-08-26T20:08:26.0257598Z * [new branch] gh/kwen2501/214/orig -> origin/gh/kwen2501/214/orig 2025-08-26T20:08:26.0257745Z * [new branch] gh/kwen2501/215/base -> origin/gh/kwen2501/215/base 2025-08-26T20:08:26.0257890Z * [new branch] gh/kwen2501/215/head -> origin/gh/kwen2501/215/head 2025-08-26T20:08:26.0264493Z * [new branch] gh/kwen2501/215/orig -> origin/gh/kwen2501/215/orig 2025-08-26T20:08:26.0265062Z * [new branch] gh/kwen2501/216/base -> origin/gh/kwen2501/216/base 2025-08-26T20:08:26.0265231Z * [new branch] gh/kwen2501/216/head -> origin/gh/kwen2501/216/head 2025-08-26T20:08:26.0265380Z * [new branch] gh/kwen2501/216/orig -> origin/gh/kwen2501/216/orig 2025-08-26T20:08:26.0265710Z * [new branch] gh/kwen2501/217/base -> origin/gh/kwen2501/217/base 2025-08-26T20:08:26.0265858Z * [new branch] gh/kwen2501/217/head -> origin/gh/kwen2501/217/head 2025-08-26T20:08:26.0266008Z * [new branch] gh/kwen2501/217/orig -> origin/gh/kwen2501/217/orig 2025-08-26T20:08:26.0266648Z * [new branch] gh/kwen2501/218/base -> origin/gh/kwen2501/218/base 2025-08-26T20:08:26.0267243Z * [new branch] gh/kwen2501/218/head -> origin/gh/kwen2501/218/head 2025-08-26T20:08:26.0267439Z * [new branch] gh/kwen2501/218/orig -> origin/gh/kwen2501/218/orig 2025-08-26T20:08:26.0267604Z * [new branch] gh/laithsakka/156/base -> origin/gh/laithsakka/156/base 2025-08-26T20:08:26.0267772Z * [new branch] gh/laithsakka/156/head -> origin/gh/laithsakka/156/head 2025-08-26T20:08:26.0267930Z * [new branch] gh/laithsakka/156/orig -> origin/gh/laithsakka/156/orig 2025-08-26T20:08:26.0268102Z * [new branch] gh/laithsakka/160/base -> origin/gh/laithsakka/160/base 2025-08-26T20:08:26.0273155Z * [new branch] gh/laithsakka/160/head -> origin/gh/laithsakka/160/head 2025-08-26T20:08:26.0273344Z * [new branch] gh/laithsakka/160/orig -> origin/gh/laithsakka/160/orig 2025-08-26T20:08:26.0273509Z * [new branch] gh/laithsakka/178/base -> origin/gh/laithsakka/178/base 2025-08-26T20:08:26.0273664Z * [new branch] gh/laithsakka/178/head -> origin/gh/laithsakka/178/head 2025-08-26T20:08:26.0273998Z * [new branch] gh/laithsakka/178/orig -> origin/gh/laithsakka/178/orig 2025-08-26T20:08:26.0274163Z * [new branch] gh/laithsakka/191/base -> origin/gh/laithsakka/191/base 2025-08-26T20:08:26.0274437Z * [new branch] gh/laithsakka/191/head -> origin/gh/laithsakka/191/head 2025-08-26T20:08:26.0274605Z * [new branch] gh/laithsakka/191/orig -> origin/gh/laithsakka/191/orig 2025-08-26T20:08:26.0274807Z * [new branch] gh/laithsakka/237/base -> origin/gh/laithsakka/237/base 2025-08-26T20:08:26.0274965Z * [new branch] gh/laithsakka/237/head -> origin/gh/laithsakka/237/head 2025-08-26T20:08:26.0275133Z * [new branch] gh/laithsakka/237/orig -> origin/gh/laithsakka/237/orig 2025-08-26T20:08:26.0275444Z * [new branch] gh/laithsakka/238/base -> origin/gh/laithsakka/238/base 2025-08-26T20:08:26.0276282Z * [new branch] gh/laithsakka/238/head -> origin/gh/laithsakka/238/head 2025-08-26T20:08:26.0276849Z * [new branch] gh/laithsakka/238/orig -> origin/gh/laithsakka/238/orig 2025-08-26T20:08:26.0278053Z * [new branch] gh/laithsakka/248/base -> origin/gh/laithsakka/248/base 2025-08-26T20:08:26.0278667Z * [new branch] gh/laithsakka/248/head -> origin/gh/laithsakka/248/head 2025-08-26T20:08:26.0279223Z * [new branch] gh/laithsakka/248/orig -> origin/gh/laithsakka/248/orig 2025-08-26T20:08:26.0280392Z * [new branch] gh/laithsakka/249/base -> origin/gh/laithsakka/249/base 2025-08-26T20:08:26.0281423Z * [new branch] gh/laithsakka/249/head -> origin/gh/laithsakka/249/head 2025-08-26T20:08:26.0281777Z * [new branch] gh/laithsakka/249/orig -> origin/gh/laithsakka/249/orig 2025-08-26T20:08:26.0283137Z * [new branch] gh/laithsakka/250/base -> origin/gh/laithsakka/250/base 2025-08-26T20:08:26.0284515Z * [new branch] gh/laithsakka/250/head -> origin/gh/laithsakka/250/head 2025-08-26T20:08:26.0284775Z * [new branch] gh/laithsakka/250/orig -> origin/gh/laithsakka/250/orig 2025-08-26T20:08:26.0285269Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-08-26T20:08:26.0285741Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-08-26T20:08:26.0286893Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-08-26T20:08:26.0287353Z * [new branch] gh/laithsakka/252/base -> origin/gh/laithsakka/252/base 2025-08-26T20:08:26.0288320Z * [new branch] gh/laithsakka/252/head -> origin/gh/laithsakka/252/head 2025-08-26T20:08:26.0288742Z * [new branch] gh/laithsakka/252/orig -> origin/gh/laithsakka/252/orig 2025-08-26T20:08:26.0292427Z * [new branch] gh/laithsakka/253/base -> origin/gh/laithsakka/253/base 2025-08-26T20:08:26.0292647Z * [new branch] gh/laithsakka/253/head -> origin/gh/laithsakka/253/head 2025-08-26T20:08:26.0293230Z * [new branch] gh/laithsakka/253/orig -> origin/gh/laithsakka/253/orig 2025-08-26T20:08:26.0300672Z * [new branch] gh/laithsakka/254/base -> origin/gh/laithsakka/254/base 2025-08-26T20:08:26.0305583Z * [new branch] gh/laithsakka/254/head -> origin/gh/laithsakka/254/head 2025-08-26T20:08:26.0308287Z * [new branch] gh/laithsakka/254/orig -> origin/gh/laithsakka/254/orig 2025-08-26T20:08:26.0308487Z * [new branch] gh/laithsakka/255/base -> origin/gh/laithsakka/255/base 2025-08-26T20:08:26.0308656Z * [new branch] gh/laithsakka/255/head -> origin/gh/laithsakka/255/head 2025-08-26T20:08:26.0308804Z * [new branch] gh/laithsakka/255/orig -> origin/gh/laithsakka/255/orig 2025-08-26T20:08:26.0309192Z * [new branch] gh/laithsakka/256/base -> origin/gh/laithsakka/256/base 2025-08-26T20:08:26.0309351Z * [new branch] gh/laithsakka/256/head -> origin/gh/laithsakka/256/head 2025-08-26T20:08:26.0309509Z * [new branch] gh/laithsakka/256/orig -> origin/gh/laithsakka/256/orig 2025-08-26T20:08:26.0309656Z * [new branch] gh/laithsakka/257/base -> origin/gh/laithsakka/257/base 2025-08-26T20:08:26.0309823Z * [new branch] gh/laithsakka/257/head -> origin/gh/laithsakka/257/head 2025-08-26T20:08:26.0309983Z * [new branch] gh/laithsakka/257/orig -> origin/gh/laithsakka/257/orig 2025-08-26T20:08:26.0310134Z * [new branch] gh/laithsakka/258/base -> origin/gh/laithsakka/258/base 2025-08-26T20:08:26.0310288Z * [new branch] gh/laithsakka/258/head -> origin/gh/laithsakka/258/head 2025-08-26T20:08:26.0310436Z * [new branch] gh/laithsakka/258/orig -> origin/gh/laithsakka/258/orig 2025-08-26T20:08:26.0310593Z * [new branch] gh/laithsakka/259/base -> origin/gh/laithsakka/259/base 2025-08-26T20:08:26.0310741Z * [new branch] gh/laithsakka/259/head -> origin/gh/laithsakka/259/head 2025-08-26T20:08:26.0310888Z * [new branch] gh/laithsakka/259/orig -> origin/gh/laithsakka/259/orig 2025-08-26T20:08:26.0312438Z * [new branch] gh/laithsakka/260/base -> origin/gh/laithsakka/260/base 2025-08-26T20:08:26.0312754Z * [new branch] gh/laithsakka/260/head -> origin/gh/laithsakka/260/head 2025-08-26T20:08:26.0312931Z * [new branch] gh/laithsakka/260/orig -> origin/gh/laithsakka/260/orig 2025-08-26T20:08:26.0313098Z * [new branch] gh/laithsakka/261/base -> origin/gh/laithsakka/261/base 2025-08-26T20:08:26.0313253Z * [new branch] gh/laithsakka/261/head -> origin/gh/laithsakka/261/head 2025-08-26T20:08:26.0313419Z * [new branch] gh/laithsakka/261/orig -> origin/gh/laithsakka/261/orig 2025-08-26T20:08:26.0313602Z * [new branch] gh/laithsakka/262/base -> origin/gh/laithsakka/262/base 2025-08-26T20:08:26.0313765Z * [new branch] gh/laithsakka/262/head -> origin/gh/laithsakka/262/head 2025-08-26T20:08:26.0313928Z * [new branch] gh/laithsakka/262/orig -> origin/gh/laithsakka/262/orig 2025-08-26T20:08:26.0318528Z * [new branch] gh/laithsakka/263/base -> origin/gh/laithsakka/263/base 2025-08-26T20:08:26.0318939Z * [new branch] gh/laithsakka/263/head -> origin/gh/laithsakka/263/head 2025-08-26T20:08:26.0319108Z * [new branch] gh/laithsakka/263/orig -> origin/gh/laithsakka/263/orig 2025-08-26T20:08:26.0319510Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-08-26T20:08:26.0319667Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-08-26T20:08:26.0319889Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-08-26T20:08:26.0320535Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-08-26T20:08:26.0321383Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-08-26T20:08:26.0321888Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-08-26T20:08:26.0323343Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-08-26T20:08:26.0324005Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-08-26T20:08:26.0326610Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-08-26T20:08:26.0327142Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-08-26T20:08:26.0330491Z * [new branch] gh/lucaskabela/10/base -> origin/gh/lucaskabela/10/base 2025-08-26T20:08:26.0331114Z * [new branch] gh/lucaskabela/10/head -> origin/gh/lucaskabela/10/head 2025-08-26T20:08:26.0331300Z * [new branch] gh/lucaskabela/10/orig -> origin/gh/lucaskabela/10/orig 2025-08-26T20:08:26.0331462Z * [new branch] gh/lucaskabela/11/base -> origin/gh/lucaskabela/11/base 2025-08-26T20:08:26.0331729Z * [new branch] gh/lucaskabela/11/head -> origin/gh/lucaskabela/11/head 2025-08-26T20:08:26.0332068Z * [new branch] gh/lucaskabela/11/orig -> origin/gh/lucaskabela/11/orig 2025-08-26T20:08:26.0333166Z * [new branch] gh/lucaskabela/12/base -> origin/gh/lucaskabela/12/base 2025-08-26T20:08:26.0333624Z * [new branch] gh/lucaskabela/12/head -> origin/gh/lucaskabela/12/head 2025-08-26T20:08:26.0334108Z * [new branch] gh/lucaskabela/12/orig -> origin/gh/lucaskabela/12/orig 2025-08-26T20:08:26.0334368Z * [new branch] gh/lucaskabela/13/base -> origin/gh/lucaskabela/13/base 2025-08-26T20:08:26.0343098Z * [new branch] gh/lucaskabela/13/head -> origin/gh/lucaskabela/13/head 2025-08-26T20:08:26.0343351Z * [new branch] gh/lucaskabela/13/orig -> origin/gh/lucaskabela/13/orig 2025-08-26T20:08:26.0343531Z * [new branch] gh/lucaskabela/14/base -> origin/gh/lucaskabela/14/base 2025-08-26T20:08:26.0343701Z * [new branch] gh/lucaskabela/14/head -> origin/gh/lucaskabela/14/head 2025-08-26T20:08:26.0343891Z * [new branch] gh/lucaskabela/14/orig -> origin/gh/lucaskabela/14/orig 2025-08-26T20:08:26.0344056Z * [new branch] gh/lucaskabela/15/base -> origin/gh/lucaskabela/15/base 2025-08-26T20:08:26.0344221Z * [new branch] gh/lucaskabela/15/head -> origin/gh/lucaskabela/15/head 2025-08-26T20:08:26.0344384Z * [new branch] gh/lucaskabela/15/orig -> origin/gh/lucaskabela/15/orig 2025-08-26T20:08:26.0344550Z * [new branch] gh/lucaskabela/16/base -> origin/gh/lucaskabela/16/base 2025-08-26T20:08:26.0344721Z * [new branch] gh/lucaskabela/16/head -> origin/gh/lucaskabela/16/head 2025-08-26T20:08:26.0344883Z * [new branch] gh/lucaskabela/16/orig -> origin/gh/lucaskabela/16/orig 2025-08-26T20:08:26.0345050Z * [new branch] gh/lucaskabela/17/base -> origin/gh/lucaskabela/17/base 2025-08-26T20:08:26.0345388Z * [new branch] gh/lucaskabela/17/head -> origin/gh/lucaskabela/17/head 2025-08-26T20:08:26.0345587Z * [new branch] gh/lucaskabela/17/orig -> origin/gh/lucaskabela/17/orig 2025-08-26T20:08:26.0352012Z * [new branch] gh/lucaskabela/2/base -> origin/gh/lucaskabela/2/base 2025-08-26T20:08:26.0352209Z * [new branch] gh/lucaskabela/2/head -> origin/gh/lucaskabela/2/head 2025-08-26T20:08:26.0352386Z * [new branch] gh/lucaskabela/2/orig -> origin/gh/lucaskabela/2/orig 2025-08-26T20:08:26.0352583Z * [new branch] gh/lucaskabela/3/base -> origin/gh/lucaskabela/3/base 2025-08-26T20:08:26.0352748Z * [new branch] gh/lucaskabela/3/head -> origin/gh/lucaskabela/3/head 2025-08-26T20:08:26.0352906Z * [new branch] gh/lucaskabela/3/orig -> origin/gh/lucaskabela/3/orig 2025-08-26T20:08:26.0353071Z * [new branch] gh/lucaskabela/4/base -> origin/gh/lucaskabela/4/base 2025-08-26T20:08:26.0353231Z * [new branch] gh/lucaskabela/4/head -> origin/gh/lucaskabela/4/head 2025-08-26T20:08:26.0353387Z * [new branch] gh/lucaskabela/4/orig -> origin/gh/lucaskabela/4/orig 2025-08-26T20:08:26.0353597Z * [new branch] gh/lucaskabela/5/base -> origin/gh/lucaskabela/5/base 2025-08-26T20:08:26.0353754Z * [new branch] gh/lucaskabela/5/head -> origin/gh/lucaskabela/5/head 2025-08-26T20:08:26.0354080Z * [new branch] gh/lucaskabela/5/orig -> origin/gh/lucaskabela/5/orig 2025-08-26T20:08:26.0354253Z * [new branch] gh/lucaskabela/6/base -> origin/gh/lucaskabela/6/base 2025-08-26T20:08:26.0354420Z * [new branch] gh/lucaskabela/6/head -> origin/gh/lucaskabela/6/head 2025-08-26T20:08:26.0354922Z * [new branch] gh/lucaskabela/6/orig -> origin/gh/lucaskabela/6/orig 2025-08-26T20:08:26.0355589Z * [new branch] gh/lucaskabela/7/base -> origin/gh/lucaskabela/7/base 2025-08-26T20:08:26.0356119Z * [new branch] gh/lucaskabela/7/head -> origin/gh/lucaskabela/7/head 2025-08-26T20:08:26.0357348Z * [new branch] gh/lucaskabela/7/orig -> origin/gh/lucaskabela/7/orig 2025-08-26T20:08:26.0357887Z * [new branch] gh/lucaskabela/8/base -> origin/gh/lucaskabela/8/base 2025-08-26T20:08:26.0358854Z * [new branch] gh/lucaskabela/8/head -> origin/gh/lucaskabela/8/head 2025-08-26T20:08:26.0359591Z * [new branch] gh/lucaskabela/8/orig -> origin/gh/lucaskabela/8/orig 2025-08-26T20:08:26.0360935Z * [new branch] gh/lucaskabela/9/base -> origin/gh/lucaskabela/9/base 2025-08-26T20:08:26.0361319Z * [new branch] gh/lucaskabela/9/head -> origin/gh/lucaskabela/9/head 2025-08-26T20:08:26.0363743Z * [new branch] gh/lucaskabela/9/orig -> origin/gh/lucaskabela/9/orig 2025-08-26T20:08:26.0364251Z * [new branch] gh/lw/1/base -> origin/gh/lw/1/base 2025-08-26T20:08:26.0364409Z * [new branch] gh/lw/1/head -> origin/gh/lw/1/head 2025-08-26T20:08:26.0364591Z * [new branch] gh/lw/1/orig -> origin/gh/lw/1/orig 2025-08-26T20:08:26.0366281Z * [new branch] gh/lw/2/base -> origin/gh/lw/2/base 2025-08-26T20:08:26.0366449Z * [new branch] gh/lw/2/head -> origin/gh/lw/2/head 2025-08-26T20:08:26.0367974Z * [new branch] gh/lw/2/orig -> origin/gh/lw/2/orig 2025-08-26T20:08:26.0368500Z * [new branch] gh/lw/3/base -> origin/gh/lw/3/base 2025-08-26T20:08:26.0368664Z * [new branch] gh/lw/3/head -> origin/gh/lw/3/head 2025-08-26T20:08:26.0373429Z * [new branch] gh/lw/3/orig -> origin/gh/lw/3/orig 2025-08-26T20:08:26.0373803Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-08-26T20:08:26.0373971Z * [new branch] gh/malfet/330/base -> origin/gh/malfet/330/base 2025-08-26T20:08:26.0374120Z * [new branch] gh/malfet/330/head -> origin/gh/malfet/330/head 2025-08-26T20:08:26.0374268Z * [new branch] gh/malfet/330/orig -> origin/gh/malfet/330/orig 2025-08-26T20:08:26.0374410Z * [new branch] gh/malfet/396/base -> origin/gh/malfet/396/base 2025-08-26T20:08:26.0377561Z * [new branch] gh/malfet/396/head -> origin/gh/malfet/396/head 2025-08-26T20:08:26.0378113Z * [new branch] gh/malfet/396/orig -> origin/gh/malfet/396/orig 2025-08-26T20:08:26.0378293Z * [new branch] gh/malfet/397/base -> origin/gh/malfet/397/base 2025-08-26T20:08:26.0378448Z * [new branch] gh/malfet/397/head -> origin/gh/malfet/397/head 2025-08-26T20:08:26.0378647Z * [new branch] gh/malfet/397/orig -> origin/gh/malfet/397/orig 2025-08-26T20:08:26.0378795Z * [new branch] gh/malfet/398/base -> origin/gh/malfet/398/base 2025-08-26T20:08:26.0382477Z * [new branch] gh/malfet/398/head -> origin/gh/malfet/398/head 2025-08-26T20:08:26.0383118Z * [new branch] gh/malfet/398/orig -> origin/gh/malfet/398/orig 2025-08-26T20:08:26.0383303Z * [new branch] gh/malfet/399/base -> origin/gh/malfet/399/base 2025-08-26T20:08:26.0383616Z * [new branch] gh/malfet/399/head -> origin/gh/malfet/399/head 2025-08-26T20:08:26.0383781Z * [new branch] gh/malfet/399/orig -> origin/gh/malfet/399/orig 2025-08-26T20:08:26.0383919Z * [new branch] gh/malfet/414/base -> origin/gh/malfet/414/base 2025-08-26T20:08:26.0384065Z * [new branch] gh/malfet/414/head -> origin/gh/malfet/414/head 2025-08-26T20:08:26.0387729Z * [new branch] gh/malfet/414/orig -> origin/gh/malfet/414/orig 2025-08-26T20:08:26.0387905Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-08-26T20:08:26.0388053Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-08-26T20:08:26.0388197Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-08-26T20:08:26.0388345Z * [new branch] gh/malfet/418/base -> origin/gh/malfet/418/base 2025-08-26T20:08:26.0388490Z * [new branch] gh/malfet/418/head -> origin/gh/malfet/418/head 2025-08-26T20:08:26.0388641Z * [new branch] gh/malfet/418/orig -> origin/gh/malfet/418/orig 2025-08-26T20:08:26.0395004Z * [new branch] gh/malfet/456/base -> origin/gh/malfet/456/base 2025-08-26T20:08:26.0395197Z * [new branch] gh/malfet/456/head -> origin/gh/malfet/456/head 2025-08-26T20:08:26.0395541Z * [new branch] gh/malfet/456/orig -> origin/gh/malfet/456/orig 2025-08-26T20:08:26.0395710Z * [new branch] gh/malfet/457/base -> origin/gh/malfet/457/base 2025-08-26T20:08:26.0395855Z * [new branch] gh/malfet/457/head -> origin/gh/malfet/457/head 2025-08-26T20:08:26.0396004Z * [new branch] gh/malfet/457/orig -> origin/gh/malfet/457/orig 2025-08-26T20:08:26.0396142Z * [new branch] gh/malfet/459/base -> origin/gh/malfet/459/base 2025-08-26T20:08:26.0396483Z * [new branch] gh/malfet/459/head -> origin/gh/malfet/459/head 2025-08-26T20:08:26.0396643Z * [new branch] gh/malfet/459/orig -> origin/gh/malfet/459/orig 2025-08-26T20:08:26.0396784Z * [new branch] gh/malfet/460/base -> origin/gh/malfet/460/base 2025-08-26T20:08:26.0396932Z * [new branch] gh/malfet/460/head -> origin/gh/malfet/460/head 2025-08-26T20:08:26.0397231Z * [new branch] gh/malfet/460/orig -> origin/gh/malfet/460/orig 2025-08-26T20:08:26.0398190Z * [new branch] gh/malfet/461/base -> origin/gh/malfet/461/base 2025-08-26T20:08:26.0398674Z * [new branch] gh/malfet/461/head -> origin/gh/malfet/461/head 2025-08-26T20:08:26.0398851Z * [new branch] gh/malfet/461/orig -> origin/gh/malfet/461/orig 2025-08-26T20:08:26.0399783Z * [new branch] gh/malfet/462/base -> origin/gh/malfet/462/base 2025-08-26T20:08:26.0399973Z * [new branch] gh/malfet/462/head -> origin/gh/malfet/462/head 2025-08-26T20:08:26.0403288Z * [new branch] gh/malfet/462/orig -> origin/gh/malfet/462/orig 2025-08-26T20:08:26.0403489Z * [new branch] gh/malfet/463/base -> origin/gh/malfet/463/base 2025-08-26T20:08:26.0403641Z * [new branch] gh/malfet/463/head -> origin/gh/malfet/463/head 2025-08-26T20:08:26.0403806Z * [new branch] gh/malfet/463/orig -> origin/gh/malfet/463/orig 2025-08-26T20:08:26.0404456Z * [new branch] gh/malfet/464/base -> origin/gh/malfet/464/base 2025-08-26T20:08:26.0406401Z * [new branch] gh/malfet/464/head -> origin/gh/malfet/464/head 2025-08-26T20:08:26.0406560Z * [new branch] gh/malfet/464/orig -> origin/gh/malfet/464/orig 2025-08-26T20:08:26.0409851Z * [new branch] gh/malfet/465/base -> origin/gh/malfet/465/base 2025-08-26T20:08:26.0410202Z * [new branch] gh/malfet/465/head -> origin/gh/malfet/465/head 2025-08-26T20:08:26.0410354Z * [new branch] gh/malfet/465/orig -> origin/gh/malfet/465/orig 2025-08-26T20:08:26.0410495Z * [new branch] gh/malfet/466/base -> origin/gh/malfet/466/base 2025-08-26T20:08:26.0410646Z * [new branch] gh/malfet/466/head -> origin/gh/malfet/466/head 2025-08-26T20:08:26.0413671Z * [new branch] gh/malfet/466/orig -> origin/gh/malfet/466/orig 2025-08-26T20:08:26.0414197Z * [new branch] gh/malfet/467/base -> origin/gh/malfet/467/base 2025-08-26T20:08:26.0414371Z * [new branch] gh/malfet/467/head -> origin/gh/malfet/467/head 2025-08-26T20:08:26.0414518Z * [new branch] gh/malfet/467/orig -> origin/gh/malfet/467/orig 2025-08-26T20:08:26.0414665Z * [new branch] gh/malfet/468/base -> origin/gh/malfet/468/base 2025-08-26T20:08:26.0417407Z * [new branch] gh/malfet/468/head -> origin/gh/malfet/468/head 2025-08-26T20:08:26.0417561Z * [new branch] gh/malfet/468/orig -> origin/gh/malfet/468/orig 2025-08-26T20:08:26.0417711Z * [new branch] gh/malfet/469/base -> origin/gh/malfet/469/base 2025-08-26T20:08:26.0417919Z * [new branch] gh/malfet/469/head -> origin/gh/malfet/469/head 2025-08-26T20:08:26.0418078Z * [new branch] gh/malfet/469/orig -> origin/gh/malfet/469/orig 2025-08-26T20:08:26.0422802Z * [new branch] gh/malfet/470/base -> origin/gh/malfet/470/base 2025-08-26T20:08:26.0423131Z * [new branch] gh/malfet/470/head -> origin/gh/malfet/470/head 2025-08-26T20:08:26.0423382Z * [new branch] gh/malfet/470/orig -> origin/gh/malfet/470/orig 2025-08-26T20:08:26.0423551Z * [new branch] gh/malfet/471/base -> origin/gh/malfet/471/base 2025-08-26T20:08:26.0423835Z * [new branch] gh/malfet/471/head -> origin/gh/malfet/471/head 2025-08-26T20:08:26.0424024Z * [new branch] gh/malfet/471/orig -> origin/gh/malfet/471/orig 2025-08-26T20:08:26.0429094Z * [new branch] gh/malfet/472/base -> origin/gh/malfet/472/base 2025-08-26T20:08:26.0429278Z * [new branch] gh/malfet/472/head -> origin/gh/malfet/472/head 2025-08-26T20:08:26.0429585Z * [new branch] gh/malfet/472/orig -> origin/gh/malfet/472/orig 2025-08-26T20:08:26.0429757Z * [new branch] gh/malfet/473/base -> origin/gh/malfet/473/base 2025-08-26T20:08:26.0429915Z * [new branch] gh/malfet/473/head -> origin/gh/malfet/473/head 2025-08-26T20:08:26.0430062Z * [new branch] gh/malfet/473/orig -> origin/gh/malfet/473/orig 2025-08-26T20:08:26.0430211Z * [new branch] gh/malfet/474/base -> origin/gh/malfet/474/base 2025-08-26T20:08:26.0430370Z * [new branch] gh/malfet/474/head -> origin/gh/malfet/474/head 2025-08-26T20:08:26.0430685Z * [new branch] gh/malfet/474/orig -> origin/gh/malfet/474/orig 2025-08-26T20:08:26.0430846Z * [new branch] gh/malfet/475/base -> origin/gh/malfet/475/base 2025-08-26T20:08:26.0433071Z * [new branch] gh/malfet/475/head -> origin/gh/malfet/475/head 2025-08-26T20:08:26.0433249Z * [new branch] gh/malfet/475/orig -> origin/gh/malfet/475/orig 2025-08-26T20:08:26.0433398Z * [new branch] gh/malfet/476/base -> origin/gh/malfet/476/base 2025-08-26T20:08:26.0434633Z * [new branch] gh/malfet/476/head -> origin/gh/malfet/476/head 2025-08-26T20:08:26.0434930Z * [new branch] gh/malfet/476/orig -> origin/gh/malfet/476/orig 2025-08-26T20:08:26.0435900Z * [new branch] gh/malfet/477/base -> origin/gh/malfet/477/base 2025-08-26T20:08:26.0436395Z * [new branch] gh/malfet/477/head -> origin/gh/malfet/477/head 2025-08-26T20:08:26.0437125Z * [new branch] gh/malfet/477/orig -> origin/gh/malfet/477/orig 2025-08-26T20:08:26.0441248Z * [new branch] gh/malfet/478/base -> origin/gh/malfet/478/base 2025-08-26T20:08:26.0445991Z * [new branch] gh/malfet/478/head -> origin/gh/malfet/478/head 2025-08-26T20:08:26.0453563Z * [new branch] gh/malfet/478/orig -> origin/gh/malfet/478/orig 2025-08-26T20:08:26.0459168Z * [new branch] gh/malfet/479/base -> origin/gh/malfet/479/base 2025-08-26T20:08:26.0464758Z * [new branch] gh/malfet/479/head -> origin/gh/malfet/479/head 2025-08-26T20:08:26.0464984Z * [new branch] gh/malfet/479/orig -> origin/gh/malfet/479/orig 2025-08-26T20:08:26.0465157Z * [new branch] gh/malfet/480/base -> origin/gh/malfet/480/base 2025-08-26T20:08:26.0465313Z * [new branch] gh/malfet/480/head -> origin/gh/malfet/480/head 2025-08-26T20:08:26.0465623Z * [new branch] gh/malfet/480/orig -> origin/gh/malfet/480/orig 2025-08-26T20:08:26.0465783Z * [new branch] gh/malfet/481/base -> origin/gh/malfet/481/base 2025-08-26T20:08:26.0465940Z * [new branch] gh/malfet/481/head -> origin/gh/malfet/481/head 2025-08-26T20:08:26.0466076Z * [new branch] gh/malfet/481/orig -> origin/gh/malfet/481/orig 2025-08-26T20:08:26.0466201Z * [new branch] gh/malfet/482/base -> origin/gh/malfet/482/base 2025-08-26T20:08:26.0466337Z * [new branch] gh/malfet/482/head -> origin/gh/malfet/482/head 2025-08-26T20:08:26.0466469Z * [new branch] gh/malfet/482/orig -> origin/gh/malfet/482/orig 2025-08-26T20:08:26.0466610Z * [new branch] gh/malfet/483/base -> origin/gh/malfet/483/base 2025-08-26T20:08:26.0466752Z * [new branch] gh/malfet/483/head -> origin/gh/malfet/483/head 2025-08-26T20:08:26.0466902Z * [new branch] gh/malfet/483/orig -> origin/gh/malfet/483/orig 2025-08-26T20:08:26.0467047Z * [new branch] gh/malfet/484/base -> origin/gh/malfet/484/base 2025-08-26T20:08:26.0467252Z * [new branch] gh/malfet/484/head -> origin/gh/malfet/484/head 2025-08-26T20:08:26.0467395Z * [new branch] gh/malfet/484/orig -> origin/gh/malfet/484/orig 2025-08-26T20:08:26.0467627Z * [new branch] gh/malfet/485/base -> origin/gh/malfet/485/base 2025-08-26T20:08:26.0467771Z * [new branch] gh/malfet/485/head -> origin/gh/malfet/485/head 2025-08-26T20:08:26.0467911Z * [new branch] gh/malfet/485/orig -> origin/gh/malfet/485/orig 2025-08-26T20:08:26.0468050Z * [new branch] gh/malfet/486/base -> origin/gh/malfet/486/base 2025-08-26T20:08:26.0468193Z * [new branch] gh/malfet/486/head -> origin/gh/malfet/486/head 2025-08-26T20:08:26.0468327Z * [new branch] gh/malfet/486/orig -> origin/gh/malfet/486/orig 2025-08-26T20:08:26.0468459Z * [new branch] gh/malfet/487/base -> origin/gh/malfet/487/base 2025-08-26T20:08:26.0468604Z * [new branch] gh/malfet/487/head -> origin/gh/malfet/487/head 2025-08-26T20:08:26.0468740Z * [new branch] gh/malfet/487/orig -> origin/gh/malfet/487/orig 2025-08-26T20:08:26.0468882Z * [new branch] gh/malfet/488/base -> origin/gh/malfet/488/base 2025-08-26T20:08:26.0469016Z * [new branch] gh/malfet/488/head -> origin/gh/malfet/488/head 2025-08-26T20:08:26.0469157Z * [new branch] gh/malfet/488/orig -> origin/gh/malfet/488/orig 2025-08-26T20:08:26.0469342Z * [new branch] gh/malfet/489/base -> origin/gh/malfet/489/base 2025-08-26T20:08:26.0469477Z * [new branch] gh/malfet/489/head -> origin/gh/malfet/489/head 2025-08-26T20:08:26.0469618Z * [new branch] gh/malfet/489/orig -> origin/gh/malfet/489/orig 2025-08-26T20:08:26.0469760Z * [new branch] gh/malfet/490/base -> origin/gh/malfet/490/base 2025-08-26T20:08:26.0469903Z * [new branch] gh/malfet/490/head -> origin/gh/malfet/490/head 2025-08-26T20:08:26.0470168Z * [new branch] gh/malfet/490/orig -> origin/gh/malfet/490/orig 2025-08-26T20:08:26.0470310Z * [new branch] gh/malfet/491/base -> origin/gh/malfet/491/base 2025-08-26T20:08:26.0470441Z * [new branch] gh/malfet/491/head -> origin/gh/malfet/491/head 2025-08-26T20:08:26.0470581Z * [new branch] gh/malfet/491/orig -> origin/gh/malfet/491/orig 2025-08-26T20:08:26.0474973Z * [new branch] gh/malfet/492/base -> origin/gh/malfet/492/base 2025-08-26T20:08:26.0475162Z * [new branch] gh/malfet/492/head -> origin/gh/malfet/492/head 2025-08-26T20:08:26.0475385Z * [new branch] gh/malfet/492/orig -> origin/gh/malfet/492/orig 2025-08-26T20:08:26.0475624Z * [new branch] gh/malfet/493/base -> origin/gh/malfet/493/base 2025-08-26T20:08:26.0475849Z * [new branch] gh/malfet/493/head -> origin/gh/malfet/493/head 2025-08-26T20:08:26.0476029Z * [new branch] gh/malfet/493/orig -> origin/gh/malfet/493/orig 2025-08-26T20:08:26.0476235Z * [new branch] gh/malfet/494/base -> origin/gh/malfet/494/base 2025-08-26T20:08:26.0476414Z * [new branch] gh/malfet/494/head -> origin/gh/malfet/494/head 2025-08-26T20:08:26.0477425Z * [new branch] gh/malfet/494/orig -> origin/gh/malfet/494/orig 2025-08-26T20:08:26.0480537Z * [new branch] gh/malfet/495/base -> origin/gh/malfet/495/base 2025-08-26T20:08:26.0480737Z * [new branch] gh/malfet/495/head -> origin/gh/malfet/495/head 2025-08-26T20:08:26.0480887Z * [new branch] gh/malfet/495/orig -> origin/gh/malfet/495/orig 2025-08-26T20:08:26.0481034Z * [new branch] gh/malfet/496/base -> origin/gh/malfet/496/base 2025-08-26T20:08:26.0481523Z * [new branch] gh/malfet/496/head -> origin/gh/malfet/496/head 2025-08-26T20:08:26.0481686Z * [new branch] gh/malfet/496/orig -> origin/gh/malfet/496/orig 2025-08-26T20:08:26.0483515Z * [new branch] gh/malfet/497/base -> origin/gh/malfet/497/base 2025-08-26T20:08:26.0487036Z * [new branch] gh/malfet/497/head -> origin/gh/malfet/497/head 2025-08-26T20:08:26.0489806Z * [new branch] gh/malfet/497/orig -> origin/gh/malfet/497/orig 2025-08-26T20:08:26.0490115Z * [new branch] gh/malfet/498/base -> origin/gh/malfet/498/base 2025-08-26T20:08:26.0495135Z * [new branch] gh/malfet/498/head -> origin/gh/malfet/498/head 2025-08-26T20:08:26.0501643Z * [new branch] gh/malfet/498/orig -> origin/gh/malfet/498/orig 2025-08-26T20:08:26.0501831Z * [new branch] gh/malfet/499/base -> origin/gh/malfet/499/base 2025-08-26T20:08:26.0502275Z * [new branch] gh/malfet/499/head -> origin/gh/malfet/499/head 2025-08-26T20:08:26.0502437Z * [new branch] gh/malfet/499/orig -> origin/gh/malfet/499/orig 2025-08-26T20:08:26.0502606Z * [new branch] gh/malfet/500/base -> origin/gh/malfet/500/base 2025-08-26T20:08:26.0502742Z * [new branch] gh/malfet/500/head -> origin/gh/malfet/500/head 2025-08-26T20:08:26.0502878Z * [new branch] gh/malfet/500/orig -> origin/gh/malfet/500/orig 2025-08-26T20:08:26.0503274Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-08-26T20:08:26.0503418Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-08-26T20:08:26.0503604Z * [new branch] gh/manuelcandales/10/base -> origin/gh/manuelcandales/10/base 2025-08-26T20:08:26.0503764Z * [new branch] gh/manuelcandales/10/head -> origin/gh/manuelcandales/10/head 2025-08-26T20:08:26.0503957Z * [new branch] gh/manuelcandales/10/orig -> origin/gh/manuelcandales/10/orig 2025-08-26T20:08:26.0504114Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-08-26T20:08:26.0504271Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-08-26T20:08:26.0504435Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-08-26T20:08:26.0504611Z * [new branch] gh/manuelcandales/9/base -> origin/gh/manuelcandales/9/base 2025-08-26T20:08:26.0504776Z * [new branch] gh/manuelcandales/9/head -> origin/gh/manuelcandales/9/head 2025-08-26T20:08:26.0504931Z * [new branch] gh/manuelcandales/9/orig -> origin/gh/manuelcandales/9/orig 2025-08-26T20:08:26.0505075Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-08-26T20:08:26.0505237Z * [new branch] gh/masnesral/204/base -> origin/gh/masnesral/204/base 2025-08-26T20:08:26.0505381Z * [new branch] gh/masnesral/204/head -> origin/gh/masnesral/204/head 2025-08-26T20:08:26.0505532Z * [new branch] gh/masnesral/204/orig -> origin/gh/masnesral/204/orig 2025-08-26T20:08:26.0505684Z * [new branch] gh/masnesral/232/base -> origin/gh/masnesral/232/base 2025-08-26T20:08:26.0506258Z * [new branch] gh/masnesral/232/head -> origin/gh/masnesral/232/head 2025-08-26T20:08:26.0506419Z * [new branch] gh/masnesral/232/orig -> origin/gh/masnesral/232/orig 2025-08-26T20:08:26.0511293Z * [new branch] gh/masnesral/233/base -> origin/gh/masnesral/233/base 2025-08-26T20:08:26.0511480Z * [new branch] gh/masnesral/233/head -> origin/gh/masnesral/233/head 2025-08-26T20:08:26.0511642Z * [new branch] gh/masnesral/233/orig -> origin/gh/masnesral/233/orig 2025-08-26T20:08:26.0512027Z * [new branch] gh/masnesral/234/base -> origin/gh/masnesral/234/base 2025-08-26T20:08:26.0512181Z * [new branch] gh/masnesral/234/head -> origin/gh/masnesral/234/head 2025-08-26T20:08:26.0515966Z * [new branch] gh/masnesral/234/orig -> origin/gh/masnesral/234/orig 2025-08-26T20:08:26.0516152Z * [new branch] gh/masnesral/235/base -> origin/gh/masnesral/235/base 2025-08-26T20:08:26.0516317Z * [new branch] gh/masnesral/235/head -> origin/gh/masnesral/235/head 2025-08-26T20:08:26.0516495Z * [new branch] gh/masnesral/235/orig -> origin/gh/masnesral/235/orig 2025-08-26T20:08:26.0516656Z * [new branch] gh/masnesral/236/base -> origin/gh/masnesral/236/base 2025-08-26T20:08:26.0516821Z * [new branch] gh/masnesral/236/head -> origin/gh/masnesral/236/head 2025-08-26T20:08:26.0516993Z * [new branch] gh/masnesral/236/orig -> origin/gh/masnesral/236/orig 2025-08-26T20:08:26.0517728Z * [new branch] gh/masnesral/34/base -> origin/gh/masnesral/34/base 2025-08-26T20:08:26.0519218Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-08-26T20:08:26.0523088Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-08-26T20:08:26.0523670Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-08-26T20:08:26.0524189Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-08-26T20:08:26.0524368Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-08-26T20:08:26.0524517Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-08-26T20:08:26.0524662Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-08-26T20:08:26.0524966Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-08-26T20:08:26.0525292Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-08-26T20:08:26.0527918Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-08-26T20:08:26.0528116Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-08-26T20:08:26.0528268Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-08-26T20:08:26.0528442Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-08-26T20:08:26.0534114Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-08-26T20:08:26.0534526Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-08-26T20:08:26.0534867Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-08-26T20:08:26.0535252Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-08-26T20:08:26.0535503Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-08-26T20:08:26.0535778Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-08-26T20:08:26.0535976Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-08-26T20:08:26.0536293Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-08-26T20:08:26.0536483Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-08-26T20:08:26.0537241Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-08-26T20:08:26.0542465Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-08-26T20:08:26.0543060Z * [new branch] gh/mikaylagawarecki/317/base -> origin/gh/mikaylagawarecki/317/base 2025-08-26T20:08:26.0543375Z * [new branch] gh/mikaylagawarecki/317/head -> origin/gh/mikaylagawarecki/317/head 2025-08-26T20:08:26.0543618Z * [new branch] gh/mikaylagawarecki/317/orig -> origin/gh/mikaylagawarecki/317/orig 2025-08-26T20:08:26.0543809Z * [new branch] gh/mikaylagawarecki/320/base -> origin/gh/mikaylagawarecki/320/base 2025-08-26T20:08:26.0544591Z * [new branch] gh/mikaylagawarecki/320/head -> origin/gh/mikaylagawarecki/320/head 2025-08-26T20:08:26.0544858Z * [new branch] gh/mikaylagawarecki/320/orig -> origin/gh/mikaylagawarecki/320/orig 2025-08-26T20:08:26.0545084Z * [new branch] gh/mikaylagawarecki/329/base -> origin/gh/mikaylagawarecki/329/base 2025-08-26T20:08:26.0545351Z * [new branch] gh/mikaylagawarecki/329/head -> origin/gh/mikaylagawarecki/329/head 2025-08-26T20:08:26.0545595Z * [new branch] gh/mikaylagawarecki/329/orig -> origin/gh/mikaylagawarecki/329/orig 2025-08-26T20:08:26.0545887Z * [new branch] gh/mikaylagawarecki/330/base -> origin/gh/mikaylagawarecki/330/base 2025-08-26T20:08:26.0546081Z * [new branch] gh/mikaylagawarecki/330/head -> origin/gh/mikaylagawarecki/330/head 2025-08-26T20:08:26.0546360Z * [new branch] gh/mikaylagawarecki/330/orig -> origin/gh/mikaylagawarecki/330/orig 2025-08-26T20:08:26.0546793Z * [new branch] gh/mikaylagawarecki/331/base -> origin/gh/mikaylagawarecki/331/base 2025-08-26T20:08:26.0548317Z * [new branch] gh/mikaylagawarecki/331/head -> origin/gh/mikaylagawarecki/331/head 2025-08-26T20:08:26.0548718Z * [new branch] gh/mikaylagawarecki/331/orig -> origin/gh/mikaylagawarecki/331/orig 2025-08-26T20:08:26.0550920Z * [new branch] gh/mikaylagawarecki/332/base -> origin/gh/mikaylagawarecki/332/base 2025-08-26T20:08:26.0551331Z * [new branch] gh/mikaylagawarecki/332/head -> origin/gh/mikaylagawarecki/332/head 2025-08-26T20:08:26.0551602Z * [new branch] gh/mikaylagawarecki/332/orig -> origin/gh/mikaylagawarecki/332/orig 2025-08-26T20:08:26.0552056Z * [new branch] gh/mikaylagawarecki/333/base -> origin/gh/mikaylagawarecki/333/base 2025-08-26T20:08:26.0553189Z * [new branch] gh/mikaylagawarecki/333/head -> origin/gh/mikaylagawarecki/333/head 2025-08-26T20:08:26.0553493Z * [new branch] gh/mikaylagawarecki/333/orig -> origin/gh/mikaylagawarecki/333/orig 2025-08-26T20:08:26.0555485Z * [new branch] gh/mikaylagawarecki/334/base -> origin/gh/mikaylagawarecki/334/base 2025-08-26T20:08:26.0555705Z * [new branch] gh/mikaylagawarecki/334/head -> origin/gh/mikaylagawarecki/334/head 2025-08-26T20:08:26.0555890Z * [new branch] gh/mikaylagawarecki/334/orig -> origin/gh/mikaylagawarecki/334/orig 2025-08-26T20:08:26.0557459Z * [new branch] gh/mikaylagawarecki/335/base -> origin/gh/mikaylagawarecki/335/base 2025-08-26T20:08:26.0557692Z * [new branch] gh/mikaylagawarecki/335/head -> origin/gh/mikaylagawarecki/335/head 2025-08-26T20:08:26.0559438Z * [new branch] gh/mikaylagawarecki/335/orig -> origin/gh/mikaylagawarecki/335/orig 2025-08-26T20:08:26.0559680Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-08-26T20:08:26.0560401Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-08-26T20:08:26.0561439Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-08-26T20:08:26.0562008Z * [new branch] gh/mikaylagawarecki/337/base -> origin/gh/mikaylagawarecki/337/base 2025-08-26T20:08:26.0563061Z * [new branch] gh/mikaylagawarecki/337/head -> origin/gh/mikaylagawarecki/337/head 2025-08-26T20:08:26.0564272Z * [new branch] gh/mikaylagawarecki/337/orig -> origin/gh/mikaylagawarecki/337/orig 2025-08-26T20:08:26.0564707Z * [new branch] gh/mlazos/1/base -> origin/gh/mlazos/1/base 2025-08-26T20:08:26.0565764Z * [new branch] gh/mlazos/1/head -> origin/gh/mlazos/1/head 2025-08-26T20:08:26.0566364Z * [new branch] gh/mlazos/1/orig -> origin/gh/mlazos/1/orig 2025-08-26T20:08:26.0567837Z * [new branch] gh/mlazos/10/base -> origin/gh/mlazos/10/base 2025-08-26T20:08:26.0568382Z * [new branch] gh/mlazos/10/head -> origin/gh/mlazos/10/head 2025-08-26T20:08:26.0569163Z * [new branch] gh/mlazos/10/orig -> origin/gh/mlazos/10/orig 2025-08-26T20:08:26.0570120Z * [new branch] gh/mlazos/11/base -> origin/gh/mlazos/11/base 2025-08-26T20:08:26.0570404Z * [new branch] gh/mlazos/11/head -> origin/gh/mlazos/11/head 2025-08-26T20:08:26.0574109Z * [new branch] gh/mlazos/11/orig -> origin/gh/mlazos/11/orig 2025-08-26T20:08:26.0574702Z * [new branch] gh/mlazos/12/base -> origin/gh/mlazos/12/base 2025-08-26T20:08:26.0574884Z * [new branch] gh/mlazos/12/head -> origin/gh/mlazos/12/head 2025-08-26T20:08:26.0575034Z * [new branch] gh/mlazos/12/orig -> origin/gh/mlazos/12/orig 2025-08-26T20:08:26.0575174Z * [new branch] gh/mlazos/13/base -> origin/gh/mlazos/13/base 2025-08-26T20:08:26.0575314Z * [new branch] gh/mlazos/13/head -> origin/gh/mlazos/13/head 2025-08-26T20:08:26.0575707Z * [new branch] gh/mlazos/13/orig -> origin/gh/mlazos/13/orig 2025-08-26T20:08:26.0576856Z * [new branch] gh/mlazos/14/base -> origin/gh/mlazos/14/base 2025-08-26T20:08:26.0581608Z * [new branch] gh/mlazos/14/head -> origin/gh/mlazos/14/head 2025-08-26T20:08:26.0581882Z * [new branch] gh/mlazos/14/orig -> origin/gh/mlazos/14/orig 2025-08-26T20:08:26.0582045Z * [new branch] gh/mlazos/15/base -> origin/gh/mlazos/15/base 2025-08-26T20:08:26.0582183Z * [new branch] gh/mlazos/15/head -> origin/gh/mlazos/15/head 2025-08-26T20:08:26.0582324Z * [new branch] gh/mlazos/15/orig -> origin/gh/mlazos/15/orig 2025-08-26T20:08:26.0582459Z * [new branch] gh/mlazos/16/base -> origin/gh/mlazos/16/base 2025-08-26T20:08:26.0582593Z * [new branch] gh/mlazos/16/head -> origin/gh/mlazos/16/head 2025-08-26T20:08:26.0582786Z * [new branch] gh/mlazos/16/orig -> origin/gh/mlazos/16/orig 2025-08-26T20:08:26.0585839Z * [new branch] gh/mlazos/17/base -> origin/gh/mlazos/17/base 2025-08-26T20:08:26.0586023Z * [new branch] gh/mlazos/17/head -> origin/gh/mlazos/17/head 2025-08-26T20:08:26.0586215Z * [new branch] gh/mlazos/17/orig -> origin/gh/mlazos/17/orig 2025-08-26T20:08:26.0586405Z * [new branch] gh/mlazos/2/base -> origin/gh/mlazos/2/base 2025-08-26T20:08:26.0587434Z * [new branch] gh/mlazos/2/head -> origin/gh/mlazos/2/head 2025-08-26T20:08:26.0587686Z * [new branch] gh/mlazos/2/orig -> origin/gh/mlazos/2/orig 2025-08-26T20:08:26.0589203Z * [new branch] gh/mlazos/3/base -> origin/gh/mlazos/3/base 2025-08-26T20:08:26.0589357Z * [new branch] gh/mlazos/3/head -> origin/gh/mlazos/3/head 2025-08-26T20:08:26.0590965Z * [new branch] gh/mlazos/3/orig -> origin/gh/mlazos/3/orig 2025-08-26T20:08:26.0591132Z * [new branch] gh/mlazos/4/base -> origin/gh/mlazos/4/base 2025-08-26T20:08:26.0591972Z * [new branch] gh/mlazos/4/head -> origin/gh/mlazos/4/head 2025-08-26T20:08:26.0592399Z * [new branch] gh/mlazos/4/orig -> origin/gh/mlazos/4/orig 2025-08-26T20:08:26.0593723Z * [new branch] gh/mlazos/5/base -> origin/gh/mlazos/5/base 2025-08-26T20:08:26.0594053Z * [new branch] gh/mlazos/5/head -> origin/gh/mlazos/5/head 2025-08-26T20:08:26.0595113Z * [new branch] gh/mlazos/5/orig -> origin/gh/mlazos/5/orig 2025-08-26T20:08:26.0596132Z * [new branch] gh/mlazos/6/base -> origin/gh/mlazos/6/base 2025-08-26T20:08:26.0596515Z * [new branch] gh/mlazos/6/head -> origin/gh/mlazos/6/head 2025-08-26T20:08:26.0597740Z * [new branch] gh/mlazos/6/orig -> origin/gh/mlazos/6/orig 2025-08-26T20:08:26.0599308Z * [new branch] gh/mlazos/7/base -> origin/gh/mlazos/7/base 2025-08-26T20:08:26.0599666Z * [new branch] gh/mlazos/7/head -> origin/gh/mlazos/7/head 2025-08-26T20:08:26.0600756Z * [new branch] gh/mlazos/7/orig -> origin/gh/mlazos/7/orig 2025-08-26T20:08:26.0601291Z * [new branch] gh/mlazos/8/base -> origin/gh/mlazos/8/base 2025-08-26T20:08:26.0602276Z * [new branch] gh/mlazos/8/head -> origin/gh/mlazos/8/head 2025-08-26T20:08:26.0602622Z * [new branch] gh/mlazos/8/orig -> origin/gh/mlazos/8/orig 2025-08-26T20:08:26.0604308Z * [new branch] gh/mlazos/9/base -> origin/gh/mlazos/9/base 2025-08-26T20:08:26.0604466Z * [new branch] gh/mlazos/9/head -> origin/gh/mlazos/9/head 2025-08-26T20:08:26.0605590Z * [new branch] gh/mlazos/9/orig -> origin/gh/mlazos/9/orig 2025-08-26T20:08:26.0606842Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-08-26T20:08:26.0607233Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-08-26T20:08:26.0608884Z * [new branch] gh/muchulee8/62/base -> origin/gh/muchulee8/62/base 2025-08-26T20:08:26.0609294Z * [new branch] gh/muchulee8/62/head -> origin/gh/muchulee8/62/head 2025-08-26T20:08:26.0610479Z * [new branch] gh/muchulee8/62/orig -> origin/gh/muchulee8/62/orig 2025-08-26T20:08:26.0611189Z * [new branch] gh/muchulee8/63/base -> origin/gh/muchulee8/63/base 2025-08-26T20:08:26.0611859Z * [new branch] gh/muchulee8/63/head -> origin/gh/muchulee8/63/head 2025-08-26T20:08:26.0612421Z * [new branch] gh/muchulee8/63/orig -> origin/gh/muchulee8/63/orig 2025-08-26T20:08:26.0613800Z * [new branch] gh/muchulee8/64/base -> origin/gh/muchulee8/64/base 2025-08-26T20:08:26.0613993Z * [new branch] gh/muchulee8/64/head -> origin/gh/muchulee8/64/head 2025-08-26T20:08:26.0615119Z * [new branch] gh/muchulee8/64/orig -> origin/gh/muchulee8/64/orig 2025-08-26T20:08:26.0616192Z * [new branch] gh/muchulee8/65/base -> origin/gh/muchulee8/65/base 2025-08-26T20:08:26.0616635Z * [new branch] gh/muchulee8/65/head -> origin/gh/muchulee8/65/head 2025-08-26T20:08:26.0617723Z * [new branch] gh/muchulee8/65/orig -> origin/gh/muchulee8/65/orig 2025-08-26T20:08:26.0618945Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-08-26T20:08:26.0619278Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-08-26T20:08:26.0620409Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-08-26T20:08:26.0620997Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-08-26T20:08:26.0621970Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-08-26T20:08:26.0622277Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-08-26T20:08:26.0623589Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-08-26T20:08:26.0623844Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-08-26T20:08:26.0625061Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-08-26T20:08:26.0625659Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-08-26T20:08:26.0626384Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-08-26T20:08:26.0627250Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-08-26T20:08:26.0628186Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-08-26T20:08:26.0628574Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-08-26T20:08:26.0629727Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-08-26T20:08:26.0630652Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-08-26T20:08:26.0631176Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-08-26T20:08:26.0631942Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-08-26T20:08:26.0633194Z * [new branch] gh/oulgen/35/base -> origin/gh/oulgen/35/base 2025-08-26T20:08:26.0633507Z * [new branch] gh/oulgen/35/head -> origin/gh/oulgen/35/head 2025-08-26T20:08:26.0634995Z * [new branch] gh/oulgen/35/orig -> origin/gh/oulgen/35/orig 2025-08-26T20:08:26.0635236Z * [new branch] gh/oulgen/44/base -> origin/gh/oulgen/44/base 2025-08-26T20:08:26.0636348Z * [new branch] gh/oulgen/44/head -> origin/gh/oulgen/44/head 2025-08-26T20:08:26.0636679Z * [new branch] gh/oulgen/44/orig -> origin/gh/oulgen/44/orig 2025-08-26T20:08:26.0638014Z * [new branch] gh/oulgen/45/base -> origin/gh/oulgen/45/base 2025-08-26T20:08:26.0638340Z * [new branch] gh/oulgen/45/head -> origin/gh/oulgen/45/head 2025-08-26T20:08:26.0639621Z * [new branch] gh/oulgen/45/orig -> origin/gh/oulgen/45/orig 2025-08-26T20:08:26.0644563Z * [new branch] gh/oulgen/46/base -> origin/gh/oulgen/46/base 2025-08-26T20:08:26.0644870Z * [new branch] gh/oulgen/46/head -> origin/gh/oulgen/46/head 2025-08-26T20:08:26.0645108Z * [new branch] gh/oulgen/46/orig -> origin/gh/oulgen/46/orig 2025-08-26T20:08:26.0645255Z * [new branch] gh/oulgen/47/base -> origin/gh/oulgen/47/base 2025-08-26T20:08:26.0645385Z * [new branch] gh/oulgen/47/head -> origin/gh/oulgen/47/head 2025-08-26T20:08:26.0645531Z * [new branch] gh/oulgen/47/orig -> origin/gh/oulgen/47/orig 2025-08-26T20:08:26.0650257Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-08-26T20:08:26.0650851Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-08-26T20:08:26.0651021Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-08-26T20:08:26.0656228Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-08-26T20:08:26.0656916Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-08-26T20:08:26.0660973Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-08-26T20:08:26.0661143Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-08-26T20:08:26.0661372Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-08-26T20:08:26.0661754Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-08-26T20:08:26.0667391Z * [new branch] gh/qqaatw/29/base -> origin/gh/qqaatw/29/base 2025-08-26T20:08:26.0672532Z * [new branch] gh/qqaatw/29/head -> origin/gh/qqaatw/29/head 2025-08-26T20:08:26.0675089Z * [new branch] gh/qqaatw/29/orig -> origin/gh/qqaatw/29/orig 2025-08-26T20:08:26.0675336Z * [new branch] gh/raymo/cleanup-dynamo-logging -> origin/gh/raymo/cleanup-dynamo-logging 2025-08-26T20:08:26.0675534Z * [new branch] gh/raymo/refresh-script -> origin/gh/raymo/refresh-script 2025-08-26T20:08:26.0675689Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-08-26T20:08:26.0675823Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-08-26T20:08:26.0675989Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-08-26T20:08:26.0676146Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-08-26T20:08:26.0676286Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-08-26T20:08:26.0676416Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-08-26T20:08:26.0676543Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-08-26T20:08:26.0676686Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-08-26T20:08:26.0676962Z * [new branch] gh/rec/156/base -> origin/gh/rec/156/base 2025-08-26T20:08:26.0677112Z * [new branch] gh/rec/156/head -> origin/gh/rec/156/head 2025-08-26T20:08:26.0677249Z * [new branch] gh/rec/156/orig -> origin/gh/rec/156/orig 2025-08-26T20:08:26.0677385Z * [new branch] gh/rec/158/base -> origin/gh/rec/158/base 2025-08-26T20:08:26.0677524Z * [new branch] gh/rec/158/head -> origin/gh/rec/158/head 2025-08-26T20:08:26.0677662Z * [new branch] gh/rec/158/orig -> origin/gh/rec/158/orig 2025-08-26T20:08:26.0677798Z * [new branch] gh/rec/159/base -> origin/gh/rec/159/base 2025-08-26T20:08:26.0677937Z * [new branch] gh/rec/159/head -> origin/gh/rec/159/head 2025-08-26T20:08:26.0678070Z * [new branch] gh/rec/160/base -> origin/gh/rec/160/base 2025-08-26T20:08:26.0678209Z * [new branch] gh/rec/160/head -> origin/gh/rec/160/head 2025-08-26T20:08:26.0678334Z * [new branch] gh/rec/160/orig -> origin/gh/rec/160/orig 2025-08-26T20:08:26.0678480Z * [new branch] gh/rec/161/base -> origin/gh/rec/161/base 2025-08-26T20:08:26.0678627Z * [new branch] gh/rec/161/head -> origin/gh/rec/161/head 2025-08-26T20:08:26.0678776Z * [new branch] gh/rec/161/orig -> origin/gh/rec/161/orig 2025-08-26T20:08:26.0678902Z * [new branch] gh/rec/162/base -> origin/gh/rec/162/base 2025-08-26T20:08:26.0679027Z * [new branch] gh/rec/162/head -> origin/gh/rec/162/head 2025-08-26T20:08:26.0679343Z * [new branch] gh/rec/162/orig -> origin/gh/rec/162/orig 2025-08-26T20:08:26.0679495Z * [new branch] gh/rec/163/base -> origin/gh/rec/163/base 2025-08-26T20:08:26.0679643Z * [new branch] gh/rec/163/head -> origin/gh/rec/163/head 2025-08-26T20:08:26.0679769Z * [new branch] gh/rec/163/orig -> origin/gh/rec/163/orig 2025-08-26T20:08:26.0679912Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-08-26T20:08:26.0680045Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-08-26T20:08:26.0680529Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-08-26T20:08:26.0684802Z * [new branch] gh/rec/165/base -> origin/gh/rec/165/base 2025-08-26T20:08:26.0685440Z * [new branch] gh/rec/165/head -> origin/gh/rec/165/head 2025-08-26T20:08:26.0685741Z * [new branch] gh/rec/165/orig -> origin/gh/rec/165/orig 2025-08-26T20:08:26.0686336Z * [new branch] gh/robert-hardwick/1/base -> origin/gh/robert-hardwick/1/base 2025-08-26T20:08:26.0686600Z * [new branch] gh/robert-hardwick/1/head -> origin/gh/robert-hardwick/1/head 2025-08-26T20:08:26.0686779Z * [new branch] gh/robert-hardwick/1/orig -> origin/gh/robert-hardwick/1/orig 2025-08-26T20:08:26.0687083Z * [new branch] gh/robert-hardwick/2/base -> origin/gh/robert-hardwick/2/base 2025-08-26T20:08:26.0687554Z * [new branch] gh/robert-hardwick/2/head -> origin/gh/robert-hardwick/2/head 2025-08-26T20:08:26.0692395Z * [new branch] gh/robert-hardwick/2/orig -> origin/gh/robert-hardwick/2/orig 2025-08-26T20:08:26.0692623Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-08-26T20:08:26.0692801Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-08-26T20:08:26.0692985Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-08-26T20:08:26.0693144Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-08-26T20:08:26.0693564Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-08-26T20:08:26.0693735Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-08-26T20:08:26.0694820Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-08-26T20:08:26.0695276Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-08-26T20:08:26.0695650Z * [new branch] gh/rtimpe/10/base -> origin/gh/rtimpe/10/base 2025-08-26T20:08:26.0696641Z * [new branch] gh/rtimpe/10/head -> origin/gh/rtimpe/10/head 2025-08-26T20:08:26.0704013Z * [new branch] gh/rtimpe/10/orig -> origin/gh/rtimpe/10/orig 2025-08-26T20:08:26.0708949Z * [new branch] gh/rtimpe/11/base -> origin/gh/rtimpe/11/base 2025-08-26T20:08:26.0709142Z * [new branch] gh/rtimpe/11/head -> origin/gh/rtimpe/11/head 2025-08-26T20:08:26.0709308Z * [new branch] gh/rtimpe/11/orig -> origin/gh/rtimpe/11/orig 2025-08-26T20:08:26.0709454Z * [new branch] gh/rtimpe/12/base -> origin/gh/rtimpe/12/base 2025-08-26T20:08:26.0709592Z * [new branch] gh/rtimpe/12/head -> origin/gh/rtimpe/12/head 2025-08-26T20:08:26.0709737Z * [new branch] gh/rtimpe/12/orig -> origin/gh/rtimpe/12/orig 2025-08-26T20:08:26.0709880Z * [new branch] gh/rtimpe/13/base -> origin/gh/rtimpe/13/base 2025-08-26T20:08:26.0710014Z * [new branch] gh/rtimpe/13/head -> origin/gh/rtimpe/13/head 2025-08-26T20:08:26.0710163Z * [new branch] gh/rtimpe/13/orig -> origin/gh/rtimpe/13/orig 2025-08-26T20:08:26.0715296Z * [new branch] gh/rtimpe/14/base -> origin/gh/rtimpe/14/base 2025-08-26T20:08:26.0715510Z * [new branch] gh/rtimpe/14/head -> origin/gh/rtimpe/14/head 2025-08-26T20:08:26.0715684Z * [new branch] gh/rtimpe/14/orig -> origin/gh/rtimpe/14/orig 2025-08-26T20:08:26.0715850Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-08-26T20:08:26.0716361Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-08-26T20:08:26.0716779Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-08-26T20:08:26.0716931Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-08-26T20:08:26.0717080Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-08-26T20:08:26.0717222Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-08-26T20:08:26.0717608Z * [new branch] gh/rtimpe/6/base -> origin/gh/rtimpe/6/base 2025-08-26T20:08:26.0718647Z * [new branch] gh/rtimpe/6/head -> origin/gh/rtimpe/6/head 2025-08-26T20:08:26.0718976Z * [new branch] gh/rtimpe/6/orig -> origin/gh/rtimpe/6/orig 2025-08-26T20:08:26.0724548Z * [new branch] gh/rtimpe/7/base -> origin/gh/rtimpe/7/base 2025-08-26T20:08:26.0724786Z * [new branch] gh/rtimpe/7/head -> origin/gh/rtimpe/7/head 2025-08-26T20:08:26.0724927Z * [new branch] gh/rtimpe/7/orig -> origin/gh/rtimpe/7/orig 2025-08-26T20:08:26.0725080Z * [new branch] gh/rtimpe/8/base -> origin/gh/rtimpe/8/base 2025-08-26T20:08:26.0725217Z * [new branch] gh/rtimpe/8/head -> origin/gh/rtimpe/8/head 2025-08-26T20:08:26.0725356Z * [new branch] gh/rtimpe/8/orig -> origin/gh/rtimpe/8/orig 2025-08-26T20:08:26.0725506Z * [new branch] gh/rtimpe/9/base -> origin/gh/rtimpe/9/base 2025-08-26T20:08:26.0731125Z * [new branch] gh/rtimpe/9/head -> origin/gh/rtimpe/9/head 2025-08-26T20:08:26.0731602Z * [new branch] gh/rtimpe/9/orig -> origin/gh/rtimpe/9/orig 2025-08-26T20:08:26.0731800Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-08-26T20:08:26.0731971Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-08-26T20:08:26.0732137Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-08-26T20:08:26.0732304Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-08-26T20:08:26.0738026Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-08-26T20:08:26.0738226Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-08-26T20:08:26.0738393Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-08-26T20:08:26.0738563Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-08-26T20:08:26.0738724Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-08-26T20:08:26.0738873Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-08-26T20:08:26.0739030Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-08-26T20:08:26.0739185Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-08-26T20:08:26.0739336Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-08-26T20:08:26.0739494Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-08-26T20:08:26.0739642Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-08-26T20:08:26.0739801Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-08-26T20:08:26.0739957Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-08-26T20:08:26.0740120Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-08-26T20:08:26.0745358Z * [new branch] gh/sarckk/2/base -> origin/gh/sarckk/2/base 2025-08-26T20:08:26.0745530Z * [new branch] gh/sarckk/2/head -> origin/gh/sarckk/2/head 2025-08-26T20:08:26.0745841Z * [new branch] gh/sarckk/2/orig -> origin/gh/sarckk/2/orig 2025-08-26T20:08:26.0746043Z * [new branch] gh/seemethere/23/head -> origin/gh/seemethere/23/head 2025-08-26T20:08:26.0746212Z * [new branch] gh/seemethere/32/base -> origin/gh/seemethere/32/base 2025-08-26T20:08:26.0746407Z * [new branch] gh/seemethere/32/head -> origin/gh/seemethere/32/head 2025-08-26T20:08:26.0746644Z * [new branch] gh/seemethere/32/orig -> origin/gh/seemethere/32/orig 2025-08-26T20:08:26.0746907Z * [new branch] gh/seemethere/33/base -> origin/gh/seemethere/33/base 2025-08-26T20:08:26.0747097Z * [new branch] gh/seemethere/33/head -> origin/gh/seemethere/33/head 2025-08-26T20:08:26.0747356Z * [new branch] gh/seemethere/33/orig -> origin/gh/seemethere/33/orig 2025-08-26T20:08:26.0747525Z * [new branch] gh/seemethere/34/base -> origin/gh/seemethere/34/base 2025-08-26T20:08:26.0747922Z * [new branch] gh/seemethere/34/head -> origin/gh/seemethere/34/head 2025-08-26T20:08:26.0749310Z * [new branch] gh/seemethere/34/orig -> origin/gh/seemethere/34/orig 2025-08-26T20:08:26.0749716Z * [new branch] gh/seemethere/35/base -> origin/gh/seemethere/35/base 2025-08-26T20:08:26.0751932Z * [new branch] gh/seemethere/35/head -> origin/gh/seemethere/35/head 2025-08-26T20:08:26.0752450Z * [new branch] gh/seemethere/35/orig -> origin/gh/seemethere/35/orig 2025-08-26T20:08:26.0752770Z * [new branch] gh/seemethere/37/base -> origin/gh/seemethere/37/base 2025-08-26T20:08:26.0752984Z * [new branch] gh/seemethere/37/head -> origin/gh/seemethere/37/head 2025-08-26T20:08:26.0753677Z * [new branch] gh/seemethere/37/orig -> origin/gh/seemethere/37/orig 2025-08-26T20:08:26.0755990Z * [new branch] gh/seemethere/43/base -> origin/gh/seemethere/43/base 2025-08-26T20:08:26.0756619Z * [new branch] gh/seemethere/43/head -> origin/gh/seemethere/43/head 2025-08-26T20:08:26.0756808Z * [new branch] gh/seemethere/43/orig -> origin/gh/seemethere/43/orig 2025-08-26T20:08:26.0757373Z * [new branch] gh/seemethere/44/base -> origin/gh/seemethere/44/base 2025-08-26T20:08:26.0757928Z * [new branch] gh/seemethere/44/head -> origin/gh/seemethere/44/head 2025-08-26T20:08:26.0758975Z * [new branch] gh/seemethere/44/orig -> origin/gh/seemethere/44/orig 2025-08-26T20:08:26.0760207Z * [new branch] gh/seemethere/48/base -> origin/gh/seemethere/48/base 2025-08-26T20:08:26.0760459Z * [new branch] gh/seemethere/48/head -> origin/gh/seemethere/48/head 2025-08-26T20:08:26.0761446Z * [new branch] gh/seemethere/48/orig -> origin/gh/seemethere/48/orig 2025-08-26T20:08:26.0765346Z * [new branch] gh/seemethere/49/base -> origin/gh/seemethere/49/base 2025-08-26T20:08:26.0765543Z * [new branch] gh/seemethere/49/head -> origin/gh/seemethere/49/head 2025-08-26T20:08:26.0765717Z * [new branch] gh/seemethere/49/orig -> origin/gh/seemethere/49/orig 2025-08-26T20:08:26.0765873Z * [new branch] gh/seemethere/51/base -> origin/gh/seemethere/51/base 2025-08-26T20:08:26.0766039Z * [new branch] gh/seemethere/51/head -> origin/gh/seemethere/51/head 2025-08-26T20:08:26.0766404Z * [new branch] gh/seemethere/51/orig -> origin/gh/seemethere/51/orig 2025-08-26T20:08:26.0766828Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-08-26T20:08:26.0767211Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-08-26T20:08:26.0767811Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-08-26T20:08:26.0771461Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-08-26T20:08:26.0771658Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-08-26T20:08:26.0771850Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-08-26T20:08:26.0772012Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-08-26T20:08:26.0772200Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-08-26T20:08:26.0772780Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-08-26T20:08:26.0773806Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-08-26T20:08:26.0774439Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-08-26T20:08:26.0778870Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-08-26T20:08:26.0779068Z * [new branch] gh/seemethere/56/base -> origin/gh/seemethere/56/base 2025-08-26T20:08:26.0779230Z * [new branch] gh/seemethere/56/head -> origin/gh/seemethere/56/head 2025-08-26T20:08:26.0779397Z * [new branch] gh/seemethere/56/orig -> origin/gh/seemethere/56/orig 2025-08-26T20:08:26.0779552Z * [new branch] gh/seemethere/57/base -> origin/gh/seemethere/57/base 2025-08-26T20:08:26.0779891Z * [new branch] gh/seemethere/57/head -> origin/gh/seemethere/57/head 2025-08-26T20:08:26.0780056Z * [new branch] gh/seemethere/57/orig -> origin/gh/seemethere/57/orig 2025-08-26T20:08:26.0780696Z * [new branch] gh/seemethere/58/base -> origin/gh/seemethere/58/base 2025-08-26T20:08:26.0781296Z * [new branch] gh/seemethere/58/head -> origin/gh/seemethere/58/head 2025-08-26T20:08:26.0785859Z * [new branch] gh/seemethere/58/orig -> origin/gh/seemethere/58/orig 2025-08-26T20:08:26.0786050Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-08-26T20:08:26.0786199Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-08-26T20:08:26.0786346Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-08-26T20:08:26.0786515Z * [new branch] gh/seemethere/7/head -> origin/gh/seemethere/7/head 2025-08-26T20:08:26.0787076Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-08-26T20:08:26.0787904Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-08-26T20:08:26.0788635Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-08-26T20:08:26.0793444Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-08-26T20:08:26.0794080Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-08-26T20:08:26.0794282Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-08-26T20:08:26.0794447Z * [new branch] gh/shunting314/211/base -> origin/gh/shunting314/211/base 2025-08-26T20:08:26.0794605Z * [new branch] gh/shunting314/211/head -> origin/gh/shunting314/211/head 2025-08-26T20:08:26.0794788Z * [new branch] gh/shunting314/211/orig -> origin/gh/shunting314/211/orig 2025-08-26T20:08:26.0794949Z * [new branch] gh/shunting314/212/base -> origin/gh/shunting314/212/base 2025-08-26T20:08:26.0795228Z * [new branch] gh/shunting314/212/head -> origin/gh/shunting314/212/head 2025-08-26T20:08:26.0795694Z * [new branch] gh/shunting314/212/orig -> origin/gh/shunting314/212/orig 2025-08-26T20:08:26.0798473Z * [new branch] gh/shunting314/213/base -> origin/gh/shunting314/213/base 2025-08-26T20:08:26.0798668Z * [new branch] gh/shunting314/213/head -> origin/gh/shunting314/213/head 2025-08-26T20:08:26.0798831Z * [new branch] gh/shunting314/213/orig -> origin/gh/shunting314/213/orig 2025-08-26T20:08:26.0806743Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-08-26T20:08:26.0806930Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-08-26T20:08:26.0807105Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-08-26T20:08:26.0807274Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-08-26T20:08:26.0807419Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-08-26T20:08:26.0807573Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-08-26T20:08:26.0807721Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-08-26T20:08:26.0807876Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-08-26T20:08:26.0813725Z * [new branch] gh/sinhaanhsul/1/base -> origin/gh/sinhaanhsul/1/base 2025-08-26T20:08:26.0813914Z * [new branch] gh/sinhaanhsul/1/head -> origin/gh/sinhaanhsul/1/head 2025-08-26T20:08:26.0814076Z * [new branch] gh/skarjala/13/base -> origin/gh/skarjala/13/base 2025-08-26T20:08:26.0814455Z * [new branch] gh/skarjala/13/head -> origin/gh/skarjala/13/head 2025-08-26T20:08:26.0814618Z * [new branch] gh/skarjala/13/orig -> origin/gh/skarjala/13/orig 2025-08-26T20:08:26.0814806Z * [new branch] gh/skarjala/15/base -> origin/gh/skarjala/15/base 2025-08-26T20:08:26.0814967Z * [new branch] gh/skarjala/15/head -> origin/gh/skarjala/15/head 2025-08-26T20:08:26.0815124Z * [new branch] gh/skarjala/15/orig -> origin/gh/skarjala/15/orig 2025-08-26T20:08:26.0815274Z * [new branch] gh/skarjala/16/base -> origin/gh/skarjala/16/base 2025-08-26T20:08:26.0815430Z * [new branch] gh/skarjala/16/head -> origin/gh/skarjala/16/head 2025-08-26T20:08:26.0815589Z * [new branch] gh/skarjala/16/orig -> origin/gh/skarjala/16/orig 2025-08-26T20:08:26.0815741Z * [new branch] gh/skarjala/17/base -> origin/gh/skarjala/17/base 2025-08-26T20:08:26.0822586Z * [new branch] gh/skarjala/17/head -> origin/gh/skarjala/17/head 2025-08-26T20:08:26.0827080Z * [new branch] gh/skarjala/17/orig -> origin/gh/skarjala/17/orig 2025-08-26T20:08:26.0831504Z * [new branch] gh/skarjala/18/base -> origin/gh/skarjala/18/base 2025-08-26T20:08:26.0831721Z * [new branch] gh/skarjala/18/head -> origin/gh/skarjala/18/head 2025-08-26T20:08:26.0832311Z * [new branch] gh/skarjala/18/orig -> origin/gh/skarjala/18/orig 2025-08-26T20:08:26.0832493Z * [new branch] gh/skarjala/19/base -> origin/gh/skarjala/19/base 2025-08-26T20:08:26.0832644Z * [new branch] gh/skarjala/19/head -> origin/gh/skarjala/19/head 2025-08-26T20:08:26.0832794Z * [new branch] gh/skarjala/19/orig -> origin/gh/skarjala/19/orig 2025-08-26T20:08:26.0832980Z * [new branch] gh/slayton58/1/base -> origin/gh/slayton58/1/base 2025-08-26T20:08:26.0833133Z * [new branch] gh/slayton58/1/head -> origin/gh/slayton58/1/head 2025-08-26T20:08:26.0833321Z * [new branch] gh/slayton58/1/orig -> origin/gh/slayton58/1/orig 2025-08-26T20:08:26.0833477Z * [new branch] gh/slayton58/2/base -> origin/gh/slayton58/2/base 2025-08-26T20:08:26.0833830Z * [new branch] gh/slayton58/2/head -> origin/gh/slayton58/2/head 2025-08-26T20:08:26.0833986Z * [new branch] gh/slayton58/2/orig -> origin/gh/slayton58/2/orig 2025-08-26T20:08:26.0834133Z * [new branch] gh/slayton58/3/base -> origin/gh/slayton58/3/base 2025-08-26T20:08:26.0834289Z * [new branch] gh/slayton58/3/head -> origin/gh/slayton58/3/head 2025-08-26T20:08:26.0834434Z * [new branch] gh/slayton58/3/orig -> origin/gh/slayton58/3/orig 2025-08-26T20:08:26.0834585Z * [new branch] gh/slayton58/4/base -> origin/gh/slayton58/4/base 2025-08-26T20:08:26.0834743Z * [new branch] gh/slayton58/4/head -> origin/gh/slayton58/4/head 2025-08-26T20:08:26.0834891Z * [new branch] gh/slayton58/4/orig -> origin/gh/slayton58/4/orig 2025-08-26T20:08:26.0835042Z * [new branch] gh/slayton58/5/base -> origin/gh/slayton58/5/base 2025-08-26T20:08:26.0835191Z * [new branch] gh/slayton58/5/head -> origin/gh/slayton58/5/head 2025-08-26T20:08:26.0835335Z * [new branch] gh/slayton58/5/orig -> origin/gh/slayton58/5/orig 2025-08-26T20:08:26.0835521Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-08-26T20:08:26.0835682Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-08-26T20:08:26.0835843Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-08-26T20:08:26.0836040Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-08-26T20:08:26.0836199Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-08-26T20:08:26.0836356Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-08-26T20:08:26.0836583Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-08-26T20:08:26.0837166Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-08-26T20:08:26.0838069Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-08-26T20:08:26.0839415Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-08-26T20:08:26.0839839Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-08-26T20:08:26.0841712Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-08-26T20:08:26.0841938Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-08-26T20:08:26.0843822Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-08-26T20:08:26.0844005Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-08-26T20:08:26.0853447Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-08-26T20:08:26.0853691Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-08-26T20:08:26.0853865Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-08-26T20:08:26.0854027Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-08-26T20:08:26.0854329Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-08-26T20:08:26.0854526Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-08-26T20:08:26.0854673Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-08-26T20:08:26.0854961Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-08-26T20:08:26.0855128Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-08-26T20:08:26.0855571Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-08-26T20:08:26.0855848Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-08-26T20:08:26.0856117Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-08-26T20:08:26.0856817Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-08-26T20:08:26.0857012Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-08-26T20:08:26.0857198Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-08-26T20:08:26.0857350Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-08-26T20:08:26.0857508Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-08-26T20:08:26.0857661Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-08-26T20:08:26.0861608Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-08-26T20:08:26.0861922Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-08-26T20:08:26.0862111Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-08-26T20:08:26.0862351Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-08-26T20:08:26.0862530Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-08-26T20:08:26.0862936Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-08-26T20:08:26.0866007Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-08-26T20:08:26.0866182Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-08-26T20:08:26.0866460Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-08-26T20:08:26.0866637Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-08-26T20:08:26.0866863Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-08-26T20:08:26.0867039Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-08-26T20:08:26.0867257Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-08-26T20:08:26.0870125Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-08-26T20:08:26.0870539Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-08-26T20:08:26.0870807Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-08-26T20:08:26.0870991Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-08-26T20:08:26.0871264Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-08-26T20:08:26.0875180Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-08-26T20:08:26.0875821Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-08-26T20:08:26.0876015Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-08-26T20:08:26.0876185Z * [new branch] gh/soulitzer/362/base -> origin/gh/soulitzer/362/base 2025-08-26T20:08:26.0876365Z * [new branch] gh/soulitzer/362/head -> origin/gh/soulitzer/362/head 2025-08-26T20:08:26.0876526Z * [new branch] gh/soulitzer/362/orig -> origin/gh/soulitzer/362/orig 2025-08-26T20:08:26.0876701Z * [new branch] gh/soulitzer/372/base -> origin/gh/soulitzer/372/base 2025-08-26T20:08:26.0877183Z * [new branch] gh/soulitzer/372/head -> origin/gh/soulitzer/372/head 2025-08-26T20:08:26.0889677Z * [new branch] gh/soulitzer/372/orig -> origin/gh/soulitzer/372/orig 2025-08-26T20:08:26.0890372Z * [new branch] gh/soulitzer/373/base -> origin/gh/soulitzer/373/base 2025-08-26T20:08:26.0890648Z * [new branch] gh/soulitzer/373/head -> origin/gh/soulitzer/373/head 2025-08-26T20:08:26.0890831Z * [new branch] gh/soulitzer/373/orig -> origin/gh/soulitzer/373/orig 2025-08-26T20:08:26.0890987Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-08-26T20:08:26.0891272Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-08-26T20:08:26.0891445Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-08-26T20:08:26.0891677Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-08-26T20:08:26.0891865Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-08-26T20:08:26.0892117Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-08-26T20:08:26.0892289Z * [new branch] gh/soulitzer/376/base -> origin/gh/soulitzer/376/base 2025-08-26T20:08:26.0892445Z * [new branch] gh/soulitzer/376/head -> origin/gh/soulitzer/376/head 2025-08-26T20:08:26.0892593Z * [new branch] gh/soulitzer/376/orig -> origin/gh/soulitzer/376/orig 2025-08-26T20:08:26.0892899Z * [new branch] gh/soulitzer/377/base -> origin/gh/soulitzer/377/base 2025-08-26T20:08:26.0893101Z * [new branch] gh/soulitzer/377/head -> origin/gh/soulitzer/377/head 2025-08-26T20:08:26.0893294Z * [new branch] gh/soulitzer/377/orig -> origin/gh/soulitzer/377/orig 2025-08-26T20:08:26.0893448Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-08-26T20:08:26.0893610Z * [new branch] gh/swolchok/758/base -> origin/gh/swolchok/758/base 2025-08-26T20:08:26.0893765Z * [new branch] gh/swolchok/758/head -> origin/gh/swolchok/758/head 2025-08-26T20:08:26.0893916Z * [new branch] gh/swolchok/758/orig -> origin/gh/swolchok/758/orig 2025-08-26T20:08:26.0894102Z * [new branch] gh/swolchok/767/base -> origin/gh/swolchok/767/base 2025-08-26T20:08:26.0895558Z * [new branch] gh/swolchok/767/head -> origin/gh/swolchok/767/head 2025-08-26T20:08:26.0896040Z * [new branch] gh/swolchok/767/orig -> origin/gh/swolchok/767/orig 2025-08-26T20:08:26.0897862Z * [new branch] gh/swolchok/768/base -> origin/gh/swolchok/768/base 2025-08-26T20:08:26.0898479Z * [new branch] gh/swolchok/768/head -> origin/gh/swolchok/768/head 2025-08-26T20:08:26.0898897Z * [new branch] gh/swolchok/768/orig -> origin/gh/swolchok/768/orig 2025-08-26T20:08:26.0900176Z * [new branch] gh/swolchok/769/base -> origin/gh/swolchok/769/base 2025-08-26T20:08:26.0900768Z * [new branch] gh/swolchok/769/head -> origin/gh/swolchok/769/head 2025-08-26T20:08:26.0901692Z * [new branch] gh/swolchok/769/orig -> origin/gh/swolchok/769/orig 2025-08-26T20:08:26.0902756Z * [new branch] gh/swolchok/771/base -> origin/gh/swolchok/771/base 2025-08-26T20:08:26.0907268Z * [new branch] gh/swolchok/771/head -> origin/gh/swolchok/771/head 2025-08-26T20:08:26.0907444Z * [new branch] gh/swolchok/771/orig -> origin/gh/swolchok/771/orig 2025-08-26T20:08:26.0907598Z * [new branch] gh/swolchok/772/base -> origin/gh/swolchok/772/base 2025-08-26T20:08:26.0907735Z * [new branch] gh/swolchok/772/head -> origin/gh/swolchok/772/head 2025-08-26T20:08:26.0907880Z * [new branch] gh/swolchok/772/orig -> origin/gh/swolchok/772/orig 2025-08-26T20:08:26.0908235Z * [new branch] gh/swolchok/773/base -> origin/gh/swolchok/773/base 2025-08-26T20:08:26.0912795Z * [new branch] gh/swolchok/773/head -> origin/gh/swolchok/773/head 2025-08-26T20:08:26.0913093Z * [new branch] gh/swolchok/773/orig -> origin/gh/swolchok/773/orig 2025-08-26T20:08:26.0913274Z * [new branch] gh/swolchok/786/base -> origin/gh/swolchok/786/base 2025-08-26T20:08:26.0913421Z * [new branch] gh/swolchok/786/head -> origin/gh/swolchok/786/head 2025-08-26T20:08:26.0913731Z * [new branch] gh/swolchok/786/orig -> origin/gh/swolchok/786/orig 2025-08-26T20:08:26.0914421Z * [new branch] gh/swolchok/787/base -> origin/gh/swolchok/787/base 2025-08-26T20:08:26.0914630Z * [new branch] gh/swolchok/787/head -> origin/gh/swolchok/787/head 2025-08-26T20:08:26.0914824Z * [new branch] gh/swolchok/787/orig -> origin/gh/swolchok/787/orig 2025-08-26T20:08:26.0914996Z * [new branch] gh/swolchok/788/base -> origin/gh/swolchok/788/base 2025-08-26T20:08:26.0915324Z * [new branch] gh/swolchok/788/head -> origin/gh/swolchok/788/head 2025-08-26T20:08:26.0915713Z * [new branch] gh/swolchok/788/orig -> origin/gh/swolchok/788/orig 2025-08-26T20:08:26.0917550Z * [new branch] gh/swolchok/789/base -> origin/gh/swolchok/789/base 2025-08-26T20:08:26.0917988Z * [new branch] gh/swolchok/789/head -> origin/gh/swolchok/789/head 2025-08-26T20:08:26.0918212Z * [new branch] gh/swolchok/789/orig -> origin/gh/swolchok/789/orig 2025-08-26T20:08:26.0919633Z * [new branch] gh/swolchok/790/base -> origin/gh/swolchok/790/base 2025-08-26T20:08:26.0919922Z * [new branch] gh/swolchok/790/head -> origin/gh/swolchok/790/head 2025-08-26T20:08:26.0923393Z * [new branch] gh/swolchok/790/orig -> origin/gh/swolchok/790/orig 2025-08-26T20:08:26.0923577Z * [new branch] gh/swolchok/791/base -> origin/gh/swolchok/791/base 2025-08-26T20:08:26.0930413Z * [new branch] gh/swolchok/791/head -> origin/gh/swolchok/791/head 2025-08-26T20:08:26.0931091Z * [new branch] gh/swolchok/791/orig -> origin/gh/swolchok/791/orig 2025-08-26T20:08:26.0931419Z * [new branch] gh/swolchok/792/base -> origin/gh/swolchok/792/base 2025-08-26T20:08:26.0931630Z * [new branch] gh/swolchok/792/head -> origin/gh/swolchok/792/head 2025-08-26T20:08:26.0931784Z * [new branch] gh/swolchok/792/orig -> origin/gh/swolchok/792/orig 2025-08-26T20:08:26.0932091Z * [new branch] gh/swolchok/793/base -> origin/gh/swolchok/793/base 2025-08-26T20:08:26.0932315Z * [new branch] gh/swolchok/793/head -> origin/gh/swolchok/793/head 2025-08-26T20:08:26.0932488Z * [new branch] gh/swolchok/793/orig -> origin/gh/swolchok/793/orig 2025-08-26T20:08:26.0932664Z * [new branch] gh/swolchok/794/base -> origin/gh/swolchok/794/base 2025-08-26T20:08:26.0932814Z * [new branch] gh/swolchok/794/head -> origin/gh/swolchok/794/head 2025-08-26T20:08:26.0932980Z * [new branch] gh/swolchok/794/orig -> origin/gh/swolchok/794/orig 2025-08-26T20:08:26.0933130Z * [new branch] gh/swolchok/795/base -> origin/gh/swolchok/795/base 2025-08-26T20:08:26.0933288Z * [new branch] gh/swolchok/795/head -> origin/gh/swolchok/795/head 2025-08-26T20:08:26.0933442Z * [new branch] gh/swolchok/795/orig -> origin/gh/swolchok/795/orig 2025-08-26T20:08:26.0933617Z * [new branch] gh/swolchok/796/base -> origin/gh/swolchok/796/base 2025-08-26T20:08:26.0934610Z * [new branch] gh/swolchok/796/head -> origin/gh/swolchok/796/head 2025-08-26T20:08:26.0935044Z * [new branch] gh/swolchok/796/orig -> origin/gh/swolchok/796/orig 2025-08-26T20:08:26.0935442Z * [new branch] gh/swolchok/797/base -> origin/gh/swolchok/797/base 2025-08-26T20:08:26.0938625Z * [new branch] gh/swolchok/797/head -> origin/gh/swolchok/797/head 2025-08-26T20:08:26.0938989Z * [new branch] gh/swolchok/797/orig -> origin/gh/swolchok/797/orig 2025-08-26T20:08:26.0939241Z * [new branch] gh/swolchok/798/base -> origin/gh/swolchok/798/base 2025-08-26T20:08:26.0939461Z * [new branch] gh/swolchok/798/head -> origin/gh/swolchok/798/head 2025-08-26T20:08:26.0944359Z * [new branch] gh/swolchok/798/orig -> origin/gh/swolchok/798/orig 2025-08-26T20:08:26.0944708Z * [new branch] gh/swolchok/799/base -> origin/gh/swolchok/799/base 2025-08-26T20:08:26.0945019Z * [new branch] gh/swolchok/799/head -> origin/gh/swolchok/799/head 2025-08-26T20:08:26.0945231Z * [new branch] gh/swolchok/799/orig -> origin/gh/swolchok/799/orig 2025-08-26T20:08:26.0945413Z * [new branch] gh/swolchok/800/base -> origin/gh/swolchok/800/base 2025-08-26T20:08:26.0948251Z * [new branch] gh/swolchok/800/head -> origin/gh/swolchok/800/head 2025-08-26T20:08:26.0948526Z * [new branch] gh/swolchok/800/orig -> origin/gh/swolchok/800/orig 2025-08-26T20:08:26.0948884Z * [new branch] gh/swolchok/801/base -> origin/gh/swolchok/801/base 2025-08-26T20:08:26.0949159Z * [new branch] gh/swolchok/801/head -> origin/gh/swolchok/801/head 2025-08-26T20:08:26.0949331Z * [new branch] gh/swolchok/801/orig -> origin/gh/swolchok/801/orig 2025-08-26T20:08:26.0949569Z * [new branch] gh/swolchok/802/base -> origin/gh/swolchok/802/base 2025-08-26T20:08:26.0949729Z * [new branch] gh/swolchok/802/head -> origin/gh/swolchok/802/head 2025-08-26T20:08:26.0949968Z * [new branch] gh/swolchok/802/orig -> origin/gh/swolchok/802/orig 2025-08-26T20:08:26.0955131Z * [new branch] gh/swolchok/803/base -> origin/gh/swolchok/803/base 2025-08-26T20:08:26.0955333Z * [new branch] gh/swolchok/803/head -> origin/gh/swolchok/803/head 2025-08-26T20:08:26.0955489Z * [new branch] gh/swolchok/803/orig -> origin/gh/swolchok/803/orig 2025-08-26T20:08:26.0955687Z * [new branch] gh/swolchok/804/base -> origin/gh/swolchok/804/base 2025-08-26T20:08:26.0955842Z * [new branch] gh/swolchok/804/head -> origin/gh/swolchok/804/head 2025-08-26T20:08:26.0955997Z * [new branch] gh/swolchok/804/orig -> origin/gh/swolchok/804/orig 2025-08-26T20:08:26.0956164Z * [new branch] gh/swolchok/805/base -> origin/gh/swolchok/805/base 2025-08-26T20:08:26.0956351Z * [new branch] gh/swolchok/805/head -> origin/gh/swolchok/805/head 2025-08-26T20:08:26.0956503Z * [new branch] gh/swolchok/805/orig -> origin/gh/swolchok/805/orig 2025-08-26T20:08:26.0956841Z * [new branch] gh/swolchok/806/base -> origin/gh/swolchok/806/base 2025-08-26T20:08:26.0957493Z * [new branch] gh/swolchok/806/head -> origin/gh/swolchok/806/head 2025-08-26T20:08:26.0957694Z * [new branch] gh/swolchok/806/orig -> origin/gh/swolchok/806/orig 2025-08-26T20:08:26.0959675Z * [new branch] gh/swolchok/807/base -> origin/gh/swolchok/807/base 2025-08-26T20:08:26.0959975Z * [new branch] gh/swolchok/807/head -> origin/gh/swolchok/807/head 2025-08-26T20:08:26.0960590Z * [new branch] gh/swolchok/807/orig -> origin/gh/swolchok/807/orig 2025-08-26T20:08:26.0966060Z * [new branch] gh/swolchok/808/base -> origin/gh/swolchok/808/base 2025-08-26T20:08:26.0966407Z * [new branch] gh/swolchok/808/head -> origin/gh/swolchok/808/head 2025-08-26T20:08:26.0966976Z * [new branch] gh/swolchok/808/orig -> origin/gh/swolchok/808/orig 2025-08-26T20:08:26.0967139Z * [new branch] gh/swolchok/809/base -> origin/gh/swolchok/809/base 2025-08-26T20:08:26.0967291Z * [new branch] gh/swolchok/809/head -> origin/gh/swolchok/809/head 2025-08-26T20:08:26.0967459Z * [new branch] gh/swolchok/809/orig -> origin/gh/swolchok/809/orig 2025-08-26T20:08:26.0967643Z * [new branch] gh/syed-ahmed/2/base -> origin/gh/syed-ahmed/2/base 2025-08-26T20:08:26.0967801Z * [new branch] gh/syed-ahmed/2/head -> origin/gh/syed-ahmed/2/head 2025-08-26T20:08:26.0967943Z * [new branch] gh/syed-ahmed/2/orig -> origin/gh/syed-ahmed/2/orig 2025-08-26T20:08:26.0969399Z * [new branch] gh/syed-ahmed/3/base -> origin/gh/syed-ahmed/3/base 2025-08-26T20:08:26.0969554Z * [new branch] gh/syed-ahmed/3/head -> origin/gh/syed-ahmed/3/head 2025-08-26T20:08:26.0970312Z * [new branch] gh/syed-ahmed/3/orig -> origin/gh/syed-ahmed/3/orig 2025-08-26T20:08:26.0975688Z * [new branch] gh/syed-ahmed/4/base -> origin/gh/syed-ahmed/4/base 2025-08-26T20:08:26.0975877Z * [new branch] gh/syed-ahmed/4/head -> origin/gh/syed-ahmed/4/head 2025-08-26T20:08:26.0976225Z * [new branch] gh/syed-ahmed/4/orig -> origin/gh/syed-ahmed/4/orig 2025-08-26T20:08:26.0976386Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-08-26T20:08:26.0976541Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-08-26T20:08:26.0976683Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-08-26T20:08:26.0976862Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-08-26T20:08:26.0977004Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-08-26T20:08:26.0980703Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-08-26T20:08:26.0980922Z * [new branch] gh/tugsbayasgalan/1/base -> origin/gh/tugsbayasgalan/1/base 2025-08-26T20:08:26.0981096Z * [new branch] gh/tugsbayasgalan/1/head -> origin/gh/tugsbayasgalan/1/head 2025-08-26T20:08:26.0981283Z * [new branch] gh/tugsbayasgalan/1/orig -> origin/gh/tugsbayasgalan/1/orig 2025-08-26T20:08:26.0981721Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-08-26T20:08:26.0982411Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-08-26T20:08:26.0983278Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-08-26T20:08:26.0983862Z * [new branch] gh/tugsbayasgalan/3/base -> origin/gh/tugsbayasgalan/3/base 2025-08-26T20:08:26.0985184Z * [new branch] gh/tugsbayasgalan/3/head -> origin/gh/tugsbayasgalan/3/head 2025-08-26T20:08:26.0985348Z * [new branch] gh/tugsbayasgalan/3/orig -> origin/gh/tugsbayasgalan/3/orig 2025-08-26T20:08:26.0990209Z * [new branch] gh/v0i0/1/base -> origin/gh/v0i0/1/base 2025-08-26T20:08:26.0990374Z * [new branch] gh/v0i0/1/head -> origin/gh/v0i0/1/head 2025-08-26T20:08:26.0990532Z * [new branch] gh/v0i0/1/orig -> origin/gh/v0i0/1/orig 2025-08-26T20:08:26.0990665Z * [new branch] gh/v0i0/2/base -> origin/gh/v0i0/2/base 2025-08-26T20:08:26.0990802Z * [new branch] gh/v0i0/2/head -> origin/gh/v0i0/2/head 2025-08-26T20:08:26.0990934Z * [new branch] gh/v0i0/2/orig -> origin/gh/v0i0/2/orig 2025-08-26T20:08:26.0991665Z * [new branch] gh/v0i0/3/base -> origin/gh/v0i0/3/base 2025-08-26T20:08:26.0992315Z * [new branch] gh/v0i0/3/head -> origin/gh/v0i0/3/head 2025-08-26T20:08:26.0993284Z * [new branch] gh/v0i0/3/orig -> origin/gh/v0i0/3/orig 2025-08-26T20:08:26.0994319Z * [new branch] gh/v0i0/4/base -> origin/gh/v0i0/4/base 2025-08-26T20:08:26.0994807Z * [new branch] gh/v0i0/4/head -> origin/gh/v0i0/4/head 2025-08-26T20:08:26.0995425Z * [new branch] gh/v0i0/4/orig -> origin/gh/v0i0/4/orig 2025-08-26T20:08:26.0996886Z * [new branch] gh/v0i0/5/base -> origin/gh/v0i0/5/base 2025-08-26T20:08:26.0997194Z * [new branch] gh/v0i0/5/head -> origin/gh/v0i0/5/head 2025-08-26T20:08:26.1003941Z * [new branch] gh/v0i0/5/orig -> origin/gh/v0i0/5/orig 2025-08-26T20:08:26.1004382Z * [new branch] gh/v0i0/6/base -> origin/gh/v0i0/6/base 2025-08-26T20:08:26.1004533Z * [new branch] gh/v0i0/6/head -> origin/gh/v0i0/6/head 2025-08-26T20:08:26.1004669Z * [new branch] gh/v0i0/6/orig -> origin/gh/v0i0/6/orig 2025-08-26T20:08:26.1004812Z * [new branch] gh/v0i0/7/base -> origin/gh/v0i0/7/base 2025-08-26T20:08:26.1004940Z * [new branch] gh/v0i0/7/head -> origin/gh/v0i0/7/head 2025-08-26T20:08:26.1005294Z * [new branch] gh/v0i0/7/orig -> origin/gh/v0i0/7/orig 2025-08-26T20:08:26.1005461Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-08-26T20:08:26.1005603Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-08-26T20:08:26.1006259Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-08-26T20:08:26.1011424Z * [new branch] gh/vkuzo/4/base -> origin/gh/vkuzo/4/base 2025-08-26T20:08:26.1011603Z * [new branch] gh/vkuzo/4/head -> origin/gh/vkuzo/4/head 2025-08-26T20:08:26.1011743Z * [new branch] gh/vkuzo/4/orig -> origin/gh/vkuzo/4/orig 2025-08-26T20:08:26.1011921Z * [new branch] gh/wconstab/392/base -> origin/gh/wconstab/392/base 2025-08-26T20:08:26.1012072Z * [new branch] gh/wconstab/392/head -> origin/gh/wconstab/392/head 2025-08-26T20:08:26.1012234Z * [new branch] gh/wconstab/392/orig -> origin/gh/wconstab/392/orig 2025-08-26T20:08:26.1012431Z * [new branch] gh/wconstab/419/base -> origin/gh/wconstab/419/base 2025-08-26T20:08:26.1013348Z * [new branch] gh/wconstab/419/head -> origin/gh/wconstab/419/head 2025-08-26T20:08:26.1013797Z * [new branch] gh/wconstab/419/orig -> origin/gh/wconstab/419/orig 2025-08-26T20:08:26.1018486Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-08-26T20:08:26.1018671Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-08-26T20:08:26.1018833Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-08-26T20:08:26.1018989Z * [new branch] gh/wconstab/432/base -> origin/gh/wconstab/432/base 2025-08-26T20:08:26.1019134Z * [new branch] gh/wconstab/432/head -> origin/gh/wconstab/432/head 2025-08-26T20:08:26.1019296Z * [new branch] gh/wconstab/432/orig -> origin/gh/wconstab/432/orig 2025-08-26T20:08:26.1019658Z * [new branch] gh/wconstab/433/base -> origin/gh/wconstab/433/base 2025-08-26T20:08:26.1024203Z * [new branch] gh/wconstab/433/head -> origin/gh/wconstab/433/head 2025-08-26T20:08:26.1024392Z * [new branch] gh/wconstab/433/orig -> origin/gh/wconstab/433/orig 2025-08-26T20:08:26.1024754Z * [new branch] gh/wconstab/434/base -> origin/gh/wconstab/434/base 2025-08-26T20:08:26.1024909Z * [new branch] gh/wconstab/434/head -> origin/gh/wconstab/434/head 2025-08-26T20:08:26.1025050Z * [new branch] gh/wconstab/434/orig -> origin/gh/wconstab/434/orig 2025-08-26T20:08:26.1025195Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-08-26T20:08:26.1025384Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-08-26T20:08:26.1026541Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-08-26T20:08:26.1026907Z * [new branch] gh/wconstab/436/base -> origin/gh/wconstab/436/base 2025-08-26T20:08:26.1030702Z * [new branch] gh/wconstab/436/head -> origin/gh/wconstab/436/head 2025-08-26T20:08:26.1030908Z * [new branch] gh/wconstab/436/orig -> origin/gh/wconstab/436/orig 2025-08-26T20:08:26.1031060Z * [new branch] gh/wconstab/437/base -> origin/gh/wconstab/437/base 2025-08-26T20:08:26.1035908Z * [new branch] gh/wconstab/437/head -> origin/gh/wconstab/437/head 2025-08-26T20:08:26.1036106Z * [new branch] gh/wconstab/437/orig -> origin/gh/wconstab/437/orig 2025-08-26T20:08:26.1036271Z * [new branch] gh/wconstab/438/base -> origin/gh/wconstab/438/base 2025-08-26T20:08:26.1036600Z * [new branch] gh/wconstab/438/head -> origin/gh/wconstab/438/head 2025-08-26T20:08:26.1036765Z * [new branch] gh/wconstab/438/orig -> origin/gh/wconstab/438/orig 2025-08-26T20:08:26.1036930Z * [new branch] gh/wconstab/439/base -> origin/gh/wconstab/439/base 2025-08-26T20:08:26.1037084Z * [new branch] gh/wconstab/439/head -> origin/gh/wconstab/439/head 2025-08-26T20:08:26.1037250Z * [new branch] gh/wconstab/439/orig -> origin/gh/wconstab/439/orig 2025-08-26T20:08:26.1037400Z * [new branch] gh/wconstab/440/base -> origin/gh/wconstab/440/base 2025-08-26T20:08:26.1037695Z * [new branch] gh/wconstab/440/head -> origin/gh/wconstab/440/head 2025-08-26T20:08:26.1038079Z * [new branch] gh/wconstab/440/orig -> origin/gh/wconstab/440/orig 2025-08-26T20:08:26.1038553Z * [new branch] gh/wconstab/441/base -> origin/gh/wconstab/441/base 2025-08-26T20:08:26.1039792Z * [new branch] gh/wconstab/441/head -> origin/gh/wconstab/441/head 2025-08-26T20:08:26.1047429Z * [new branch] gh/wconstab/441/orig -> origin/gh/wconstab/441/orig 2025-08-26T20:08:26.1052614Z * [new branch] gh/wconstab/442/base -> origin/gh/wconstab/442/base 2025-08-26T20:08:26.1052806Z * [new branch] gh/wconstab/442/head -> origin/gh/wconstab/442/head 2025-08-26T20:08:26.1052988Z * [new branch] gh/wconstab/442/orig -> origin/gh/wconstab/442/orig 2025-08-26T20:08:26.1053143Z * [new branch] gh/wconstab/443/base -> origin/gh/wconstab/443/base 2025-08-26T20:08:26.1053306Z * [new branch] gh/wconstab/443/head -> origin/gh/wconstab/443/head 2025-08-26T20:08:26.1053453Z * [new branch] gh/wconstab/443/orig -> origin/gh/wconstab/443/orig 2025-08-26T20:08:26.1053604Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-08-26T20:08:26.1053774Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-08-26T20:08:26.1053952Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-08-26T20:08:26.1054106Z * [new branch] gh/wconstab/445/base -> origin/gh/wconstab/445/base 2025-08-26T20:08:26.1054259Z * [new branch] gh/wconstab/445/head -> origin/gh/wconstab/445/head 2025-08-26T20:08:26.1054572Z * [new branch] gh/wconstab/445/orig -> origin/gh/wconstab/445/orig 2025-08-26T20:08:26.1054729Z * [new branch] gh/weifengpy/27/base -> origin/gh/weifengpy/27/base 2025-08-26T20:08:26.1054879Z * [new branch] gh/weifengpy/27/head -> origin/gh/weifengpy/27/head 2025-08-26T20:08:26.1055036Z * [new branch] gh/weifengpy/27/orig -> origin/gh/weifengpy/27/orig 2025-08-26T20:08:26.1055203Z * [new branch] gh/weifengpy/30/base -> origin/gh/weifengpy/30/base 2025-08-26T20:08:26.1055367Z * [new branch] gh/weifengpy/30/head -> origin/gh/weifengpy/30/head 2025-08-26T20:08:26.1056021Z * [new branch] gh/weifengpy/30/orig -> origin/gh/weifengpy/30/orig 2025-08-26T20:08:26.1056232Z * [new branch] gh/weifengpy/33/base -> origin/gh/weifengpy/33/base 2025-08-26T20:08:26.1057063Z * [new branch] gh/weifengpy/33/head -> origin/gh/weifengpy/33/head 2025-08-26T20:08:26.1057512Z * [new branch] gh/weifengpy/33/orig -> origin/gh/weifengpy/33/orig 2025-08-26T20:08:26.1061483Z * [new branch] gh/williamwen42/196/base -> origin/gh/williamwen42/196/base 2025-08-26T20:08:26.1061678Z * [new branch] gh/williamwen42/196/head -> origin/gh/williamwen42/196/head 2025-08-26T20:08:26.1061831Z * [new branch] gh/williamwen42/196/orig -> origin/gh/williamwen42/196/orig 2025-08-26T20:08:26.1062144Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-08-26T20:08:26.1062750Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-08-26T20:08:26.1066951Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-08-26T20:08:26.1067154Z * [new branch] gh/williamwen42/258/base -> origin/gh/williamwen42/258/base 2025-08-26T20:08:26.1067340Z * [new branch] gh/williamwen42/258/head -> origin/gh/williamwen42/258/head 2025-08-26T20:08:26.1067505Z * [new branch] gh/williamwen42/258/orig -> origin/gh/williamwen42/258/orig 2025-08-26T20:08:26.1067661Z * [new branch] gh/williamwen42/260/base -> origin/gh/williamwen42/260/base 2025-08-26T20:08:26.1068054Z * [new branch] gh/williamwen42/260/head -> origin/gh/williamwen42/260/head 2025-08-26T20:08:26.1068558Z * [new branch] gh/williamwen42/260/orig -> origin/gh/williamwen42/260/orig 2025-08-26T20:08:26.1069108Z * [new branch] gh/williamwen42/261/base -> origin/gh/williamwen42/261/base 2025-08-26T20:08:26.1073971Z * [new branch] gh/williamwen42/261/head -> origin/gh/williamwen42/261/head 2025-08-26T20:08:26.1074178Z * [new branch] gh/williamwen42/261/orig -> origin/gh/williamwen42/261/orig 2025-08-26T20:08:26.1074352Z * [new branch] gh/williamwen42/263/base -> origin/gh/williamwen42/263/base 2025-08-26T20:08:26.1074507Z * [new branch] gh/williamwen42/263/head -> origin/gh/williamwen42/263/head 2025-08-26T20:08:26.1074669Z * [new branch] gh/williamwen42/263/orig -> origin/gh/williamwen42/263/orig 2025-08-26T20:08:26.1074820Z * [new branch] gh/williamwen42/264/base -> origin/gh/williamwen42/264/base 2025-08-26T20:08:26.1074978Z * [new branch] gh/williamwen42/264/head -> origin/gh/williamwen42/264/head 2025-08-26T20:08:26.1075318Z * [new branch] gh/williamwen42/264/orig -> origin/gh/williamwen42/264/orig 2025-08-26T20:08:26.1075503Z * [new branch] gh/williamwen42/265/base -> origin/gh/williamwen42/265/base 2025-08-26T20:08:26.1076799Z * [new branch] gh/williamwen42/265/head -> origin/gh/williamwen42/265/head 2025-08-26T20:08:26.1077256Z * [new branch] gh/williamwen42/265/orig -> origin/gh/williamwen42/265/orig 2025-08-26T20:08:26.1077896Z * [new branch] gh/williamwen42/266/base -> origin/gh/williamwen42/266/base 2025-08-26T20:08:26.1078864Z * [new branch] gh/williamwen42/266/head -> origin/gh/williamwen42/266/head 2025-08-26T20:08:26.1080416Z * [new branch] gh/williamwen42/266/orig -> origin/gh/williamwen42/266/orig 2025-08-26T20:08:26.1081011Z * [new branch] gh/williamwen42/267/base -> origin/gh/williamwen42/267/base 2025-08-26T20:08:26.1081167Z * [new branch] gh/williamwen42/267/head -> origin/gh/williamwen42/267/head 2025-08-26T20:08:26.1085280Z * [new branch] gh/williamwen42/267/orig -> origin/gh/williamwen42/267/orig 2025-08-26T20:08:26.1085480Z * [new branch] gh/williamwen42/268/base -> origin/gh/williamwen42/268/base 2025-08-26T20:08:26.1085663Z * [new branch] gh/williamwen42/268/head -> origin/gh/williamwen42/268/head 2025-08-26T20:08:26.1085863Z * [new branch] gh/williamwen42/268/orig -> origin/gh/williamwen42/268/orig 2025-08-26T20:08:26.1086035Z * [new branch] gh/williamwen42/269/base -> origin/gh/williamwen42/269/base 2025-08-26T20:08:26.1091955Z * [new branch] gh/williamwen42/269/head -> origin/gh/williamwen42/269/head 2025-08-26T20:08:26.1092158Z * [new branch] gh/williamwen42/269/orig -> origin/gh/williamwen42/269/orig 2025-08-26T20:08:26.1092369Z * [new branch] gh/williamwen42/270/base -> origin/gh/williamwen42/270/base 2025-08-26T20:08:26.1092705Z * [new branch] gh/williamwen42/270/head -> origin/gh/williamwen42/270/head 2025-08-26T20:08:26.1092888Z * [new branch] gh/williamwen42/270/orig -> origin/gh/williamwen42/270/orig 2025-08-26T20:08:26.1093059Z * [new branch] gh/williamwen42/271/base -> origin/gh/williamwen42/271/base 2025-08-26T20:08:26.1093238Z * [new branch] gh/williamwen42/271/head -> origin/gh/williamwen42/271/head 2025-08-26T20:08:26.1093402Z * [new branch] gh/williamwen42/271/orig -> origin/gh/williamwen42/271/orig 2025-08-26T20:08:26.1093571Z * [new branch] gh/williamwen42/272/base -> origin/gh/williamwen42/272/base 2025-08-26T20:08:26.1093730Z * [new branch] gh/williamwen42/272/head -> origin/gh/williamwen42/272/head 2025-08-26T20:08:26.1093902Z * [new branch] gh/williamwen42/272/orig -> origin/gh/williamwen42/272/orig 2025-08-26T20:08:26.1099893Z * [new branch] gh/williamwen42/273/base -> origin/gh/williamwen42/273/base 2025-08-26T20:08:26.1103219Z * [new branch] gh/williamwen42/273/head -> origin/gh/williamwen42/273/head 2025-08-26T20:08:26.1103497Z * [new branch] gh/williamwen42/273/orig -> origin/gh/williamwen42/273/orig 2025-08-26T20:08:26.1103795Z * [new branch] gh/williamwen42/274/base -> origin/gh/williamwen42/274/base 2025-08-26T20:08:26.1104062Z * [new branch] gh/williamwen42/274/head -> origin/gh/williamwen42/274/head 2025-08-26T20:08:26.1104274Z * [new branch] gh/williamwen42/274/orig -> origin/gh/williamwen42/274/orig 2025-08-26T20:08:26.1104538Z * [new branch] gh/williamwen42/275/base -> origin/gh/williamwen42/275/base 2025-08-26T20:08:26.1105309Z * [new branch] gh/williamwen42/275/head -> origin/gh/williamwen42/275/head 2025-08-26T20:08:26.1105515Z * [new branch] gh/williamwen42/276/base -> origin/gh/williamwen42/276/base 2025-08-26T20:08:26.1105712Z * [new branch] gh/williamwen42/276/head -> origin/gh/williamwen42/276/head 2025-08-26T20:08:26.1105878Z * [new branch] gh/williamwen42/276/orig -> origin/gh/williamwen42/276/orig 2025-08-26T20:08:26.1106980Z * [new branch] gh/williamwen42/277/base -> origin/gh/williamwen42/277/base 2025-08-26T20:08:26.1107244Z * [new branch] gh/williamwen42/277/head -> origin/gh/williamwen42/277/head 2025-08-26T20:08:26.1107991Z * [new branch] gh/williamwen42/277/orig -> origin/gh/williamwen42/277/orig 2025-08-26T20:08:26.1109677Z * [new branch] gh/williamwen42/278/base -> origin/gh/williamwen42/278/base 2025-08-26T20:08:26.1109869Z * [new branch] gh/williamwen42/278/head -> origin/gh/williamwen42/278/head 2025-08-26T20:08:26.1110286Z * [new branch] gh/williamwen42/278/orig -> origin/gh/williamwen42/278/orig 2025-08-26T20:08:26.1112241Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-08-26T20:08:26.1112450Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-08-26T20:08:26.1112622Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-08-26T20:08:26.1114779Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-08-26T20:08:26.1115275Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-08-26T20:08:26.1116286Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-08-26T20:08:26.1116591Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-08-26T20:08:26.1117993Z * [new branch] gh/xmfan/18/base -> origin/gh/xmfan/18/base 2025-08-26T20:08:26.1118362Z * [new branch] gh/xmfan/18/head -> origin/gh/xmfan/18/head 2025-08-26T20:08:26.1120026Z * [new branch] gh/xmfan/229/base -> origin/gh/xmfan/229/base 2025-08-26T20:08:26.1120183Z * [new branch] gh/xmfan/229/head -> origin/gh/xmfan/229/head 2025-08-26T20:08:26.1125737Z * [new branch] gh/xmfan/229/orig -> origin/gh/xmfan/229/orig 2025-08-26T20:08:26.1126374Z * [new branch] gh/xmfan/237/base -> origin/gh/xmfan/237/base 2025-08-26T20:08:26.1126704Z * [new branch] gh/xmfan/237/head -> origin/gh/xmfan/237/head 2025-08-26T20:08:26.1126927Z * [new branch] gh/xmfan/237/orig -> origin/gh/xmfan/237/orig 2025-08-26T20:08:26.1127192Z * [new branch] gh/xmfan/244/base -> origin/gh/xmfan/244/base 2025-08-26T20:08:26.1127348Z * [new branch] gh/xmfan/244/head -> origin/gh/xmfan/244/head 2025-08-26T20:08:26.1127581Z * [new branch] gh/xmfan/244/orig -> origin/gh/xmfan/244/orig 2025-08-26T20:08:26.1133334Z * [new branch] gh/xmfan/246/base -> origin/gh/xmfan/246/base 2025-08-26T20:08:26.1133676Z * [new branch] gh/xmfan/246/head -> origin/gh/xmfan/246/head 2025-08-26T20:08:26.1134048Z * [new branch] gh/xmfan/246/orig -> origin/gh/xmfan/246/orig 2025-08-26T20:08:26.1134315Z * [new branch] gh/xmfan/253/base -> origin/gh/xmfan/253/base 2025-08-26T20:08:26.1134568Z * [new branch] gh/xmfan/253/head -> origin/gh/xmfan/253/head 2025-08-26T20:08:26.1134803Z * [new branch] gh/xmfan/253/orig -> origin/gh/xmfan/253/orig 2025-08-26T20:08:26.1135286Z * [new branch] gh/xmfan/254/base -> origin/gh/xmfan/254/base 2025-08-26T20:08:26.1135433Z * [new branch] gh/xmfan/254/head -> origin/gh/xmfan/254/head 2025-08-26T20:08:26.1135705Z * [new branch] gh/xmfan/254/orig -> origin/gh/xmfan/254/orig 2025-08-26T20:08:26.1136457Z * [new branch] gh/xmfan/260/base -> origin/gh/xmfan/260/base 2025-08-26T20:08:26.1136630Z * [new branch] gh/xmfan/260/head -> origin/gh/xmfan/260/head 2025-08-26T20:08:26.1136785Z * [new branch] gh/xmfan/260/orig -> origin/gh/xmfan/260/orig 2025-08-26T20:08:26.1136931Z * [new branch] gh/xmfan/262/base -> origin/gh/xmfan/262/base 2025-08-26T20:08:26.1137290Z * [new branch] gh/xmfan/262/head -> origin/gh/xmfan/262/head 2025-08-26T20:08:26.1141950Z * [new branch] gh/xmfan/262/orig -> origin/gh/xmfan/262/orig 2025-08-26T20:08:26.1142124Z * [new branch] gh/xmfan/263/base -> origin/gh/xmfan/263/base 2025-08-26T20:08:26.1142520Z * [new branch] gh/xmfan/263/head -> origin/gh/xmfan/263/head 2025-08-26T20:08:26.1142686Z * [new branch] gh/xmfan/263/orig -> origin/gh/xmfan/263/orig 2025-08-26T20:08:26.1142838Z * [new branch] gh/xmfan/264/base -> origin/gh/xmfan/264/base 2025-08-26T20:08:26.1143030Z * [new branch] gh/xmfan/264/head -> origin/gh/xmfan/264/head 2025-08-26T20:08:26.1143538Z * [new branch] gh/xmfan/264/orig -> origin/gh/xmfan/264/orig 2025-08-26T20:08:26.1144140Z * [new branch] gh/xmfan/270/base -> origin/gh/xmfan/270/base 2025-08-26T20:08:26.1144334Z * [new branch] gh/xmfan/270/head -> origin/gh/xmfan/270/head 2025-08-26T20:08:26.1144479Z * [new branch] gh/xmfan/270/orig -> origin/gh/xmfan/270/orig 2025-08-26T20:08:26.1144619Z * [new branch] gh/xmfan/271/base -> origin/gh/xmfan/271/base 2025-08-26T20:08:26.1144766Z * [new branch] gh/xmfan/271/head -> origin/gh/xmfan/271/head 2025-08-26T20:08:26.1150727Z * [new branch] gh/xmfan/271/orig -> origin/gh/xmfan/271/orig 2025-08-26T20:08:26.1151221Z * [new branch] gh/xmfan/272/base -> origin/gh/xmfan/272/base 2025-08-26T20:08:26.1151385Z * [new branch] gh/xmfan/272/head -> origin/gh/xmfan/272/head 2025-08-26T20:08:26.1151540Z * [new branch] gh/xmfan/272/orig -> origin/gh/xmfan/272/orig 2025-08-26T20:08:26.1151838Z * [new branch] gh/xmfan/273/base -> origin/gh/xmfan/273/base 2025-08-26T20:08:26.1152008Z * [new branch] gh/xmfan/273/head -> origin/gh/xmfan/273/head 2025-08-26T20:08:26.1154408Z * [new branch] gh/xmfan/273/orig -> origin/gh/xmfan/273/orig 2025-08-26T20:08:26.1154581Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-08-26T20:08:26.1154822Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-08-26T20:08:26.1154980Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-08-26T20:08:26.1155137Z * [new branch] gh/xmfan/275/base -> origin/gh/xmfan/275/base 2025-08-26T20:08:26.1155283Z * [new branch] gh/xmfan/275/head -> origin/gh/xmfan/275/head 2025-08-26T20:08:26.1155441Z * [new branch] gh/xmfan/275/orig -> origin/gh/xmfan/275/orig 2025-08-26T20:08:26.1155591Z * [new branch] gh/xmfan/276/base -> origin/gh/xmfan/276/base 2025-08-26T20:08:26.1155752Z * [new branch] gh/xmfan/276/head -> origin/gh/xmfan/276/head 2025-08-26T20:08:26.1156188Z * [new branch] gh/xmfan/276/orig -> origin/gh/xmfan/276/orig 2025-08-26T20:08:26.1157489Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-08-26T20:08:26.1158049Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-08-26T20:08:26.1158763Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-08-26T20:08:26.1163540Z * [new branch] gh/xmfan/278/base -> origin/gh/xmfan/278/base 2025-08-26T20:08:26.1163884Z * [new branch] gh/xmfan/278/head -> origin/gh/xmfan/278/head 2025-08-26T20:08:26.1164122Z * [new branch] gh/xmfan/278/orig -> origin/gh/xmfan/278/orig 2025-08-26T20:08:26.1164296Z * [new branch] gh/xmfan/279/base -> origin/gh/xmfan/279/base 2025-08-26T20:08:26.1164606Z * [new branch] gh/xmfan/279/head -> origin/gh/xmfan/279/head 2025-08-26T20:08:26.1164893Z * [new branch] gh/xmfan/279/orig -> origin/gh/xmfan/279/orig 2025-08-26T20:08:26.1165048Z * [new branch] gh/xmfan/280/base -> origin/gh/xmfan/280/base 2025-08-26T20:08:26.1165413Z * [new branch] gh/xmfan/280/head -> origin/gh/xmfan/280/head 2025-08-26T20:08:26.1171443Z * [new branch] gh/xmfan/280/orig -> origin/gh/xmfan/280/orig 2025-08-26T20:08:26.1172053Z * [new branch] gh/xmfan/281/base -> origin/gh/xmfan/281/base 2025-08-26T20:08:26.1172261Z * [new branch] gh/xmfan/281/head -> origin/gh/xmfan/281/head 2025-08-26T20:08:26.1172428Z * [new branch] gh/xmfan/281/orig -> origin/gh/xmfan/281/orig 2025-08-26T20:08:26.1172728Z * [new branch] gh/xmfan/282/base -> origin/gh/xmfan/282/base 2025-08-26T20:08:26.1173355Z * [new branch] gh/xmfan/282/head -> origin/gh/xmfan/282/head 2025-08-26T20:08:26.1173576Z * [new branch] gh/xmfan/283/base -> origin/gh/xmfan/283/base 2025-08-26T20:08:26.1173809Z * [new branch] gh/xmfan/283/head -> origin/gh/xmfan/283/head 2025-08-26T20:08:26.1173982Z * [new branch] gh/xmfan/283/orig -> origin/gh/xmfan/283/orig 2025-08-26T20:08:26.1174200Z * [new branch] gh/xuanzhang816/14/base -> origin/gh/xuanzhang816/14/base 2025-08-26T20:08:26.1176697Z * [new branch] gh/xuanzhang816/14/head -> origin/gh/xuanzhang816/14/head 2025-08-26T20:08:26.1177059Z * [new branch] gh/xuanzhang816/14/orig -> origin/gh/xuanzhang816/14/orig 2025-08-26T20:08:26.1177232Z * [new branch] gh/xuanzhang816/19/base -> origin/gh/xuanzhang816/19/base 2025-08-26T20:08:26.1177400Z * [new branch] gh/xuanzhang816/19/head -> origin/gh/xuanzhang816/19/head 2025-08-26T20:08:26.1177565Z * [new branch] gh/xuanzhang816/19/orig -> origin/gh/xuanzhang816/19/orig 2025-08-26T20:08:26.1179382Z * [new branch] gh/xuanzhang816/22/base -> origin/gh/xuanzhang816/22/base 2025-08-26T20:08:26.1179578Z * [new branch] gh/xuanzhang816/22/head -> origin/gh/xuanzhang816/22/head 2025-08-26T20:08:26.1180092Z * [new branch] gh/xuanzhang816/22/orig -> origin/gh/xuanzhang816/22/orig 2025-08-26T20:08:26.1180361Z * [new branch] gh/xuanzhang816/23/base -> origin/gh/xuanzhang816/23/base 2025-08-26T20:08:26.1181690Z * [new branch] gh/xuanzhang816/23/head -> origin/gh/xuanzhang816/23/head 2025-08-26T20:08:26.1182020Z * [new branch] gh/xuanzhang816/23/orig -> origin/gh/xuanzhang816/23/orig 2025-08-26T20:08:26.1184159Z * [new branch] gh/xuanzhang816/24/base -> origin/gh/xuanzhang816/24/base 2025-08-26T20:08:26.1184409Z * [new branch] gh/xuanzhang816/24/head -> origin/gh/xuanzhang816/24/head 2025-08-26T20:08:26.1184607Z * [new branch] gh/xuanzhang816/24/orig -> origin/gh/xuanzhang816/24/orig 2025-08-26T20:08:26.1187393Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-08-26T20:08:26.1187697Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-08-26T20:08:26.1187879Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-08-26T20:08:26.1188123Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-08-26T20:08:26.1188320Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-08-26T20:08:26.1188826Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-08-26T20:08:26.1190366Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-08-26T20:08:26.1190781Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-08-26T20:08:26.1191186Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-08-26T20:08:26.1193643Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-08-26T20:08:26.1193840Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-08-26T20:08:26.1193989Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-08-26T20:08:26.1195476Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-08-26T20:08:26.1195633Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-08-26T20:08:26.1196387Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-08-26T20:08:26.1197532Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-08-26T20:08:26.1197955Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-08-26T20:08:26.1198783Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-08-26T20:08:26.1203226Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-08-26T20:08:26.1203573Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-08-26T20:08:26.1203728Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-08-26T20:08:26.1204130Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-08-26T20:08:26.1204293Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-08-26T20:08:26.1204446Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-08-26T20:08:26.1204835Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-08-26T20:08:26.1205078Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-08-26T20:08:26.1209605Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-08-26T20:08:26.1209828Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-08-26T20:08:26.1210002Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-08-26T20:08:26.1210155Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-08-26T20:08:26.1210325Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-08-26T20:08:26.1210476Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-08-26T20:08:26.1210944Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-08-26T20:08:26.1212815Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-08-26T20:08:26.1213133Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-08-26T20:08:26.1213317Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-08-26T20:08:26.1216173Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-08-26T20:08:26.1216363Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-08-26T20:08:26.1216552Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-08-26T20:08:26.1216734Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-08-26T20:08:26.1217817Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-08-26T20:08:26.1218458Z * [new branch] gh/yanbing-j/36/base -> origin/gh/yanbing-j/36/base 2025-08-26T20:08:26.1223579Z * [new branch] gh/yanbing-j/36/head -> origin/gh/yanbing-j/36/head 2025-08-26T20:08:26.1228618Z * [new branch] gh/yanbing-j/36/orig -> origin/gh/yanbing-j/36/orig 2025-08-26T20:08:26.1233047Z * [new branch] gh/yanbing-j/37/base -> origin/gh/yanbing-j/37/base 2025-08-26T20:08:26.1236149Z * [new branch] gh/yanbing-j/37/head -> origin/gh/yanbing-j/37/head 2025-08-26T20:08:26.1236319Z * [new branch] gh/yanbing-j/37/orig -> origin/gh/yanbing-j/37/orig 2025-08-26T20:08:26.1236742Z * [new branch] gh/yangw-dev/1/base -> origin/gh/yangw-dev/1/base 2025-08-26T20:08:26.1236920Z * [new branch] gh/yangw-dev/10/base -> origin/gh/yangw-dev/10/base 2025-08-26T20:08:26.1237063Z * [new branch] gh/yangw-dev/10/head -> origin/gh/yangw-dev/10/head 2025-08-26T20:08:26.1237227Z * [new branch] gh/yangw-dev/10/orig -> origin/gh/yangw-dev/10/orig 2025-08-26T20:08:26.1237371Z * [new branch] gh/yangw-dev/11/base -> origin/gh/yangw-dev/11/base 2025-08-26T20:08:26.1237550Z * [new branch] gh/yangw-dev/11/head -> origin/gh/yangw-dev/11/head 2025-08-26T20:08:26.1237692Z * [new branch] gh/yangw-dev/11/orig -> origin/gh/yangw-dev/11/orig 2025-08-26T20:08:26.1237826Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-08-26T20:08:26.1237969Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-08-26T20:08:26.1238217Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-08-26T20:08:26.1238364Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-08-26T20:08:26.1238500Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-08-26T20:08:26.1238634Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-08-26T20:08:26.1238784Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-08-26T20:08:26.1238921Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-08-26T20:08:26.1239070Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-08-26T20:08:26.1239288Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-08-26T20:08:26.1239442Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-08-26T20:08:26.1243510Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-08-26T20:08:26.1243663Z * [new branch] gh/yangw-dev/16/base -> origin/gh/yangw-dev/16/base 2025-08-26T20:08:26.1243803Z * [new branch] gh/yangw-dev/16/head -> origin/gh/yangw-dev/16/head 2025-08-26T20:08:26.1243940Z * [new branch] gh/yangw-dev/16/orig -> origin/gh/yangw-dev/16/orig 2025-08-26T20:08:26.1244083Z * [new branch] gh/yangw-dev/17/base -> origin/gh/yangw-dev/17/base 2025-08-26T20:08:26.1244218Z * [new branch] gh/yangw-dev/17/head -> origin/gh/yangw-dev/17/head 2025-08-26T20:08:26.1250450Z * [new branch] gh/yangw-dev/17/orig -> origin/gh/yangw-dev/17/orig 2025-08-26T20:08:26.1253832Z * [new branch] gh/yangw-dev/18/base -> origin/gh/yangw-dev/18/base 2025-08-26T20:08:26.1255610Z * [new branch] gh/yangw-dev/18/head -> origin/gh/yangw-dev/18/head 2025-08-26T20:08:26.1255774Z * [new branch] gh/yangw-dev/18/orig -> origin/gh/yangw-dev/18/orig 2025-08-26T20:08:26.1256513Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-08-26T20:08:26.1256681Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-08-26T20:08:26.1256997Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-08-26T20:08:26.1257156Z * [new branch] gh/yangw-dev/2/base -> origin/gh/yangw-dev/2/base 2025-08-26T20:08:26.1257313Z * [new branch] gh/yangw-dev/2/head -> origin/gh/yangw-dev/2/head 2025-08-26T20:08:26.1257461Z * [new branch] gh/yangw-dev/20/base -> origin/gh/yangw-dev/20/base 2025-08-26T20:08:26.1257613Z * [new branch] gh/yangw-dev/20/head -> origin/gh/yangw-dev/20/head 2025-08-26T20:08:26.1257782Z * [new branch] gh/yangw-dev/20/orig -> origin/gh/yangw-dev/20/orig 2025-08-26T20:08:26.1257921Z * [new branch] gh/yangw-dev/21/base -> origin/gh/yangw-dev/21/base 2025-08-26T20:08:26.1258063Z * [new branch] gh/yangw-dev/21/head -> origin/gh/yangw-dev/21/head 2025-08-26T20:08:26.1258209Z * [new branch] gh/yangw-dev/21/orig -> origin/gh/yangw-dev/21/orig 2025-08-26T20:08:26.1258353Z * [new branch] gh/yangw-dev/22/base -> origin/gh/yangw-dev/22/base 2025-08-26T20:08:26.1260048Z * [new branch] gh/yangw-dev/22/head -> origin/gh/yangw-dev/22/head 2025-08-26T20:08:26.1260426Z * [new branch] gh/yangw-dev/22/orig -> origin/gh/yangw-dev/22/orig 2025-08-26T20:08:26.1260619Z * [new branch] gh/yangw-dev/23/base -> origin/gh/yangw-dev/23/base 2025-08-26T20:08:26.1260771Z * [new branch] gh/yangw-dev/23/head -> origin/gh/yangw-dev/23/head 2025-08-26T20:08:26.1261092Z * [new branch] gh/yangw-dev/23/orig -> origin/gh/yangw-dev/23/orig 2025-08-26T20:08:26.1261251Z * [new branch] gh/yangw-dev/3/base -> origin/gh/yangw-dev/3/base 2025-08-26T20:08:26.1263711Z * [new branch] gh/yangw-dev/3/head -> origin/gh/yangw-dev/3/head 2025-08-26T20:08:26.1264210Z * [new branch] gh/yangw-dev/4/base -> origin/gh/yangw-dev/4/base 2025-08-26T20:08:26.1264414Z * [new branch] gh/yangw-dev/4/head -> origin/gh/yangw-dev/4/head 2025-08-26T20:08:26.1264570Z * [new branch] gh/yangw-dev/5/base -> origin/gh/yangw-dev/5/base 2025-08-26T20:08:26.1267922Z * [new branch] gh/yangw-dev/5/head -> origin/gh/yangw-dev/5/head 2025-08-26T20:08:26.1268108Z * [new branch] gh/yangw-dev/6/base -> origin/gh/yangw-dev/6/base 2025-08-26T20:08:26.1268261Z * [new branch] gh/yangw-dev/6/head -> origin/gh/yangw-dev/6/head 2025-08-26T20:08:26.1268417Z * [new branch] gh/yangw-dev/7/base -> origin/gh/yangw-dev/7/base 2025-08-26T20:08:26.1268562Z * [new branch] gh/yangw-dev/7/head -> origin/gh/yangw-dev/7/head 2025-08-26T20:08:26.1272946Z * [new branch] gh/yangw-dev/8/base -> origin/gh/yangw-dev/8/base 2025-08-26T20:08:26.1273544Z * [new branch] gh/yangw-dev/8/head -> origin/gh/yangw-dev/8/head 2025-08-26T20:08:26.1273759Z * [new branch] gh/yangw-dev/8/orig -> origin/gh/yangw-dev/8/orig 2025-08-26T20:08:26.1273922Z * [new branch] gh/yangw-dev/9/base -> origin/gh/yangw-dev/9/base 2025-08-26T20:08:26.1274087Z * [new branch] gh/yangw-dev/9/head -> origin/gh/yangw-dev/9/head 2025-08-26T20:08:26.1274243Z * [new branch] gh/yangw-dev/9/orig -> origin/gh/yangw-dev/9/orig 2025-08-26T20:08:26.1274401Z * [new branch] gh/ydwu4/233/base -> origin/gh/ydwu4/233/base 2025-08-26T20:08:26.1274563Z * [new branch] gh/ydwu4/233/head -> origin/gh/ydwu4/233/head 2025-08-26T20:08:26.1274705Z * [new branch] gh/ydwu4/233/orig -> origin/gh/ydwu4/233/orig 2025-08-26T20:08:26.1275842Z * [new branch] gh/ydwu4/246/base -> origin/gh/ydwu4/246/base 2025-08-26T20:08:26.1276147Z * [new branch] gh/ydwu4/246/head -> origin/gh/ydwu4/246/head 2025-08-26T20:08:26.1277151Z * [new branch] gh/ydwu4/246/orig -> origin/gh/ydwu4/246/orig 2025-08-26T20:08:26.1278267Z * [new branch] gh/ydwu4/253/base -> origin/gh/ydwu4/253/base 2025-08-26T20:08:26.1278811Z * [new branch] gh/ydwu4/253/head -> origin/gh/ydwu4/253/head 2025-08-26T20:08:26.1279837Z * [new branch] gh/ydwu4/253/orig -> origin/gh/ydwu4/253/orig 2025-08-26T20:08:26.1283537Z * [new branch] gh/ydwu4/255/base -> origin/gh/ydwu4/255/base 2025-08-26T20:08:26.1283729Z * [new branch] gh/ydwu4/255/head -> origin/gh/ydwu4/255/head 2025-08-26T20:08:26.1283878Z * [new branch] gh/ydwu4/255/orig -> origin/gh/ydwu4/255/orig 2025-08-26T20:08:26.1284017Z * [new branch] gh/ydwu4/259/base -> origin/gh/ydwu4/259/base 2025-08-26T20:08:26.1284181Z * [new branch] gh/ydwu4/259/head -> origin/gh/ydwu4/259/head 2025-08-26T20:08:26.1284356Z * [new branch] gh/ydwu4/259/orig -> origin/gh/ydwu4/259/orig 2025-08-26T20:08:26.1287232Z * [new branch] gh/ydwu4/262/base -> origin/gh/ydwu4/262/base 2025-08-26T20:08:26.1287565Z * [new branch] gh/ydwu4/262/head -> origin/gh/ydwu4/262/head 2025-08-26T20:08:26.1287761Z * [new branch] gh/ydwu4/262/orig -> origin/gh/ydwu4/262/orig 2025-08-26T20:08:26.1287941Z * [new branch] gh/ydwu4/263/base -> origin/gh/ydwu4/263/base 2025-08-26T20:08:26.1288598Z * [new branch] gh/ydwu4/263/head -> origin/gh/ydwu4/263/head 2025-08-26T20:08:26.1289266Z * [new branch] gh/ydwu4/263/orig -> origin/gh/ydwu4/263/orig 2025-08-26T20:08:26.1293265Z * [new branch] gh/ydwu4/269/base -> origin/gh/ydwu4/269/base 2025-08-26T20:08:26.1293615Z * [new branch] gh/ydwu4/269/head -> origin/gh/ydwu4/269/head 2025-08-26T20:08:26.1293837Z * [new branch] gh/ydwu4/269/orig -> origin/gh/ydwu4/269/orig 2025-08-26T20:08:26.1294065Z * [new branch] gh/ydwu4/270/base -> origin/gh/ydwu4/270/base 2025-08-26T20:08:26.1294221Z * [new branch] gh/ydwu4/270/head -> origin/gh/ydwu4/270/head 2025-08-26T20:08:26.1294451Z * [new branch] gh/ydwu4/270/orig -> origin/gh/ydwu4/270/orig 2025-08-26T20:08:26.1297063Z * [new branch] gh/ydwu4/272/base -> origin/gh/ydwu4/272/base 2025-08-26T20:08:26.1297403Z * [new branch] gh/ydwu4/272/head -> origin/gh/ydwu4/272/head 2025-08-26T20:08:26.1297648Z * [new branch] gh/ydwu4/272/orig -> origin/gh/ydwu4/272/orig 2025-08-26T20:08:26.1298047Z * [new branch] gh/ydwu4/275/base -> origin/gh/ydwu4/275/base 2025-08-26T20:08:26.1298859Z * [new branch] gh/ydwu4/275/head -> origin/gh/ydwu4/275/head 2025-08-26T20:08:26.1301580Z * [new branch] gh/ydwu4/275/orig -> origin/gh/ydwu4/275/orig 2025-08-26T20:08:26.1301962Z * [new branch] gh/ydwu4/276/base -> origin/gh/ydwu4/276/base 2025-08-26T20:08:26.1302644Z * [new branch] gh/ydwu4/276/head -> origin/gh/ydwu4/276/head 2025-08-26T20:08:26.1302808Z * [new branch] gh/ydwu4/276/orig -> origin/gh/ydwu4/276/orig 2025-08-26T20:08:26.1303095Z * [new branch] gh/ydwu4/279/base -> origin/gh/ydwu4/279/base 2025-08-26T20:08:26.1303603Z * [new branch] gh/ydwu4/279/head -> origin/gh/ydwu4/279/head 2025-08-26T20:08:26.1304869Z * [new branch] gh/ydwu4/279/orig -> origin/gh/ydwu4/279/orig 2025-08-26T20:08:26.1305445Z * [new branch] gh/ydwu4/283/base -> origin/gh/ydwu4/283/base 2025-08-26T20:08:26.1306168Z * [new branch] gh/ydwu4/283/head -> origin/gh/ydwu4/283/head 2025-08-26T20:08:26.1307028Z * [new branch] gh/ydwu4/283/orig -> origin/gh/ydwu4/283/orig 2025-08-26T20:08:26.1308272Z * [new branch] gh/ydwu4/289/base -> origin/gh/ydwu4/289/base 2025-08-26T20:08:26.1308414Z * [new branch] gh/ydwu4/289/head -> origin/gh/ydwu4/289/head 2025-08-26T20:08:26.1309670Z * [new branch] gh/ydwu4/289/orig -> origin/gh/ydwu4/289/orig 2025-08-26T20:08:26.1310244Z * [new branch] gh/ydwu4/290/base -> origin/gh/ydwu4/290/base 2025-08-26T20:08:26.1311503Z * [new branch] gh/ydwu4/290/head -> origin/gh/ydwu4/290/head 2025-08-26T20:08:26.1311642Z * [new branch] gh/ydwu4/290/orig -> origin/gh/ydwu4/290/orig 2025-08-26T20:08:26.1313345Z * [new branch] gh/ydwu4/291/base -> origin/gh/ydwu4/291/base 2025-08-26T20:08:26.1313749Z * [new branch] gh/ydwu4/291/head -> origin/gh/ydwu4/291/head 2025-08-26T20:08:26.1315288Z * [new branch] gh/ydwu4/291/orig -> origin/gh/ydwu4/291/orig 2025-08-26T20:08:26.1315556Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-08-26T20:08:26.1316364Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-08-26T20:08:26.1316676Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-08-26T20:08:26.1317921Z * [new branch] gh/ydwu4/293/base -> origin/gh/ydwu4/293/base 2025-08-26T20:08:26.1318344Z * [new branch] gh/ydwu4/293/head -> origin/gh/ydwu4/293/head 2025-08-26T20:08:26.1319435Z * [new branch] gh/ydwu4/293/orig -> origin/gh/ydwu4/293/orig 2025-08-26T20:08:26.1320669Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-08-26T20:08:26.1325262Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-08-26T20:08:26.1325410Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-08-26T20:08:26.1325544Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-08-26T20:08:26.1325678Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-08-26T20:08:26.1325803Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-08-26T20:08:26.1325931Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-08-26T20:08:26.1330983Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-08-26T20:08:26.1331589Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-08-26T20:08:26.1331754Z * [new branch] gh/ydwu4/300/base -> origin/gh/ydwu4/300/base 2025-08-26T20:08:26.1331915Z * [new branch] gh/ydwu4/300/head -> origin/gh/ydwu4/300/head 2025-08-26T20:08:26.1335482Z * [new branch] gh/ydwu4/300/orig -> origin/gh/ydwu4/300/orig 2025-08-26T20:08:26.1335664Z * [new branch] gh/ydwu4/301/base -> origin/gh/ydwu4/301/base 2025-08-26T20:08:26.1336351Z * [new branch] gh/ydwu4/301/head -> origin/gh/ydwu4/301/head 2025-08-26T20:08:26.1339896Z * [new branch] gh/ydwu4/301/orig -> origin/gh/ydwu4/301/orig 2025-08-26T20:08:26.1343631Z * [new branch] gh/ydwu4/302/base -> origin/gh/ydwu4/302/base 2025-08-26T20:08:26.1343789Z * [new branch] gh/ydwu4/302/head -> origin/gh/ydwu4/302/head 2025-08-26T20:08:26.1343937Z * [new branch] gh/ydwu4/302/orig -> origin/gh/ydwu4/302/orig 2025-08-26T20:08:26.1344076Z * [new branch] gh/ydwu4/303/base -> origin/gh/ydwu4/303/base 2025-08-26T20:08:26.1344363Z * [new branch] gh/ydwu4/303/head -> origin/gh/ydwu4/303/head 2025-08-26T20:08:26.1344487Z * [new branch] gh/ydwu4/303/orig -> origin/gh/ydwu4/303/orig 2025-08-26T20:08:26.1344622Z * [new branch] gh/ydwu4/304/base -> origin/gh/ydwu4/304/base 2025-08-26T20:08:26.1344745Z * [new branch] gh/ydwu4/304/head -> origin/gh/ydwu4/304/head 2025-08-26T20:08:26.1344879Z * [new branch] gh/ydwu4/304/orig -> origin/gh/ydwu4/304/orig 2025-08-26T20:08:26.1345015Z * [new branch] gh/ydwu4/305/base -> origin/gh/ydwu4/305/base 2025-08-26T20:08:26.1345154Z * [new branch] gh/ydwu4/305/head -> origin/gh/ydwu4/305/head 2025-08-26T20:08:26.1345285Z * [new branch] gh/ydwu4/305/orig -> origin/gh/ydwu4/305/orig 2025-08-26T20:08:26.1345417Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-08-26T20:08:26.1345558Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-08-26T20:08:26.1345687Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-08-26T20:08:26.1348620Z * [new branch] gh/ydwu4/307/base -> origin/gh/ydwu4/307/base 2025-08-26T20:08:26.1353690Z * [new branch] gh/ydwu4/307/head -> origin/gh/ydwu4/307/head 2025-08-26T20:08:26.1353854Z * [new branch] gh/ydwu4/307/orig -> origin/gh/ydwu4/307/orig 2025-08-26T20:08:26.1354168Z * [new branch] gh/ydwu4/308/base -> origin/gh/ydwu4/308/base 2025-08-26T20:08:26.1354313Z * [new branch] gh/ydwu4/308/head -> origin/gh/ydwu4/308/head 2025-08-26T20:08:26.1354476Z * [new branch] gh/ydwu4/308/orig -> origin/gh/ydwu4/308/orig 2025-08-26T20:08:26.1354618Z * [new branch] gh/ydwu4/309/base -> origin/gh/ydwu4/309/base 2025-08-26T20:08:26.1354779Z * [new branch] gh/ydwu4/309/head -> origin/gh/ydwu4/309/head 2025-08-26T20:08:26.1354935Z * [new branch] gh/ydwu4/309/orig -> origin/gh/ydwu4/309/orig 2025-08-26T20:08:26.1355076Z * [new branch] gh/ydwu4/310/base -> origin/gh/ydwu4/310/base 2025-08-26T20:08:26.1355286Z * [new branch] gh/ydwu4/310/head -> origin/gh/ydwu4/310/head 2025-08-26T20:08:26.1355472Z * [new branch] gh/ydwu4/310/orig -> origin/gh/ydwu4/310/orig 2025-08-26T20:08:26.1355616Z * [new branch] gh/ydwu4/311/base -> origin/gh/ydwu4/311/base 2025-08-26T20:08:26.1355763Z * [new branch] gh/ydwu4/311/head -> origin/gh/ydwu4/311/head 2025-08-26T20:08:26.1355906Z * [new branch] gh/ydwu4/311/orig -> origin/gh/ydwu4/311/orig 2025-08-26T20:08:26.1356043Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-08-26T20:08:26.1356357Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-08-26T20:08:26.1357391Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-08-26T20:08:26.1358672Z * [new branch] gh/ydwu4/313/base -> origin/gh/ydwu4/313/base 2025-08-26T20:08:26.1359690Z * [new branch] gh/ydwu4/313/head -> origin/gh/ydwu4/313/head 2025-08-26T20:08:26.1360236Z * [new branch] gh/ydwu4/313/orig -> origin/gh/ydwu4/313/orig 2025-08-26T20:08:26.1364498Z * [new branch] gh/ydwu4/314/base -> origin/gh/ydwu4/314/base 2025-08-26T20:08:26.1364649Z * [new branch] gh/ydwu4/314/head -> origin/gh/ydwu4/314/head 2025-08-26T20:08:26.1364794Z * [new branch] gh/ydwu4/314/orig -> origin/gh/ydwu4/314/orig 2025-08-26T20:08:26.1364926Z * [new branch] gh/ydwu4/315/base -> origin/gh/ydwu4/315/base 2025-08-26T20:08:26.1365153Z * [new branch] gh/ydwu4/315/head -> origin/gh/ydwu4/315/head 2025-08-26T20:08:26.1365285Z * [new branch] gh/ydwu4/315/orig -> origin/gh/ydwu4/315/orig 2025-08-26T20:08:26.1367629Z * [new branch] gh/ydwu4/316/base -> origin/gh/ydwu4/316/base 2025-08-26T20:08:26.1367763Z * [new branch] gh/ydwu4/316/head -> origin/gh/ydwu4/316/head 2025-08-26T20:08:26.1371958Z * [new branch] gh/ydwu4/316/orig -> origin/gh/ydwu4/316/orig 2025-08-26T20:08:26.1372118Z * [new branch] gh/ydwu4/317/base -> origin/gh/ydwu4/317/base 2025-08-26T20:08:26.1372389Z * [new branch] gh/ydwu4/317/head -> origin/gh/ydwu4/317/head 2025-08-26T20:08:26.1372530Z * [new branch] gh/ydwu4/317/orig -> origin/gh/ydwu4/317/orig 2025-08-26T20:08:26.1372659Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-08-26T20:08:26.1372805Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-08-26T20:08:26.1377643Z * [new branch] gh/yf225/171/base -> origin/gh/yf225/171/base 2025-08-26T20:08:26.1377814Z * [new branch] gh/yf225/171/head -> origin/gh/yf225/171/head 2025-08-26T20:08:26.1377948Z * [new branch] gh/yf225/171/orig -> origin/gh/yf225/171/orig 2025-08-26T20:08:26.1378073Z * [new branch] gh/yf225/172/base -> origin/gh/yf225/172/base 2025-08-26T20:08:26.1378371Z * [new branch] gh/yf225/172/head -> origin/gh/yf225/172/head 2025-08-26T20:08:26.1378505Z * [new branch] gh/yf225/172/orig -> origin/gh/yf225/172/orig 2025-08-26T20:08:26.1383994Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-08-26T20:08:26.1384308Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-08-26T20:08:26.1384576Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-08-26T20:08:26.1384814Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-08-26T20:08:26.1384985Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-08-26T20:08:26.1385233Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-08-26T20:08:26.1386611Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-08-26T20:08:26.1386916Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-08-26T20:08:26.1387116Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-08-26T20:08:26.1387336Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-08-26T20:08:26.1387512Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-08-26T20:08:26.1387694Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-08-26T20:08:26.1392729Z * [new branch] gh/ysiraichi/79/base -> origin/gh/ysiraichi/79/base 2025-08-26T20:08:26.1393083Z * [new branch] gh/ysiraichi/79/head -> origin/gh/ysiraichi/79/head 2025-08-26T20:08:26.1393334Z * [new branch] gh/ysiraichi/79/orig -> origin/gh/ysiraichi/79/orig 2025-08-26T20:08:26.1393675Z * [new branch] gh/ysiraichi/81/base -> origin/gh/ysiraichi/81/base 2025-08-26T20:08:26.1393850Z * [new branch] gh/ysiraichi/81/head -> origin/gh/ysiraichi/81/head 2025-08-26T20:08:26.1394369Z * [new branch] gh/ysiraichi/81/orig -> origin/gh/ysiraichi/81/orig 2025-08-26T20:08:26.1394940Z * [new branch] gh/ysiraichi/88/base -> origin/gh/ysiraichi/88/base 2025-08-26T20:08:26.1395114Z * [new branch] gh/ysiraichi/88/head -> origin/gh/ysiraichi/88/head 2025-08-26T20:08:26.1395453Z * [new branch] gh/ysiraichi/88/orig -> origin/gh/ysiraichi/88/orig 2025-08-26T20:08:26.1395626Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2025-08-26T20:08:26.1395791Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2025-08-26T20:08:26.1395954Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2025-08-26T20:08:26.1396149Z * [new branch] gh/zhxchen17/31/base -> origin/gh/zhxchen17/31/base 2025-08-26T20:08:26.1400130Z * [new branch] gh/zhxchen17/31/head -> origin/gh/zhxchen17/31/head 2025-08-26T20:08:26.1404161Z * [new branch] gh/zhxchen17/31/orig -> origin/gh/zhxchen17/31/orig 2025-08-26T20:08:26.1404319Z * [new branch] gh/zhxchen17/34/base -> origin/gh/zhxchen17/34/base 2025-08-26T20:08:26.1404467Z * [new branch] gh/zhxchen17/34/head -> origin/gh/zhxchen17/34/head 2025-08-26T20:08:26.1404630Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2025-08-26T20:08:26.1404775Z * [new branch] gh/zhxchen17/35/head -> origin/gh/zhxchen17/35/head 2025-08-26T20:08:26.1410431Z * [new branch] gh/zhxchen17/36/base -> origin/gh/zhxchen17/36/base 2025-08-26T20:08:26.1415937Z * [new branch] gh/zhxchen17/36/head -> origin/gh/zhxchen17/36/head 2025-08-26T20:08:26.1417350Z * [new branch] gh/zhxchen17/36/orig -> origin/gh/zhxchen17/36/orig 2025-08-26T20:08:26.1417545Z * [new branch] gh/zhxchen17/37/base -> origin/gh/zhxchen17/37/base 2025-08-26T20:08:26.1417702Z * [new branch] gh/zhxchen17/37/head -> origin/gh/zhxchen17/37/head 2025-08-26T20:08:26.1417848Z * [new branch] gh/zhxchen17/37/orig -> origin/gh/zhxchen17/37/orig 2025-08-26T20:08:26.1418031Z * [new branch] gh/zhxchen17/38/base -> origin/gh/zhxchen17/38/base 2025-08-26T20:08:26.1418178Z * [new branch] gh/zhxchen17/38/head -> origin/gh/zhxchen17/38/head 2025-08-26T20:08:26.1418344Z * [new branch] gh/zhxchen17/38/orig -> origin/gh/zhxchen17/38/orig 2025-08-26T20:08:26.1418500Z * [new branch] gh/zhxchen17/39/base -> origin/gh/zhxchen17/39/base 2025-08-26T20:08:26.1418653Z * [new branch] gh/zhxchen17/39/head -> origin/gh/zhxchen17/39/head 2025-08-26T20:08:26.1418808Z * [new branch] gh/zhxchen17/39/orig -> origin/gh/zhxchen17/39/orig 2025-08-26T20:08:26.1418949Z * [new branch] gh/zhxchen17/40/base -> origin/gh/zhxchen17/40/base 2025-08-26T20:08:26.1419105Z * [new branch] gh/zhxchen17/40/head -> origin/gh/zhxchen17/40/head 2025-08-26T20:08:26.1419240Z * [new branch] gh/zhxchen17/40/orig -> origin/gh/zhxchen17/40/orig 2025-08-26T20:08:26.1419382Z * [new branch] gh/zhxchen17/41/base -> origin/gh/zhxchen17/41/base 2025-08-26T20:08:26.1426559Z * [new branch] gh/zhxchen17/41/head -> origin/gh/zhxchen17/41/head 2025-08-26T20:08:26.1426743Z * [new branch] gh/zhxchen17/41/orig -> origin/gh/zhxchen17/41/orig 2025-08-26T20:08:26.1426911Z * [new branch] gh/zhxchen17/42/base -> origin/gh/zhxchen17/42/base 2025-08-26T20:08:26.1427060Z * [new branch] gh/zhxchen17/42/head -> origin/gh/zhxchen17/42/head 2025-08-26T20:08:26.1427244Z * [new branch] gh/zhxchen17/42/orig -> origin/gh/zhxchen17/42/orig 2025-08-26T20:08:26.1427391Z * [new branch] gh/zhxchen17/43/base -> origin/gh/zhxchen17/43/base 2025-08-26T20:08:26.1432004Z * [new branch] gh/zhxchen17/43/head -> origin/gh/zhxchen17/43/head 2025-08-26T20:08:26.1432447Z * [new branch] gh/zhxchen17/43/orig -> origin/gh/zhxchen17/43/orig 2025-08-26T20:08:26.1432980Z * [new branch] gh/zklaus/10/base -> origin/gh/zklaus/10/base 2025-08-26T20:08:26.1433442Z * [new branch] gh/zklaus/10/head -> origin/gh/zklaus/10/head 2025-08-26T20:08:26.1433595Z * [new branch] gh/zklaus/10/orig -> origin/gh/zklaus/10/orig 2025-08-26T20:08:26.1433743Z * [new branch] gh/zklaus/11/base -> origin/gh/zklaus/11/base 2025-08-26T20:08:26.1433881Z * [new branch] gh/zklaus/11/head -> origin/gh/zklaus/11/head 2025-08-26T20:08:26.1434032Z * [new branch] gh/zklaus/11/orig -> origin/gh/zklaus/11/orig 2025-08-26T20:08:26.1434183Z * [new branch] gh/zklaus/12/base -> origin/gh/zklaus/12/base 2025-08-26T20:08:26.1434319Z * [new branch] gh/zklaus/12/head -> origin/gh/zklaus/12/head 2025-08-26T20:08:26.1434459Z * [new branch] gh/zklaus/12/orig -> origin/gh/zklaus/12/orig 2025-08-26T20:08:26.1434603Z * [new branch] gh/zklaus/14/base -> origin/gh/zklaus/14/base 2025-08-26T20:08:26.1434741Z * [new branch] gh/zklaus/14/head -> origin/gh/zklaus/14/head 2025-08-26T20:08:26.1434879Z * [new branch] gh/zklaus/14/orig -> origin/gh/zklaus/14/orig 2025-08-26T20:08:26.1436624Z * [new branch] gh/zklaus/15/base -> origin/gh/zklaus/15/base 2025-08-26T20:08:26.1436988Z * [new branch] gh/zklaus/15/head -> origin/gh/zklaus/15/head 2025-08-26T20:08:26.1437215Z * [new branch] gh/zklaus/15/orig -> origin/gh/zklaus/15/orig 2025-08-26T20:08:26.1438480Z * [new branch] gh/zklaus/16/base -> origin/gh/zklaus/16/base 2025-08-26T20:08:26.1438641Z * [new branch] gh/zklaus/16/head -> origin/gh/zklaus/16/head 2025-08-26T20:08:26.1440154Z * [new branch] gh/zklaus/16/orig -> origin/gh/zklaus/16/orig 2025-08-26T20:08:26.1441412Z * [new branch] gh/zklaus/17/base -> origin/gh/zklaus/17/base 2025-08-26T20:08:26.1442049Z * [new branch] gh/zklaus/17/head -> origin/gh/zklaus/17/head 2025-08-26T20:08:26.1446979Z * [new branch] gh/zklaus/17/orig -> origin/gh/zklaus/17/orig 2025-08-26T20:08:26.1447152Z * [new branch] gh/zklaus/18/base -> origin/gh/zklaus/18/base 2025-08-26T20:08:26.1447644Z * [new branch] gh/zklaus/18/head -> origin/gh/zklaus/18/head 2025-08-26T20:08:26.1447831Z * [new branch] gh/zklaus/18/orig -> origin/gh/zklaus/18/orig 2025-08-26T20:08:26.1447981Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-08-26T20:08:26.1448119Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-08-26T20:08:26.1448263Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-08-26T20:08:26.1453844Z * [new branch] gh/zklaus/7/base -> origin/gh/zklaus/7/base 2025-08-26T20:08:26.1454021Z * [new branch] gh/zklaus/7/head -> origin/gh/zklaus/7/head 2025-08-26T20:08:26.1454162Z * [new branch] gh/zklaus/7/orig -> origin/gh/zklaus/7/orig 2025-08-26T20:08:26.1454291Z * [new branch] gh/zklaus/9/base -> origin/gh/zklaus/9/base 2025-08-26T20:08:26.1454420Z * [new branch] gh/zklaus/9/head -> origin/gh/zklaus/9/head 2025-08-26T20:08:26.1454571Z * [new branch] gh/zklaus/9/orig -> origin/gh/zklaus/9/orig 2025-08-26T20:08:26.1454720Z * [new branch] gh/zou3519/1175/base -> origin/gh/zou3519/1175/base 2025-08-26T20:08:26.1454868Z * [new branch] gh/zou3519/1175/head -> origin/gh/zou3519/1175/head 2025-08-26T20:08:26.1455006Z * [new branch] gh/zou3519/1175/orig -> origin/gh/zou3519/1175/orig 2025-08-26T20:08:26.1455296Z * [new branch] gh/zou3519/1177/base -> origin/gh/zou3519/1177/base 2025-08-26T20:08:26.1455440Z * [new branch] gh/zou3519/1177/head -> origin/gh/zou3519/1177/head 2025-08-26T20:08:26.1461120Z * [new branch] gh/zou3519/1177/orig -> origin/gh/zou3519/1177/orig 2025-08-26T20:08:26.1463331Z * [new branch] gh/zou3519/1188/base -> origin/gh/zou3519/1188/base 2025-08-26T20:08:26.1463668Z * [new branch] gh/zou3519/1188/head -> origin/gh/zou3519/1188/head 2025-08-26T20:08:26.1466926Z * [new branch] gh/zou3519/1188/orig -> origin/gh/zou3519/1188/orig 2025-08-26T20:08:26.1467256Z * [new branch] gh/zou3519/1189/base -> origin/gh/zou3519/1189/base 2025-08-26T20:08:26.1467438Z * [new branch] gh/zou3519/1189/head -> origin/gh/zou3519/1189/head 2025-08-26T20:08:26.1467574Z * [new branch] gh/zou3519/1189/orig -> origin/gh/zou3519/1189/orig 2025-08-26T20:08:26.1467703Z * [new branch] gh/zou3519/1190/base -> origin/gh/zou3519/1190/base 2025-08-26T20:08:26.1467841Z * [new branch] gh/zou3519/1190/head -> origin/gh/zou3519/1190/head 2025-08-26T20:08:26.1467967Z * [new branch] gh/zou3519/1190/orig -> origin/gh/zou3519/1190/orig 2025-08-26T20:08:26.1468105Z * [new branch] gh/zou3519/1191/base -> origin/gh/zou3519/1191/base 2025-08-26T20:08:26.1468378Z * [new branch] gh/zou3519/1191/head -> origin/gh/zou3519/1191/head 2025-08-26T20:08:26.1468514Z * [new branch] gh/zou3519/1191/orig -> origin/gh/zou3519/1191/orig 2025-08-26T20:08:26.1468652Z * [new branch] gh/zou3519/1192/base -> origin/gh/zou3519/1192/base 2025-08-26T20:08:26.1468788Z * [new branch] gh/zou3519/1192/head -> origin/gh/zou3519/1192/head 2025-08-26T20:08:26.1468925Z * [new branch] gh/zou3519/1192/orig -> origin/gh/zou3519/1192/orig 2025-08-26T20:08:26.1471701Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-08-26T20:08:26.1472024Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-08-26T20:08:26.1472203Z * [new branch] gh/zpcore/10/base -> origin/gh/zpcore/10/base 2025-08-26T20:08:26.1472353Z * [new branch] gh/zpcore/10/head -> origin/gh/zpcore/10/head 2025-08-26T20:08:26.1472643Z * [new branch] gh/zpcore/10/orig -> origin/gh/zpcore/10/orig 2025-08-26T20:08:26.1472815Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-08-26T20:08:26.1474619Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-08-26T20:08:26.1475037Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-08-26T20:08:26.1475417Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-08-26T20:08:26.1476818Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-08-26T20:08:26.1476991Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-08-26T20:08:26.1478281Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-08-26T20:08:26.1478521Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-08-26T20:08:26.1479775Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-08-26T20:08:26.1481035Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-08-26T20:08:26.1481719Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-08-26T20:08:26.1482674Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-08-26T20:08:26.1482940Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-08-26T20:08:26.1484088Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-08-26T20:08:26.1484425Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-08-26T20:08:26.1486447Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-08-26T20:08:26.1486623Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-08-26T20:08:26.1486791Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-08-26T20:08:26.1487349Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-08-26T20:08:26.1488343Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-08-26T20:08:26.1488560Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-08-26T20:08:26.1489830Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-08-26T20:08:26.1490418Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-08-26T20:08:26.1491802Z * [new branch] gh/zpcore/9/head -> origin/gh/zpcore/9/head 2025-08-26T20:08:26.1491944Z * [new branch] gh/zpcore/9/orig -> origin/gh/zpcore/9/orig 2025-08-26T20:08:26.1493759Z * [new branch] google-main -> origin/google-main 2025-08-26T20:08:26.1494939Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-08-26T20:08:26.1495096Z * [new branch] guangyey/host_alloc -> origin/guangyey/host_alloc 2025-08-26T20:08:26.1496734Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-08-26T20:08:26.1503181Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-08-26T20:08:26.1503427Z * [new branch] haozhe/bf16-dynamic-shape -> origin/haozhe/bf16-dynamic-shape 2025-08-26T20:08:26.1504113Z * [new branch] hc_baseline -> origin/hc_baseline 2025-08-26T20:08:26.1505364Z * [new branch] headeronlyScalarType -> origin/headeronlyScalarType 2025-08-26T20:08:26.1505904Z * [new branch] hf_update -> origin/hf_update 2025-08-26T20:08:26.1507333Z * [new branch] hhh_decomp_mul -> origin/hhh_decomp_mul 2025-08-26T20:08:26.1507579Z * [new branch] hhh_rand -> origin/hhh_rand 2025-08-26T20:08:26.1510585Z * [new branch] hoy/mmsplitk -> origin/hoy/mmsplitk 2025-08-26T20:08:26.1510931Z * [new branch] hoy/triton-PR3973 -> origin/hoy/triton-PR3973 2025-08-26T20:08:26.1511181Z * [new branch] hoy/triton-coalescing-baseline -> origin/hoy/triton-coalescing-baseline 2025-08-26T20:08:26.1511378Z * [new branch] hoy/triton-coalescing-min -> origin/hoy/triton-coalescing-min 2025-08-26T20:08:26.1511564Z * [new branch] hoy/triton-coalescing-new -> origin/hoy/triton-coalescing-new 2025-08-26T20:08:26.1515508Z * [new branch] hoy/triton-coalescing-vec -> origin/hoy/triton-coalescing-vec 2025-08-26T20:08:26.1515702Z * [new branch] inductordecompfix -> origin/inductordecompfix 2025-08-26T20:08:26.1515841Z * [new branch] inline -> origin/inline 2025-08-26T20:08:26.1516003Z * [new branch] inlining -> origin/inlining 2025-08-26T20:08:26.1516159Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-08-26T20:08:26.1516291Z * [new branch] int8_sdpa -> origin/int8_sdpa 2025-08-26T20:08:26.1516434Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-08-26T20:08:26.1516836Z * [new branch] issue#58739 -> origin/issue#58739 2025-08-26T20:08:26.1516989Z * [new branch] issue-154849 -> origin/issue-154849 2025-08-26T20:08:26.1518722Z * [new branch] ivanov/cherry-pick-ckpt-fixes -> origin/ivanov/cherry-pick-ckpt-fixes 2025-08-26T20:08:26.1519850Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-08-26T20:08:26.1527226Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-08-26T20:08:26.1527507Z * [new branch] justinchu/attention-tests -> origin/justinchu/attention-tests 2025-08-26T20:08:26.1527722Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-08-26T20:08:26.1527955Z * [new branch] justinchu/ort-122 -> origin/justinchu/ort-122 2025-08-26T20:08:26.1528231Z * [new branch] justinchuby/JitScalarType -> origin/justinchuby/JitScalarType 2025-08-26T20:08:26.1528475Z * [new branch] justinchuby/dynamo-true -> origin/justinchuby/dynamo-true 2025-08-26T20:08:26.1528758Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-08-26T20:08:26.1529240Z * [new branch] kainan_test -> origin/kainan_test 2025-08-26T20:08:26.1529447Z * [new branch] learnablebias -> origin/learnablebias 2025-08-26T20:08:26.1529892Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-08-26T20:08:26.1531396Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-08-26T20:08:26.1531559Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-08-26T20:08:26.1531716Z * [new branch] liaoxuan/tags_issue -> origin/liaoxuan/tags_issue 2025-08-26T20:08:26.1531910Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-08-26T20:08:26.1532089Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-08-26T20:08:26.1532239Z * [new branch] lintbuilddocker -> origin/lintbuilddocker 2025-08-26T20:08:26.1532375Z * [new branch] llama4-stable -> origin/llama4-stable 2025-08-26T20:08:26.1533138Z * [new branch] logdetfix -> origin/logdetfix 2025-08-26T20:08:26.1534482Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-08-26T20:08:26.1535575Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-08-26T20:08:26.1535734Z * [new branch] lucaskabela/fix_157452 -> origin/lucaskabela/fix_157452 2025-08-26T20:08:26.1536734Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-08-26T20:08:26.1536970Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-08-26T20:08:26.1538131Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-08-26T20:08:26.1538300Z * [new branch] lucaskabela/issue_120648 -> origin/lucaskabela/issue_120648 2025-08-26T20:08:26.1539610Z * [new branch] lucaskabela/misc_typing_dynamo -> origin/lucaskabela/misc_typing_dynamo 2025-08-26T20:08:26.1542876Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-08-26T20:08:26.1543048Z * [new branch] lucaskabela/registry_fix -> origin/lucaskabela/registry_fix 2025-08-26T20:08:26.1543315Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-08-26T20:08:26.1543548Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-08-26T20:08:26.1543704Z * [new branch] lucaskabela/type_guards -> origin/lucaskabela/type_guards 2025-08-26T20:08:26.1543870Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-08-26T20:08:26.1549662Z * [new branch] lucaskabela/typing_compile_autograd -> origin/lucaskabela/typing_compile_autograd 2025-08-26T20:08:26.1550285Z * [new branch] lucaskabela/typing_output_graph -> origin/lucaskabela/typing_output_graph 2025-08-26T20:08:26.1550574Z * [new branch] lucaskabela/typing_source_guard -> origin/lucaskabela/typing_source_guard 2025-08-26T20:08:26.1550838Z * [new branch] lucaskabela/typing_symbolic_convert -> origin/lucaskabela/typing_symbolic_convert 2025-08-26T20:08:26.1551028Z * [new branch] lucaskabela/typing_utils.py -> origin/lucaskabela/typing_utils.py 2025-08-26T20:08:26.1551281Z * [new branch] lucaskabela/typing_utils_improvements -> origin/lucaskabela/typing_utils_improvements 2025-08-26T20:08:26.1551406Z * [new branch] main -> origin/main 2025-08-26T20:08:26.1553648Z * [new branch] main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests 2025-08-26T20:08:26.1556402Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-08-26T20:08:26.1556565Z * [new branch] malfet-patch-11 -> origin/malfet-patch-11 2025-08-26T20:08:26.1556927Z * [new branch] malfet-patch-12 -> origin/malfet-patch-12 2025-08-26T20:08:26.1557076Z * [new branch] malfet-patch-14 -> origin/malfet-patch-14 2025-08-26T20:08:26.1557226Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-08-26T20:08:26.1557377Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-08-26T20:08:26.1557535Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-08-26T20:08:26.1557680Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-08-26T20:08:26.1557813Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-08-26T20:08:26.1557953Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-08-26T20:08:26.1558085Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-08-26T20:08:26.1558290Z * [new branch] malfet-patch-9 -> origin/malfet-patch-9 2025-08-26T20:08:26.1559747Z * [new branch] malfet/delete-upsteam-cuda -> origin/malfet/delete-upsteam-cuda 2025-08-26T20:08:26.1563074Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-08-26T20:08:26.1563301Z * [new branch] manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps 2025-08-26T20:08:26.1563597Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-08-26T20:08:26.1563763Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-08-26T20:08:26.1563898Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-08-26T20:08:26.1564077Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-08-26T20:08:26.1565543Z * [new branch] mlazos/backup-test-branch -> origin/mlazos/backup-test-branch 2025-08-26T20:08:26.1575084Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-08-26T20:08:26.1577049Z * [new branch] mlazos/baseline -> origin/mlazos/baseline 2025-08-26T20:08:26.1577733Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-08-26T20:08:26.1577934Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-08-26T20:08:26.1578238Z * [new branch] mlazos/better-msg -> origin/mlazos/better-msg 2025-08-26T20:08:26.1578389Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-08-26T20:08:26.1578532Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-08-26T20:08:26.1578669Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-08-26T20:08:26.1578809Z * [new branch] mlazos/ck2 -> origin/mlazos/ck2 2025-08-26T20:08:26.1578984Z * [new branch] mlazos/combokernels -> origin/mlazos/combokernels 2025-08-26T20:08:26.1579143Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-08-26T20:08:26.1579288Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-08-26T20:08:26.1579490Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-08-26T20:08:26.1579703Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-08-26T20:08:26.1579863Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-08-26T20:08:26.1580039Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-08-26T20:08:26.1580204Z * [new branch] mlazos/data-gather -> origin/mlazos/data-gather 2025-08-26T20:08:26.1580359Z * [new branch] mlazos/data-ptrs2 -> origin/mlazos/data-ptrs2 2025-08-26T20:08:26.1581702Z * [new branch] mlazos/data-ptrs3 -> origin/mlazos/data-ptrs3 2025-08-26T20:08:26.1581969Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-08-26T20:08:26.1582133Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-08-26T20:08:26.1582278Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-08-26T20:08:26.1582422Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-08-26T20:08:26.1582599Z * [new branch] mlazos/disable-closures -> origin/mlazos/disable-closures 2025-08-26T20:08:26.1588595Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-08-26T20:08:26.1593080Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-08-26T20:08:26.1594967Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-08-26T20:08:26.1595127Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-08-26T20:08:26.1595296Z * [new branch] mlazos/exp_disable -> origin/mlazos/exp_disable 2025-08-26T20:08:26.1595476Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-08-26T20:08:26.1595644Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-08-26T20:08:26.1595780Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-08-26T20:08:26.1595931Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-08-26T20:08:26.1596103Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-08-26T20:08:26.1596441Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-08-26T20:08:26.1596601Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-08-26T20:08:26.1596751Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-08-26T20:08:26.1596888Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-08-26T20:08:26.1597034Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-08-26T20:08:26.1597359Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-08-26T20:08:26.1597511Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-08-26T20:08:26.1597648Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-08-26T20:08:26.1597789Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-08-26T20:08:26.1597942Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-08-26T20:08:26.1598075Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-08-26T20:08:26.1598219Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-08-26T20:08:26.1598386Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-08-26T20:08:26.1599965Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-08-26T20:08:26.1600130Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-08-26T20:08:26.1601112Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-08-26T20:08:26.1601471Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-08-26T20:08:26.1605364Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-08-26T20:08:26.1605524Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-08-26T20:08:26.1605652Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-08-26T20:08:26.1605989Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-08-26T20:08:26.1606117Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-08-26T20:08:26.1611255Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-08-26T20:08:26.1611557Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-08-26T20:08:26.1611752Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-08-26T20:08:26.1611948Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-08-26T20:08:26.1612498Z * [new branch] mlazos/hop-modes -> origin/mlazos/hop-modes 2025-08-26T20:08:26.1612710Z * [new branch] mlazos/init-per-param -> origin/mlazos/init-per-param 2025-08-26T20:08:26.1612877Z * [new branch] mlazos/init_per_param -> origin/mlazos/init_per_param 2025-08-26T20:08:26.1613047Z * [new branch] mlazos/less-guards -> origin/mlazos/less-guards 2025-08-26T20:08:26.1613223Z * [new branch] mlazos/lr-composibility -> origin/mlazos/lr-composibility 2025-08-26T20:08:26.1613418Z * [new branch] mlazos/main -> origin/mlazos/main 2025-08-26T20:08:26.1613654Z * [new branch] mlazos/main-test-enablement -> origin/mlazos/main-test-enablement 2025-08-26T20:08:26.1613805Z * [new branch] mlazos/main2 -> origin/mlazos/main2 2025-08-26T20:08:26.1613934Z * [new branch] mlazos/mcg -> origin/mlazos/mcg 2025-08-26T20:08:26.1618080Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-08-26T20:08:26.1618259Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-08-26T20:08:26.1618408Z * [new branch] mlazos/mlazos/ck2 -> origin/mlazos/mlazos/ck2 2025-08-26T20:08:26.1618613Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-08-26T20:08:26.1618793Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-08-26T20:08:26.1619093Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-08-26T20:08:26.1619234Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-08-26T20:08:26.1620149Z * [new branch] mlazos/more-tests -> origin/mlazos/more-tests 2025-08-26T20:08:26.1624627Z * [new branch] mlazos/no-cpp -> origin/mlazos/no-cpp 2025-08-26T20:08:26.1629187Z * [new branch] mlazos/no-init-group-handling -> origin/mlazos/no-init-group-handling 2025-08-26T20:08:26.1634102Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-08-26T20:08:26.1634738Z * [new branch] mlazos/opt-bench-exp2 -> origin/mlazos/opt-bench-exp2 2025-08-26T20:08:26.1634941Z * [new branch] mlazos/opt-incr -> origin/mlazos/opt-incr 2025-08-26T20:08:26.1635271Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-08-26T20:08:26.1635430Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-08-26T20:08:26.1635584Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-08-26T20:08:26.1635753Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-08-26T20:08:26.1635882Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-08-26T20:08:26.1636018Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-08-26T20:08:26.1636178Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-08-26T20:08:26.1636347Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-08-26T20:08:26.1636664Z * [new branch] mlazos/sub-param-fix -> origin/mlazos/sub-param-fix 2025-08-26T20:08:26.1636808Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-08-26T20:08:26.1636974Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-08-26T20:08:26.1637104Z * [new branch] mlazos/test -> origin/mlazos/test 2025-08-26T20:08:26.1637253Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-08-26T20:08:26.1637416Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-08-26T20:08:26.1637600Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-08-26T20:08:26.1637767Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-08-26T20:08:26.1637917Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-08-26T20:08:26.1638075Z * [new branch] mlazos/topo-fix -> origin/mlazos/topo-fix 2025-08-26T20:08:26.1638232Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-08-26T20:08:26.1638418Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-08-26T20:08:26.1638572Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-08-26T20:08:26.1638734Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-08-26T20:08:26.1638896Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-08-26T20:08:26.1639046Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-08-26T20:08:26.1639528Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-08-26T20:08:26.1639720Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-08-26T20:08:26.1639879Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-08-26T20:08:26.1643347Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-08-26T20:08:26.1643502Z * [new branch] modify-setupvllm -> origin/modify-setupvllm 2025-08-26T20:08:26.1643679Z * [new branch] move-theme-out-docker -> origin/move-theme-out-docker 2025-08-26T20:08:26.1643941Z * [new branch] mps-linear-1d -> origin/mps-linear-1d 2025-08-26T20:08:26.1644083Z * [new branch] msaroufim/be1 -> origin/msaroufim/be1 2025-08-26T20:08:26.1647523Z * [new branch] msaroufim/cn_path -> origin/msaroufim/cn_path 2025-08-26T20:08:26.1647749Z * [new branch] msaroufim/dtensorfusedadam -> origin/msaroufim/dtensorfusedadam 2025-08-26T20:08:26.1647903Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-08-26T20:08:26.1648074Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-08-26T20:08:26.1648202Z * [new branch] muon_dev -> origin/muon_dev 2025-08-26T20:08:26.1651506Z * [new branch] muon_dev_1 -> origin/muon_dev_1 2025-08-26T20:08:26.1651792Z * [new branch] new-modifiy-setupvllm -> origin/new-modifiy-setupvllm 2025-08-26T20:08:26.1652036Z * [new branch] new-setupvllm -> origin/new-setupvllm 2025-08-26T20:08:26.1652261Z * [new branch] newtest-base -> origin/newtest-base 2025-08-26T20:08:26.1652436Z * [new branch] ngimel/cat_perf -> origin/ngimel/cat_perf 2025-08-26T20:08:26.1652690Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-08-26T20:08:26.1653369Z * [new branch] ngimel/fabric_check -> origin/ngimel/fabric_check 2025-08-26T20:08:26.1658654Z * [new branch] ngimel/fabric_driver_version -> origin/ngimel/fabric_driver_version 2025-08-26T20:08:26.1660365Z * [new branch] ngimel/fabric_fix -> origin/ngimel/fabric_fix 2025-08-26T20:08:26.1660535Z * [new branch] ngimel/fabric_symm -> origin/ngimel/fabric_symm 2025-08-26T20:08:26.1660944Z * [new branch] ngimel/fix_driver_init_error -> origin/ngimel/fix_driver_init_error 2025-08-26T20:08:26.1661141Z * [new branch] ngimel/fix_nccl_segment_seg -> origin/ngimel/fix_nccl_segment_seg 2025-08-26T20:08:26.1661279Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-08-26T20:08:26.1661455Z * [new branch] ngimel/grouped_mm_checks -> origin/ngimel/grouped_mm_checks 2025-08-26T20:08:26.1661607Z * [new branch] ngimel/guardfabric -> origin/ngimel/guardfabric 2025-08-26T20:08:26.1661751Z * [new branch] ngimel/modeguard -> origin/ngimel/modeguard 2025-08-26T20:08:26.1661923Z * [new branch] ngimel/multicast_fix -> origin/ngimel/multicast_fix 2025-08-26T20:08:26.1662084Z * [new branch] ngimel/unbind_multimem -> origin/ngimel/unbind_multimem 2025-08-26T20:08:26.1664682Z * [new branch] nightly -> origin/nightly 2025-08-26T20:08:26.1664869Z * [new branch] nmacchioni-patch-10 -> origin/nmacchioni-patch-10 2025-08-26T20:08:26.1665295Z * [new branch] nmacchioni-patch-7 -> origin/nmacchioni-patch-7 2025-08-26T20:08:26.1665466Z * [new branch] nmacchioni-patch-8 -> origin/nmacchioni-patch-8 2025-08-26T20:08:26.1665614Z * [new branch] nmacchioni-patch-9 -> origin/nmacchioni-patch-9 2025-08-26T20:08:26.1665792Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-08-26T20:08:26.1670839Z * [new branch] nweidia/enable-B200-inductor-nightly-ci -> origin/nweidia/enable-B200-inductor-nightly-ci 2025-08-26T20:08:26.1674258Z * [new branch] one-off -> origin/one-off 2025-08-26T20:08:26.1674815Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-08-26T20:08:26.1674998Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-08-26T20:08:26.1675326Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-08-26T20:08:26.1675467Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-08-26T20:08:26.1675620Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-08-26T20:08:26.1675756Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-08-26T20:08:26.1675899Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-08-26T20:08:26.1676042Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-08-26T20:08:26.1676173Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-08-26T20:08:26.1676315Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-08-26T20:08:26.1677317Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-08-26T20:08:26.1677644Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-08-26T20:08:26.1677799Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-08-26T20:08:26.1679026Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-08-26T20:08:26.1679986Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-08-26T20:08:26.1680445Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-08-26T20:08:26.1681796Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-08-26T20:08:26.1683196Z * [new branch] oulgen/fx_graph -> origin/oulgen/fx_graph 2025-08-26T20:08:26.1683788Z * [new branch] padded-tensor -> origin/padded-tensor 2025-08-26T20:08:26.1684356Z * [new branch] parallel_cat -> origin/parallel_cat 2025-08-26T20:08:26.1684892Z * [new branch] pca2 -> origin/pca2 2025-08-26T20:08:26.1686448Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-08-26T20:08:26.1686895Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-08-26T20:08:26.1691668Z * [new branch] pianpwk/dde_repeat_cat -> origin/pianpwk/dde_repeat_cat 2025-08-26T20:08:26.1692300Z * [new branch] pianpwk/invalidate_fake_memo -> origin/pianpwk/invalidate_fake_memo 2025-08-26T20:08:26.1692556Z * [new branch] pianpwk/max_1_strides -> origin/pianpwk/max_1_strides 2025-08-26T20:08:26.1692741Z * [new branch] pianpwk/nonzero_memo -> origin/pianpwk/nonzero_memo 2025-08-26T20:08:26.1692991Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-08-26T20:08:26.1693208Z * [new branch] pianpwk/oblivious_should_swap -> origin/pianpwk/oblivious_should_swap 2025-08-26T20:08:26.1693419Z * [new branch] pianpwk/oblivious_slice_forward -> origin/pianpwk/oblivious_slice_forward 2025-08-26T20:08:26.1693595Z * [new branch] pianpwk/oblivious_where -> origin/pianpwk/oblivious_where 2025-08-26T20:08:26.1694217Z * [new branch] pianpwk/param_static_pgo -> origin/pianpwk/param_static_pgo 2025-08-26T20:08:26.1694836Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-08-26T20:08:26.1695081Z * [new branch] pianpwk/remove_guard_fail_break -> origin/pianpwk/remove_guard_fail_break 2025-08-26T20:08:26.1695897Z * [new branch] pianpwk/slice_fresh_symbols -> origin/pianpwk/slice_fresh_symbols 2025-08-26T20:08:26.1696703Z * [new branch] pianpwk/test_slice_fake_impl -> origin/pianpwk/test_slice_fake_impl 2025-08-26T20:08:26.1698112Z * [new branch] pianpwk/unbacked_channels_last -> origin/pianpwk/unbacked_channels_last 2025-08-26T20:08:26.1698518Z * [new branch] pianpwk/unbacked_safe_conv1d -> origin/pianpwk/unbacked_safe_conv1d 2025-08-26T20:08:26.1698709Z * [new branch] pianpwk/unbacked_sdpa_flash -> origin/pianpwk/unbacked_sdpa_flash 2025-08-26T20:08:26.1699938Z * [new branch] pianpwk/unbacked_should_swap -> origin/pianpwk/unbacked_should_swap 2025-08-26T20:08:26.1700297Z * [new branch] pianpwk/unbacked_should_swap_2 -> origin/pianpwk/unbacked_should_swap_2 2025-08-26T20:08:26.1701418Z * [new branch] pianpwk/unbacked_slice_binding -> origin/pianpwk/unbacked_slice_binding 2025-08-26T20:08:26.1701650Z * [new branch] pianpwk/unbacked_slice_forward -> origin/pianpwk/unbacked_slice_forward 2025-08-26T20:08:26.1703069Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-08-26T20:08:26.1703265Z * [new branch] pianpwk/whitelist_optimizer -> origin/pianpwk/whitelist_optimizer 2025-08-26T20:08:26.1707541Z * [new branch] pin-torchao -> origin/pin-torchao 2025-08-26T20:08:26.1707848Z * [new branch] piz/fall_back_missing_0716 -> origin/piz/fall_back_missing_0716 2025-08-26T20:08:26.1714170Z * [new branch] piz/fix_sort_ -> origin/piz/fix_sort_ 2025-08-26T20:08:26.1714530Z * [new branch] piz/improve_scatter_0808 -> origin/piz/improve_scatter_0808 2025-08-26T20:08:26.1714716Z * [new branch] pool-separate -> origin/pool-separate 2025-08-26T20:08:26.1715171Z * [new branch] pr-156087 -> origin/pr-156087 2025-08-26T20:08:26.1715446Z * [new branch] pr/131860 -> origin/pr/131860 2025-08-26T20:08:26.1716142Z * [new branch] predispatch_to -> origin/predispatch_to 2025-08-26T20:08:26.1716319Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-08-26T20:08:26.1716544Z * [new branch] pt2e-cache-model-device -> origin/pt2e-cache-model-device 2025-08-26T20:08:26.1716688Z * [new branch] pyobjectslot -> origin/pyobjectslot 2025-08-26T20:08:26.1716868Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-08-26T20:08:26.1717042Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-08-26T20:08:26.1717175Z * [new branch] quint-bits -> origin/quint-bits 2025-08-26T20:08:26.1717619Z * [new branch] release/1.10 -> origin/release/1.10 2025-08-26T20:08:26.1717974Z * [new branch] release/1.11 -> origin/release/1.11 2025-08-26T20:08:26.1719912Z * [new branch] release/1.12 -> origin/release/1.12 2025-08-26T20:08:26.1720059Z * [new branch] release/1.13 -> origin/release/1.13 2025-08-26T20:08:26.1720292Z * [new branch] release/1.4 -> origin/release/1.4 2025-08-26T20:08:26.1720756Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-08-26T20:08:26.1723299Z * [new branch] release/1.5 -> origin/release/1.5 2025-08-26T20:08:26.1723462Z * [new branch] release/1.6 -> origin/release/1.6 2025-08-26T20:08:26.1723603Z * [new branch] release/1.7 -> origin/release/1.7 2025-08-26T20:08:26.1723935Z * [new branch] release/1.8 -> origin/release/1.8 2025-08-26T20:08:26.1727807Z * [new branch] release/1.9 -> origin/release/1.9 2025-08-26T20:08:26.1728076Z * [new branch] release/2.0 -> origin/release/2.0 2025-08-26T20:08:26.1734041Z * [new branch] release/2.1 -> origin/release/2.1 2025-08-26T20:08:26.1737087Z * [new branch] release/2.2 -> origin/release/2.2 2025-08-26T20:08:26.1739002Z * [new branch] release/2.3 -> origin/release/2.3 2025-08-26T20:08:26.1739320Z * [new branch] release/2.4 -> origin/release/2.4 2025-08-26T20:08:26.1739499Z * [new branch] release/2.5 -> origin/release/2.5 2025-08-26T20:08:26.1739661Z * [new branch] release/2.6 -> origin/release/2.6 2025-08-26T20:08:26.1739803Z * [new branch] release/2.7 -> origin/release/2.7 2025-08-26T20:08:26.1739942Z * [new branch] release/2.8 -> origin/release/2.8 2025-08-26T20:08:26.1740137Z * [new branch] release_notes -> origin/release_notes 2025-08-26T20:08:26.1740351Z * [new branch] remove-actionable-label -> origin/remove-actionable-label 2025-08-26T20:08:26.1740570Z * [new branch] remove-ao -> origin/remove-ao 2025-08-26T20:08:26.1741040Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-08-26T20:08:26.1741283Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-08-26T20:08:26.1741516Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-08-26T20:08:26.1741740Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-08-26T20:08:26.1742097Z * [new branch] replace-pytorch-labs-20250812-204125 -> origin/replace-pytorch-labs-20250812-204125 2025-08-26T20:08:26.1742320Z * [new branch] replace-pytorch-labs-20250812-205624 -> origin/replace-pytorch-labs-20250812-205624 2025-08-26T20:08:26.1742583Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-08-26T20:08:26.1742933Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-08-26T20:08:26.1744363Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-08-26T20:08:26.1744800Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-08-26T20:08:26.1748038Z * [new branch] revert-direct-updates -> origin/revert-direct-updates 2025-08-26T20:08:26.1748238Z * [new branch] rocm-monitoring -> origin/rocm-monitoring 2025-08-26T20:08:26.1748544Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-08-26T20:08:26.1748894Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-08-26T20:08:26.1749161Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-08-26T20:08:26.1749569Z * [new branch] rzou/njt -> origin/rzou/njt 2025-08-26T20:08:26.1750834Z * [new branch] rzou/operator -> origin/rzou/operator 2025-08-26T20:08:26.1751026Z * [new branch] rzou/pca -> origin/rzou/pca 2025-08-26T20:08:26.1752117Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-08-26T20:08:26.1752445Z * [new branch] rzou/setup_context -> origin/rzou/setup_context 2025-08-26T20:08:26.1753971Z * [new branch] sanchitintel/refactor_aten_int8_woq_gemm -> origin/sanchitintel/refactor_aten_int8_woq_gemm 2025-08-26T20:08:26.1754829Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-08-26T20:08:26.1755084Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-08-26T20:08:26.1755929Z * [new branch] save -> origin/save 2025-08-26T20:08:26.1757117Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-08-26T20:08:26.1758063Z * [new branch] seemethere-patch-1 -> origin/seemethere-patch-1 2025-08-26T20:08:26.1758390Z * [new branch] setup-torchci -> origin/setup-torchci 2025-08-26T20:08:26.1759772Z * [new branch] setupvllm -> origin/setupvllm 2025-08-26T20:08:26.1760183Z * [new branch] share_and_pin_fork -> origin/share_and_pin_fork 2025-08-26T20:08:26.1764042Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-08-26T20:08:26.1764236Z * [new branch] shikaili_fp8_allgather -> origin/shikaili_fp8_allgather 2025-08-26T20:08:26.1764415Z * [new branch] shoumikhin-patch-12 -> origin/shoumikhin-patch-12 2025-08-26T20:08:26.1764609Z * [new branch] simplify-fq-per-channel -> origin/simplify-fq-per-channel 2025-08-26T20:08:26.1764774Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-08-26T20:08:26.1765329Z * [new branch] sqzhang/flight4 -> origin/sqzhang/flight4 2025-08-26T20:08:26.1766307Z * [new branch] sqzhang/flight4plus -> origin/sqzhang/flight4plus 2025-08-26T20:08:26.1771767Z * [new branch] sraikund/record_funct_test -> origin/sraikund/record_funct_test 2025-08-26T20:08:26.1772143Z * [new branch] sraikund16/test -> origin/sraikund16/test 2025-08-26T20:08:26.1772367Z * [new branch] stablize-compilation-time -> origin/stablize-compilation-time 2025-08-26T20:08:26.1772553Z * [new branch] standalone-templates -> origin/standalone-templates 2025-08-26T20:08:26.1772737Z * [new branch] standalone_package_weights -> origin/standalone_package_weights 2025-08-26T20:08:26.1772906Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-08-26T20:08:26.1773057Z * [new branch] subgraph_fuse -> origin/subgraph_fuse 2025-08-26T20:08:26.1773237Z * [new branch] support-uv-in-collect_env -> origin/support-uv-in-collect_env 2025-08-26T20:08:26.1773388Z * [new branch] sve-poc -> origin/sve-poc 2025-08-26T20:08:26.1775667Z * [new branch] svekars-patch-1 -> origin/svekars-patch-1 2025-08-26T20:08:26.1776059Z * [new branch] switch-bn -> origin/switch-bn 2025-08-26T20:08:26.1776243Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-08-26T20:08:26.1777507Z * [new branch] tenpercent/ck_inductor_gfx950 -> origin/tenpercent/ck_inductor_gfx950 2025-08-26T20:08:26.1777886Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-08-26T20:08:26.1779359Z * [new branch] test-7054 -> origin/test-7054 2025-08-26T20:08:26.1779640Z * [new branch] test-half-migration-internally -> origin/test-half-migration-internally 2025-08-26T20:08:26.1779840Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-08-26T20:08:26.1782802Z * [new branch] test-myst-markdown-docstring -> origin/test-myst-markdown-docstring 2025-08-26T20:08:26.1782972Z * [new branch] test-old -> origin/test-old 2025-08-26T20:08:26.1783214Z * [new branch] test-vec-migration-internally -> origin/test-vec-migration-internally 2025-08-26T20:08:26.1783543Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-08-26T20:08:26.1786045Z * [new branch] test/inductor -> origin/test/inductor 2025-08-26T20:08:26.1786277Z * [new branch] tianren/flex_paged_attn_fix -> origin/tianren/flex_paged_attn_fix 2025-08-26T20:08:26.1786642Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-08-26T20:08:26.1787071Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-08-26T20:08:26.1788128Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-08-26T20:08:26.1788528Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-08-26T20:08:26.1789661Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-08-26T20:08:26.1790090Z * [new branch] tree_vec_base -> origin/tree_vec_base 2025-08-26T20:08:26.1793628Z * [new branch] triton-update -> origin/triton-update 2025-08-26T20:08:26.1793803Z * [new branch] triton_kernel -> origin/triton_kernel 2025-08-26T20:08:26.1793980Z * [new branch] triton_kernel_perf -> origin/triton_kernel_perf 2025-08-26T20:08:26.1794129Z * [new branch] try-runllm -> origin/try-runllm 2025-08-26T20:08:26.1794260Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-08-26T20:08:26.1794929Z * [new branch] tweak-transformer-dependabot -> origin/tweak-transformer-dependabot 2025-08-26T20:08:26.1795586Z * [new branch] type_dec -> origin/type_dec 2025-08-26T20:08:26.1796999Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-08-26T20:08:26.1801208Z * [new branch] update-audio-commit-hash/16583472358-1693-1 -> origin/update-audio-commit-hash/16583472358-1693-1 2025-08-26T20:08:26.1801523Z * [new branch] update-audio-commit-hash/16663082088-1700-1 -> origin/update-audio-commit-hash/16663082088-1700-1 2025-08-26T20:08:26.1801784Z * [new branch] update-audio-commit-hash/16737365217-1704-1 -> origin/update-audio-commit-hash/16737365217-1704-1 2025-08-26T20:08:26.1802045Z * [new branch] update-audio-commit-hash/16791960928-1711-1 -> origin/update-audio-commit-hash/16791960928-1711-1 2025-08-26T20:08:26.1802350Z * [new branch] update-audio-commit-hash/16818882925-1712-1 -> origin/update-audio-commit-hash/16818882925-1712-1 2025-08-26T20:08:26.1802612Z * [new branch] update-audio-commit-hash/16895560422-1720-1 -> origin/update-audio-commit-hash/16895560422-1720-1 2025-08-26T20:08:26.1802868Z * [new branch] update-audio-commit-hash/16924174496-1738-1 -> origin/update-audio-commit-hash/16924174496-1738-1 2025-08-26T20:08:26.1807623Z * [new branch] update-audio-commit-hash/17002010821-1749-1 -> origin/update-audio-commit-hash/17002010821-1749-1 2025-08-26T20:08:26.1807915Z * [new branch] update-audio-commit-hash/17056004427-1766-1 -> origin/update-audio-commit-hash/17056004427-1766-1 2025-08-26T20:08:26.1808175Z * [new branch] update-audio-commit-hash/17085054029-1767-1 -> origin/update-audio-commit-hash/17085054029-1767-1 2025-08-26T20:08:26.1808422Z * [new branch] update-audio-commit-hash/17142507405-1771-1 -> origin/update-audio-commit-hash/17142507405-1771-1 2025-08-26T20:08:26.1808658Z * [new branch] update-audio-commit-hash/17168762740-1773-1 -> origin/update-audio-commit-hash/17168762740-1773-1 2025-08-26T20:08:26.1808863Z * [new branch] update-dynamic-shapes-doc -> origin/update-dynamic-shapes-doc 2025-08-26T20:08:26.1809173Z * [new branch] update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1 2025-08-26T20:08:26.1809439Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-08-26T20:08:26.1810496Z * [new branch] update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1 2025-08-26T20:08:26.1811487Z * [new branch] update-vllm-commit-hash/16545403308-1687-1 -> origin/update-vllm-commit-hash/16545403308-1687-1 2025-08-26T20:08:26.1812160Z * [new branch] update-vllm-commit-hash/16557202787-1688-1 -> origin/update-vllm-commit-hash/16557202787-1688-1 2025-08-26T20:08:26.1812611Z * [new branch] update-vllm-commit-hash/16583472358-1693-1 -> origin/update-vllm-commit-hash/16583472358-1693-1 2025-08-26T20:08:26.1813422Z * [new branch] update-vllm-commit-hash/16663082088-1700-1 -> origin/update-vllm-commit-hash/16663082088-1700-1 2025-08-26T20:08:26.1813983Z * [new branch] update-vllm-commit-hash/16737365217-1704-1 -> origin/update-vllm-commit-hash/16737365217-1704-1 2025-08-26T20:08:26.1818669Z * [new branch] update-vllm-commit-hash/16843157111-1713-1 -> origin/update-vllm-commit-hash/16843157111-1713-1 2025-08-26T20:08:26.1818935Z * [new branch] update-vllm-commit-hash/16855312394-1714-1 -> origin/update-vllm-commit-hash/16855312394-1714-1 2025-08-26T20:08:26.1819197Z * [new branch] update-vllm-commit-hash/16924174496-1738-1 -> origin/update-vllm-commit-hash/16924174496-1738-1 2025-08-26T20:08:26.1819436Z * [new branch] update-vllm-commit-hash/16952608705-1745-1 -> origin/update-vllm-commit-hash/16952608705-1745-1 2025-08-26T20:08:26.1819681Z * [new branch] update-vllm-commit-hash/16979836546-1748-1 -> origin/update-vllm-commit-hash/16979836546-1748-1 2025-08-26T20:08:26.1820052Z * [new branch] update-vllm-commit-hash/17014576881-1756-1 -> origin/update-vllm-commit-hash/17014576881-1756-1 2025-08-26T20:08:26.1820302Z * [new branch] update-vllm-commit-hash/17027830869-1761-1 -> origin/update-vllm-commit-hash/17027830869-1761-1 2025-08-26T20:08:26.1820690Z * [new branch] update-vllm-commit-hash/17056004427-1766-1 -> origin/update-vllm-commit-hash/17056004427-1766-1 2025-08-26T20:08:26.1821053Z * [new branch] update-vllm-commit-hash/17085054029-1767-1 -> origin/update-vllm-commit-hash/17085054029-1767-1 2025-08-26T20:08:26.1824054Z * [new branch] update-vllm-commit-hash/17113610216-1768-1 -> origin/update-vllm-commit-hash/17113610216-1768-1 2025-08-26T20:08:26.1824322Z * [new branch] update-vllm-commit-hash/17142507405-1771-1 -> origin/update-vllm-commit-hash/17142507405-1771-1 2025-08-26T20:08:26.1824569Z * [new branch] update-vllm-commit-hash/17181878974-1774-1 -> origin/update-vllm-commit-hash/17181878974-1774-1 2025-08-26T20:08:26.1824840Z * [new branch] update-xla-commit-hash/16260974441-194-1 -> origin/update-xla-commit-hash/16260974441-194-1 2025-08-26T20:08:26.1829646Z * [new branch] update-xla-commit-hash/16717126778-197-1 -> origin/update-xla-commit-hash/16717126778-197-1 2025-08-26T20:08:26.1834231Z * [new branch] update-xla-commit-hash/16873912760-198-1 -> origin/update-xla-commit-hash/16873912760-198-1 2025-08-26T20:08:26.1834522Z * [new branch] update-xla-commit-hash/17034266655-199-1 -> origin/update-xla-commit-hash/17034266655-199-1 2025-08-26T20:08:26.1834760Z * [new branch] update-xla-commit-hash/17202464405-200-1 -> origin/update-xla-commit-hash/17202464405-200-1 2025-08-26T20:08:26.1835020Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-08-26T20:08:26.1835195Z * [new branch] update_executorch_pin -> origin/update_executorch_pin 2025-08-26T20:08:26.1835376Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-08-26T20:08:26.1835553Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-08-26T20:08:26.1835715Z * [new branch] update_slow_tests_1752478971 -> origin/update_slow_tests_1752478971 2025-08-26T20:08:26.1836053Z * [new branch] update_slow_tests_1755502951 -> origin/update_slow_tests_1755502951 2025-08-26T20:08:26.1836220Z * [new branch] update_slow_tests_1756107664 -> origin/update_slow_tests_1756107664 2025-08-26T20:08:26.1836422Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-08-26T20:08:26.1836588Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-08-26T20:08:26.1836766Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-08-26T20:08:26.1836910Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-08-26T20:08:26.1837044Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-08-26T20:08:26.1837349Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-08-26T20:08:26.1837878Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-08-26T20:08:26.1838332Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-08-26T20:08:26.1839502Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-08-26T20:08:26.1839740Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-08-26T20:08:26.1843701Z * [new branch] validate_fn -> origin/validate_fn 2025-08-26T20:08:26.1843999Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-08-26T20:08:26.1844156Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-08-26T20:08:26.1844455Z * [new branch] viable/strict -> origin/viable/strict 2025-08-26T20:08:26.1844599Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-08-26T20:08:26.1844738Z * [new branch] vllmpin -> origin/vllmpin 2025-08-26T20:08:26.1851569Z * [new branch] wdvr/conda_devcontainer -> origin/wdvr/conda_devcontainer 2025-08-26T20:08:26.1851844Z * [new branch] wdvr/fix_logging_test -> origin/wdvr/fix_logging_test 2025-08-26T20:08:26.1852405Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-08-26T20:08:26.1853020Z * [new branch] weight_sharing_cpp -> origin/weight_sharing_cpp 2025-08-26T20:08:26.1856823Z * [new branch] whc/flight4 -> origin/whc/flight4 2025-08-26T20:08:26.1860299Z * [new branch] whc/flight51 -> origin/whc/flight51 2025-08-26T20:08:26.1860633Z * [new branch] whc/flight53 -> origin/whc/flight53 2025-08-26T20:08:26.1860845Z * [new branch] whc/p2phang -> origin/whc/p2phang 2025-08-26T20:08:26.1860986Z * [new branch] whc/stage2 -> origin/whc/stage2 2025-08-26T20:08:26.1861107Z * [new branch] whc/uneven -> origin/whc/uneven 2025-08-26T20:08:26.1861290Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-08-26T20:08:26.1861419Z * [new branch] win_warnings -> origin/win_warnings 2025-08-26T20:08:26.1861565Z * [new branch] workonoldcommit -> origin/workonoldcommit 2025-08-26T20:08:26.1861767Z * [new branch] wwen/programming-model-2.8 -> origin/wwen/programming-model-2.8 2025-08-26T20:08:26.1861898Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-08-26T20:08:26.1862053Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-08-26T20:08:26.1862336Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-08-26T20:08:26.1862482Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-08-26T20:08:26.1864665Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-08-26T20:08:26.1865011Z * [new branch] xmfan/ca_api -> origin/xmfan/ca_api 2025-08-26T20:08:26.1865172Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-08-26T20:08:26.1865310Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-08-26T20:08:26.1865494Z * [new branch] xmfan/ca_cudagraphs -> origin/xmfan/ca_cudagraphs 2025-08-26T20:08:26.1865662Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-08-26T20:08:26.1867840Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-08-26T20:08:26.1868029Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-08-26T20:08:26.1868204Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-08-26T20:08:26.1868370Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-08-26T20:08:26.1868538Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-08-26T20:08:26.1868699Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-08-26T20:08:26.1868963Z * [new branch] xmfan/ca_mem_base -> origin/xmfan/ca_mem_base 2025-08-26T20:08:26.1869156Z * [new branch] xmfan/ca_mem_fix -> origin/xmfan/ca_mem_fix 2025-08-26T20:08:26.1869332Z * [new branch] xmfan/ca_memory_fix -> origin/xmfan/ca_memory_fix 2025-08-26T20:08:26.1871657Z * [new branch] xmfan/ca_memory_fix_rebased -> origin/xmfan/ca_memory_fix_rebased 2025-08-26T20:08:26.1872231Z * [new branch] xmfan/ca_memory_fix_rebased2 -> origin/xmfan/ca_memory_fix_rebased2 2025-08-26T20:08:26.1872407Z * [new branch] xmfan/ca_move_to_cuda -> origin/xmfan/ca_move_to_cuda 2025-08-26T20:08:26.1872589Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-08-26T20:08:26.1872742Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-08-26T20:08:26.1872964Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-08-26T20:08:26.1873135Z * [new branch] xmfan/ca_scalar -> origin/xmfan/ca_scalar 2025-08-26T20:08:26.1874920Z * [new branch] xmfan/ca_subclass_mem_fix -> origin/xmfan/ca_subclass_mem_fix 2025-08-26T20:08:26.1875086Z * [new branch] xmfan/ca_warm_mem -> origin/xmfan/ca_warm_mem 2025-08-26T20:08:26.1875246Z * [new branch] xmfan/ca_warm_mem_base -> origin/xmfan/ca_warm_mem_base 2025-08-26T20:08:26.1875403Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-08-26T20:08:26.1875578Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-08-26T20:08:26.1876644Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-08-26T20:08:26.1876849Z * [new branch] xmfan/cacu_may27 -> origin/xmfan/cacu_may27 2025-08-26T20:08:26.1877904Z * [new branch] xmfan/circular_dep -> origin/xmfan/circular_dep 2025-08-26T20:08:26.1879840Z * [new branch] xmfan/compiled_autograd_feb_29 -> origin/xmfan/compiled_autograd_feb_29 2025-08-26T20:08:26.1880042Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-08-26T20:08:26.1880270Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-08-26T20:08:26.1883561Z * [new branch] xmfan/issue_123374 -> origin/xmfan/issue_123374 2025-08-26T20:08:26.1888051Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-08-26T20:08:26.1893266Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-08-26T20:08:26.1897565Z * [new branch] xmfan/segfault_test -> origin/xmfan/segfault_test 2025-08-26T20:08:26.1897921Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-08-26T20:08:26.1898149Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-08-26T20:08:26.1898362Z * [new branch] xmfan/test -> origin/xmfan/test 2025-08-26T20:08:26.1898665Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-08-26T20:08:26.1899091Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-08-26T20:08:26.1899299Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-08-26T20:08:26.1899461Z * [new branch] yihan_quantization -> origin/yihan_quantization 2025-08-26T20:08:26.1899722Z * [new branch] yiming/add_jit_trace_benchmark -> origin/yiming/add_jit_trace_benchmark 2025-08-26T20:08:26.1905753Z * [new branch] yiming/add_nativert_benchmark -> origin/yiming/add_nativert_benchmark 2025-08-26T20:08:26.1910702Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-08-26T20:08:26.1910891Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-08-26T20:08:26.1911424Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-08-26T20:08:26.1911827Z * [new branch] zainr/git-push-v2 -> origin/zainr/git-push-v2 2025-08-26T20:08:26.1912017Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-08-26T20:08:26.1912151Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-08-26T20:08:26.1912309Z * [new branch] zainr/unstable -> origin/zainr/unstable 2025-08-26T20:08:26.1912462Z * [new branch] zainr/unstable-xla -> origin/zainr/unstable-xla 2025-08-26T20:08:26.1912606Z * [new branch] zainr/uv-pip-fix -> origin/zainr/uv-pip-fix 2025-08-26T20:08:26.1912757Z * [new branch] zainr/vs-aarch64 -> origin/zainr/vs-aarch64 2025-08-26T20:08:26.1912914Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-08-26T20:08:26.1913044Z * [new branch] zb2p -> origin/zb2p 2025-08-26T20:08:26.1913195Z * [new branch] zdevito-patch-1 -> origin/zdevito-patch-1 2025-08-26T20:08:26.1913379Z * [new branch] zero_grad_optimization -> origin/zero_grad_optimization 2025-08-26T20:08:26.1913552Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-08-26T20:08:26.1913706Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-08-26T20:08:26.1913871Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-08-26T20:08:26.1914000Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-08-26T20:08:26.1914162Z * [new branch] zxiiro/test -> origin/zxiiro/test 2025-08-26T20:08:26.1914489Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-08-26T20:08:26.1914626Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-08-26T20:08:26.1914759Z * [new tag] ciflow/binaries/153920 -> ciflow/binaries/153920 2025-08-26T20:08:26.1914903Z * [new tag] ciflow/binaries/158104 -> ciflow/binaries/158104 2025-08-26T20:08:26.1915040Z * [new tag] ciflow/binaries/160229 -> ciflow/binaries/160229 2025-08-26T20:08:26.1915240Z * [new tag] ciflow/binaries/160853 -> ciflow/binaries/160853 2025-08-26T20:08:26.1915384Z * [new tag] ciflow/binaries/161257 -> ciflow/binaries/161257 2025-08-26T20:08:26.1915865Z * [new tag] ciflow/binaries_libtorch/156049 -> ciflow/binaries_libtorch/156049 2025-08-26T20:08:26.1916391Z * [new tag] ciflow/binaries_wheel/156049 -> ciflow/binaries_wheel/156049 2025-08-26T20:08:26.1916880Z * [new tag] ciflow/binaries_wheel/158733 -> ciflow/binaries_wheel/158733 2025-08-26T20:08:26.1917875Z * [new tag] ciflow/binaries_wheel/160207 -> ciflow/binaries_wheel/160207 2025-08-26T20:08:26.1918299Z * [new tag] ciflow/h100-symm-mem/151845 -> ciflow/h100-symm-mem/151845 2025-08-26T20:08:26.1918771Z * [new tag] ciflow/h100-symm-mem/155923 -> ciflow/h100-symm-mem/155923 2025-08-26T20:08:26.1919106Z * [new tag] ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635 2025-08-26T20:08:26.1923477Z * [new tag] ciflow/h100-symm-mem/159562 -> ciflow/h100-symm-mem/159562 2025-08-26T20:08:26.1923667Z * [new tag] ciflow/h100-symm-mem/159889 -> ciflow/h100-symm-mem/159889 2025-08-26T20:08:26.1923813Z * [new tag] ciflow/h100-symm-mem/160825 -> ciflow/h100-symm-mem/160825 2025-08-26T20:08:26.1923959Z * [new tag] ciflow/h100-symm-mem/161008 -> ciflow/h100-symm-mem/161008 2025-08-26T20:08:26.1924098Z * [new tag] ciflow/h100-symm-mem/161090 -> ciflow/h100-symm-mem/161090 2025-08-26T20:08:26.1924408Z * [new tag] ciflow/h100-symm-mem/161214 -> ciflow/h100-symm-mem/161214 2025-08-26T20:08:26.1924563Z * [new tag] ciflow/h100-symm-mem/161217 -> ciflow/h100-symm-mem/161217 2025-08-26T20:08:26.1924697Z * [new tag] ciflow/h100-symm-mem/161232 -> ciflow/h100-symm-mem/161232 2025-08-26T20:08:26.1925034Z * [new tag] ciflow/h100-symm-mem/161257 -> ciflow/h100-symm-mem/161257 2025-08-26T20:08:26.1925198Z * [new tag] ciflow/h100-symm-mem/161309 -> ciflow/h100-symm-mem/161309 2025-08-26T20:08:26.1925431Z * [new tag] ciflow/h100-symm-mem/161470 -> ciflow/h100-symm-mem/161470 2025-08-26T20:08:26.1925709Z * [new tag] ciflow/h100-symm-mem/161471 -> ciflow/h100-symm-mem/161471 2025-08-26T20:08:26.1925873Z * [new tag] ciflow/h100-symm-mem/161532 -> ciflow/h100-symm-mem/161532 2025-08-26T20:08:26.1926097Z * [new tag] ciflow/h100-symm-mem/161533 -> ciflow/h100-symm-mem/161533 2025-08-26T20:08:26.1926258Z * [new tag] ciflow/h100/159158 -> ciflow/h100/159158 2025-08-26T20:08:26.1932277Z * [new tag] ciflow/h100/161225 -> ciflow/h100/161225 2025-08-26T20:08:26.1937372Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/151845 -> ciflow/inductor-perf-test-nightly-rocm/151845 2025-08-26T20:08:26.1939247Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/161512 -> ciflow/inductor-perf-test-nightly-x86-zen/161512 2025-08-26T20:08:26.1939619Z * [new tag] ciflow/inductor-periodic/158137 -> ciflow/inductor-periodic/158137 2025-08-26T20:08:26.1939887Z * [new tag] ciflow/inductor-periodic/160807 -> ciflow/inductor-periodic/160807 2025-08-26T20:08:26.1940083Z * [new tag] ciflow/inductor-periodic/161461 -> ciflow/inductor-periodic/161461 2025-08-26T20:08:26.1940326Z * [new tag] ciflow/inductor-periodic/161536 -> ciflow/inductor-periodic/161536 2025-08-26T20:08:26.1940740Z * [new tag] ciflow/inductor-periodic/2f0de0ff9361ca4f2b1e6f9edbc600b5fb6abcd6 -> ciflow/inductor-periodic/2f0de0ff9361ca4f2b1e6f9edbc600b5fb6abcd6 2025-08-26T20:08:26.1941149Z * [new tag] ciflow/inductor-periodic/3e5b021f217a42ae55dc690083f67a28126808ed -> ciflow/inductor-periodic/3e5b021f217a42ae55dc690083f67a28126808ed 2025-08-26T20:08:26.1946067Z * [new tag] ciflow/inductor-periodic/f912c93344caa74e24c8164a2e25fe84a8203073 -> ciflow/inductor-periodic/f912c93344caa74e24c8164a2e25fe84a8203073 2025-08-26T20:08:26.1948226Z * [new tag] ciflow/inductor-rocm/151845 -> ciflow/inductor-rocm/151845 2025-08-26T20:08:26.1948513Z * [new tag] ciflow/inductor-rocm/159158 -> ciflow/inductor-rocm/159158 2025-08-26T20:08:26.1951294Z * [new tag] ciflow/inductor-rocm/160671 -> ciflow/inductor-rocm/160671 2025-08-26T20:08:26.1951652Z * [new tag] ciflow/inductor-rocm/161180 -> ciflow/inductor-rocm/161180 2025-08-26T20:08:26.1951907Z * [new tag] ciflow/inductor-rocm/161225 -> ciflow/inductor-rocm/161225 2025-08-26T20:08:26.1952071Z * [new tag] ciflow/inductor-rocm/161521 -> ciflow/inductor-rocm/161521 2025-08-26T20:08:26.1952359Z * [new tag] ciflow/inductor-windows/160406 -> ciflow/inductor-windows/160406 2025-08-26T20:08:26.1952977Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-08-26T20:08:26.1953153Z * [new tag] ciflow/inductor/151845 -> ciflow/inductor/151845 2025-08-26T20:08:26.1953299Z * [new tag] ciflow/inductor/154694 -> ciflow/inductor/154694 2025-08-26T20:08:26.1953427Z * [new tag] ciflow/inductor/155072 -> ciflow/inductor/155072 2025-08-26T20:08:26.1953566Z * [new tag] ciflow/inductor/155152 -> ciflow/inductor/155152 2025-08-26T20:08:26.1953865Z * [new tag] ciflow/inductor/155153 -> ciflow/inductor/155153 2025-08-26T20:08:26.1954012Z * [new tag] ciflow/inductor/155154 -> ciflow/inductor/155154 2025-08-26T20:08:26.1954143Z * [new tag] ciflow/inductor/155501 -> ciflow/inductor/155501 2025-08-26T20:08:26.1954270Z * [new tag] ciflow/inductor/155502 -> ciflow/inductor/155502 2025-08-26T20:08:26.1954425Z * [new tag] ciflow/inductor/155503 -> ciflow/inductor/155503 2025-08-26T20:08:26.1954550Z * [new tag] ciflow/inductor/155557 -> ciflow/inductor/155557 2025-08-26T20:08:26.1954682Z * [new tag] ciflow/inductor/155608 -> ciflow/inductor/155608 2025-08-26T20:08:26.1954808Z * [new tag] ciflow/inductor/155923 -> ciflow/inductor/155923 2025-08-26T20:08:26.1954933Z * [new tag] ciflow/inductor/155928 -> ciflow/inductor/155928 2025-08-26T20:08:26.1955070Z * [new tag] ciflow/inductor/156875 -> ciflow/inductor/156875 2025-08-26T20:08:26.1955198Z * [new tag] ciflow/inductor/156967 -> ciflow/inductor/156967 2025-08-26T20:08:26.1955331Z * [new tag] ciflow/inductor/157298 -> ciflow/inductor/157298 2025-08-26T20:08:26.1955458Z * [new tag] ciflow/inductor/157572 -> ciflow/inductor/157572 2025-08-26T20:08:26.1955594Z * [new tag] ciflow/inductor/157635 -> ciflow/inductor/157635 2025-08-26T20:08:26.1955720Z * [new tag] ciflow/inductor/157743 -> ciflow/inductor/157743 2025-08-26T20:08:26.1955845Z * [new tag] ciflow/inductor/157767 -> ciflow/inductor/157767 2025-08-26T20:08:26.1955977Z * [new tag] ciflow/inductor/157944 -> ciflow/inductor/157944 2025-08-26T20:08:26.1956103Z * [new tag] ciflow/inductor/158061 -> ciflow/inductor/158061 2025-08-26T20:08:26.1956235Z * [new tag] ciflow/inductor/158097 -> ciflow/inductor/158097 2025-08-26T20:08:26.1956365Z * [new tag] ciflow/inductor/158098 -> ciflow/inductor/158098 2025-08-26T20:08:26.1956493Z * [new tag] ciflow/inductor/158104 -> ciflow/inductor/158104 2025-08-26T20:08:26.1956626Z * [new tag] ciflow/inductor/158137 -> ciflow/inductor/158137 2025-08-26T20:08:26.1956750Z * [new tag] ciflow/inductor/158321 -> ciflow/inductor/158321 2025-08-26T20:08:26.1956967Z * [new tag] ciflow/inductor/158609 -> ciflow/inductor/158609 2025-08-26T20:08:26.1957092Z * [new tag] ciflow/inductor/158932 -> ciflow/inductor/158932 2025-08-26T20:08:26.1957225Z * [new tag] ciflow/inductor/159003 -> ciflow/inductor/159003 2025-08-26T20:08:26.1957349Z * [new tag] ciflow/inductor/159158 -> ciflow/inductor/159158 2025-08-26T20:08:26.1957474Z * [new tag] ciflow/inductor/159274 -> ciflow/inductor/159274 2025-08-26T20:08:26.1957616Z * [new tag] ciflow/inductor/159387 -> ciflow/inductor/159387 2025-08-26T20:08:26.1957751Z * [new tag] ciflow/inductor/159473 -> ciflow/inductor/159473 2025-08-26T20:08:26.1957884Z * [new tag] ciflow/inductor/159664 -> ciflow/inductor/159664 2025-08-26T20:08:26.1958010Z * [new tag] ciflow/inductor/159778 -> ciflow/inductor/159778 2025-08-26T20:08:26.1958149Z * [new tag] ciflow/inductor/159786 -> ciflow/inductor/159786 2025-08-26T20:08:26.1958274Z * [new tag] ciflow/inductor/159835 -> ciflow/inductor/159835 2025-08-26T20:08:26.1958398Z * [new tag] ciflow/inductor/159889 -> ciflow/inductor/159889 2025-08-26T20:08:26.1958533Z * [new tag] ciflow/inductor/159923 -> ciflow/inductor/159923 2025-08-26T20:08:26.1958660Z * [new tag] ciflow/inductor/159944 -> ciflow/inductor/159944 2025-08-26T20:08:26.1958830Z * [new tag] ciflow/inductor/160080 -> ciflow/inductor/160080 2025-08-26T20:08:26.1958963Z * [new tag] ciflow/inductor/160111 -> ciflow/inductor/160111 2025-08-26T20:08:26.1959087Z * [new tag] ciflow/inductor/160138 -> ciflow/inductor/160138 2025-08-26T20:08:26.1959807Z * [new tag] ciflow/inductor/160156 -> ciflow/inductor/160156 2025-08-26T20:08:26.1960515Z * [new tag] ciflow/inductor/160180 -> ciflow/inductor/160180 2025-08-26T20:08:26.1961007Z * [new tag] ciflow/inductor/160198 -> ciflow/inductor/160198 2025-08-26T20:08:26.1961908Z * [new tag] ciflow/inductor/160258 -> ciflow/inductor/160258 2025-08-26T20:08:26.1962062Z * [new tag] ciflow/inductor/160266 -> ciflow/inductor/160266 2025-08-26T20:08:26.1962594Z * [new tag] ciflow/inductor/160282 -> ciflow/inductor/160282 2025-08-26T20:08:26.1963308Z * [new tag] ciflow/inductor/160323 -> ciflow/inductor/160323 2025-08-26T20:08:26.1964075Z * [new tag] ciflow/inductor/160324 -> ciflow/inductor/160324 2025-08-26T20:08:26.1964570Z * [new tag] ciflow/inductor/160325 -> ciflow/inductor/160325 2025-08-26T20:08:26.1965921Z * [new tag] ciflow/inductor/160326 -> ciflow/inductor/160326 2025-08-26T20:08:26.1970601Z * [new tag] ciflow/inductor/160327 -> ciflow/inductor/160327 2025-08-26T20:08:26.1970807Z * [new tag] ciflow/inductor/160328 -> ciflow/inductor/160328 2025-08-26T20:08:26.1970946Z * [new tag] ciflow/inductor/160329 -> ciflow/inductor/160329 2025-08-26T20:08:26.1971265Z * [new tag] ciflow/inductor/160431 -> ciflow/inductor/160431 2025-08-26T20:08:26.1971405Z * [new tag] ciflow/inductor/160448 -> ciflow/inductor/160448 2025-08-26T20:08:26.1971546Z * [new tag] ciflow/inductor/160449 -> ciflow/inductor/160449 2025-08-26T20:08:26.1971672Z * [new tag] ciflow/inductor/160467 -> ciflow/inductor/160467 2025-08-26T20:08:26.1971795Z * [new tag] ciflow/inductor/160470 -> ciflow/inductor/160470 2025-08-26T20:08:26.1971927Z * [new tag] ciflow/inductor/160483 -> ciflow/inductor/160483 2025-08-26T20:08:26.1972219Z * [new tag] ciflow/inductor/160527 -> ciflow/inductor/160527 2025-08-26T20:08:26.1972350Z * [new tag] ciflow/inductor/160532 -> ciflow/inductor/160532 2025-08-26T20:08:26.1972597Z * [new tag] ciflow/inductor/160539 -> ciflow/inductor/160539 2025-08-26T20:08:26.1972747Z * [new tag] ciflow/inductor/160580 -> ciflow/inductor/160580 2025-08-26T20:08:26.1975466Z * [new tag] ciflow/inductor/160601 -> ciflow/inductor/160601 2025-08-26T20:08:26.1975645Z * [new tag] ciflow/inductor/160611 -> ciflow/inductor/160611 2025-08-26T20:08:26.1975786Z * [new tag] ciflow/inductor/160669 -> ciflow/inductor/160669 2025-08-26T20:08:26.1975915Z * [new tag] ciflow/inductor/160670 -> ciflow/inductor/160670 2025-08-26T20:08:26.1976047Z * [new tag] ciflow/inductor/160671 -> ciflow/inductor/160671 2025-08-26T20:08:26.1976197Z * [new tag] ciflow/inductor/160677 -> ciflow/inductor/160677 2025-08-26T20:08:26.1976391Z * [new tag] ciflow/inductor/160690 -> ciflow/inductor/160690 2025-08-26T20:08:26.1976807Z * [new tag] ciflow/inductor/160763 -> ciflow/inductor/160763 2025-08-26T20:08:26.1977579Z * [new tag] ciflow/inductor/160772 -> ciflow/inductor/160772 2025-08-26T20:08:26.1977780Z * [new tag] ciflow/inductor/160798 -> ciflow/inductor/160798 2025-08-26T20:08:26.1980798Z * [new tag] ciflow/inductor/160807 -> ciflow/inductor/160807 2025-08-26T20:08:26.1981147Z * [new tag] ciflow/inductor/160836 -> ciflow/inductor/160836 2025-08-26T20:08:26.1981326Z * [new tag] ciflow/inductor/160861 -> ciflow/inductor/160861 2025-08-26T20:08:26.1981548Z * [new tag] ciflow/inductor/160883 -> ciflow/inductor/160883 2025-08-26T20:08:26.1981781Z * [new tag] ciflow/inductor/160888 -> ciflow/inductor/160888 2025-08-26T20:08:26.1982443Z * [new tag] ciflow/inductor/160903 -> ciflow/inductor/160903 2025-08-26T20:08:26.1982608Z * [new tag] ciflow/inductor/160913 -> ciflow/inductor/160913 2025-08-26T20:08:26.1982857Z * [new tag] ciflow/inductor/160941 -> ciflow/inductor/160941 2025-08-26T20:08:26.1982997Z * [new tag] ciflow/inductor/160943 -> ciflow/inductor/160943 2025-08-26T20:08:26.1983138Z * [new tag] ciflow/inductor/160991 -> ciflow/inductor/160991 2025-08-26T20:08:26.1983472Z * [new tag] ciflow/inductor/160997 -> ciflow/inductor/160997 2025-08-26T20:08:26.1983975Z * [new tag] ciflow/inductor/161003 -> ciflow/inductor/161003 2025-08-26T20:08:26.1984456Z * [new tag] ciflow/inductor/161026 -> ciflow/inductor/161026 2025-08-26T20:08:26.1984904Z * [new tag] ciflow/inductor/161032 -> ciflow/inductor/161032 2025-08-26T20:08:26.1989186Z * [new tag] ciflow/inductor/161040 -> ciflow/inductor/161040 2025-08-26T20:08:26.1989543Z * [new tag] ciflow/inductor/161055 -> ciflow/inductor/161055 2025-08-26T20:08:26.1989690Z * [new tag] ciflow/inductor/161062 -> ciflow/inductor/161062 2025-08-26T20:08:26.1989831Z * [new tag] ciflow/inductor/161069 -> ciflow/inductor/161069 2025-08-26T20:08:26.1989960Z * [new tag] ciflow/inductor/161092 -> ciflow/inductor/161092 2025-08-26T20:08:26.1990117Z * [new tag] ciflow/inductor/161093 -> ciflow/inductor/161093 2025-08-26T20:08:26.1990242Z * [new tag] ciflow/inductor/161097 -> ciflow/inductor/161097 2025-08-26T20:08:26.1990378Z * [new tag] ciflow/inductor/161098 -> ciflow/inductor/161098 2025-08-26T20:08:26.1990507Z * [new tag] ciflow/inductor/161100 -> ciflow/inductor/161100 2025-08-26T20:08:26.1990799Z * [new tag] ciflow/inductor/161107 -> ciflow/inductor/161107 2025-08-26T20:08:26.1991047Z * [new tag] ciflow/inductor/161110 -> ciflow/inductor/161110 2025-08-26T20:08:26.1991583Z * [new tag] ciflow/inductor/161117 -> ciflow/inductor/161117 2025-08-26T20:08:26.1992191Z * [new tag] ciflow/inductor/161118 -> ciflow/inductor/161118 2025-08-26T20:08:26.1992576Z * [new tag] ciflow/inductor/161123 -> ciflow/inductor/161123 2025-08-26T20:08:26.1992859Z * [new tag] ciflow/inductor/161124 -> ciflow/inductor/161124 2025-08-26T20:08:26.1994082Z * [new tag] ciflow/inductor/161125 -> ciflow/inductor/161125 2025-08-26T20:08:26.1994230Z * [new tag] ciflow/inductor/161126 -> ciflow/inductor/161126 2025-08-26T20:08:26.1994851Z * [new tag] ciflow/inductor/161144 -> ciflow/inductor/161144 2025-08-26T20:08:26.1995047Z * [new tag] ciflow/inductor/161148 -> ciflow/inductor/161148 2025-08-26T20:08:26.1995546Z * [new tag] ciflow/inductor/161158 -> ciflow/inductor/161158 2025-08-26T20:08:26.1996700Z * [new tag] ciflow/inductor/161178 -> ciflow/inductor/161178 2025-08-26T20:08:26.1997032Z * [new tag] ciflow/inductor/161190 -> ciflow/inductor/161190 2025-08-26T20:08:26.1997585Z * [new tag] ciflow/inductor/161208 -> ciflow/inductor/161208 2025-08-26T20:08:26.1998206Z * [new tag] ciflow/inductor/161225 -> ciflow/inductor/161225 2025-08-26T20:08:26.1998642Z * [new tag] ciflow/inductor/161229 -> ciflow/inductor/161229 2025-08-26T20:08:26.1999752Z * [new tag] ciflow/inductor/161237 -> ciflow/inductor/161237 2025-08-26T20:08:26.2000203Z * [new tag] ciflow/inductor/161241 -> ciflow/inductor/161241 2025-08-26T20:08:26.2000361Z * [new tag] ciflow/inductor/161246 -> ciflow/inductor/161246 2025-08-26T20:08:26.2002186Z * [new tag] ciflow/inductor/161274 -> ciflow/inductor/161274 2025-08-26T20:08:26.2002536Z * [new tag] ciflow/inductor/161278 -> ciflow/inductor/161278 2025-08-26T20:08:26.2002702Z * [new tag] ciflow/inductor/161279 -> ciflow/inductor/161279 2025-08-26T20:08:26.2002932Z * [new tag] ciflow/inductor/161288 -> ciflow/inductor/161288 2025-08-26T20:08:26.2003112Z * [new tag] ciflow/inductor/161314 -> ciflow/inductor/161314 2025-08-26T20:08:26.2005680Z * [new tag] ciflow/inductor/161320 -> ciflow/inductor/161320 2025-08-26T20:08:26.2005993Z * [new tag] ciflow/inductor/161336 -> ciflow/inductor/161336 2025-08-26T20:08:26.2006151Z * [new tag] ciflow/inductor/161337 -> ciflow/inductor/161337 2025-08-26T20:08:26.2006286Z * [new tag] ciflow/inductor/161338 -> ciflow/inductor/161338 2025-08-26T20:08:26.2006404Z * [new tag] ciflow/inductor/161339 -> ciflow/inductor/161339 2025-08-26T20:08:26.2006665Z * [new tag] ciflow/inductor/161340 -> ciflow/inductor/161340 2025-08-26T20:08:26.2006806Z * [new tag] ciflow/inductor/161341 -> ciflow/inductor/161341 2025-08-26T20:08:26.2007183Z * [new tag] ciflow/inductor/161342 -> ciflow/inductor/161342 2025-08-26T20:08:26.2007894Z * [new tag] ciflow/inductor/161343 -> ciflow/inductor/161343 2025-08-26T20:08:26.2008098Z * [new tag] ciflow/inductor/161344 -> ciflow/inductor/161344 2025-08-26T20:08:26.2011292Z * [new tag] ciflow/inductor/161345 -> ciflow/inductor/161345 2025-08-26T20:08:26.2011942Z * [new tag] ciflow/inductor/161346 -> ciflow/inductor/161346 2025-08-26T20:08:26.2012350Z * [new tag] ciflow/inductor/161347 -> ciflow/inductor/161347 2025-08-26T20:08:26.2012475Z * [new tag] ciflow/inductor/161348 -> ciflow/inductor/161348 2025-08-26T20:08:26.2012597Z * [new tag] ciflow/inductor/161349 -> ciflow/inductor/161349 2025-08-26T20:08:26.2012725Z * [new tag] ciflow/inductor/161350 -> ciflow/inductor/161350 2025-08-26T20:08:26.2012846Z * [new tag] ciflow/inductor/161351 -> ciflow/inductor/161351 2025-08-26T20:08:26.2012980Z * [new tag] ciflow/inductor/161353 -> ciflow/inductor/161353 2025-08-26T20:08:26.2013264Z * [new tag] ciflow/inductor/161354 -> ciflow/inductor/161354 2025-08-26T20:08:26.2013417Z * [new tag] ciflow/inductor/161355 -> ciflow/inductor/161355 2025-08-26T20:08:26.2013651Z * [new tag] ciflow/inductor/161362 -> ciflow/inductor/161362 2025-08-26T20:08:26.2014046Z * [new tag] ciflow/inductor/161363 -> ciflow/inductor/161363 2025-08-26T20:08:26.2014458Z * [new tag] ciflow/inductor/161382 -> ciflow/inductor/161382 2025-08-26T20:08:26.2016864Z * [new tag] ciflow/inductor/161383 -> ciflow/inductor/161383 2025-08-26T20:08:26.2017021Z * [new tag] ciflow/inductor/161385 -> ciflow/inductor/161385 2025-08-26T20:08:26.2017153Z * [new tag] ciflow/inductor/161396 -> ciflow/inductor/161396 2025-08-26T20:08:26.2017279Z * [new tag] ciflow/inductor/161397 -> ciflow/inductor/161397 2025-08-26T20:08:26.2017557Z * [new tag] ciflow/inductor/161404 -> ciflow/inductor/161404 2025-08-26T20:08:26.2017700Z * [new tag] ciflow/inductor/161405 -> ciflow/inductor/161405 2025-08-26T20:08:26.2017868Z * [new tag] ciflow/inductor/161406 -> ciflow/inductor/161406 2025-08-26T20:08:26.2018728Z * [new tag] ciflow/inductor/161409 -> ciflow/inductor/161409 2025-08-26T20:08:26.2019218Z * [new tag] ciflow/inductor/161410 -> ciflow/inductor/161410 2025-08-26T20:08:26.2019634Z * [new tag] ciflow/inductor/161414 -> ciflow/inductor/161414 2025-08-26T20:08:26.2020000Z * [new tag] ciflow/inductor/161416 -> ciflow/inductor/161416 2025-08-26T20:08:26.2020456Z * [new tag] ciflow/inductor/161420 -> ciflow/inductor/161420 2025-08-26T20:08:26.2022579Z * [new tag] ciflow/inductor/161431 -> ciflow/inductor/161431 2025-08-26T20:08:26.2022759Z * [new tag] ciflow/inductor/161435 -> ciflow/inductor/161435 2025-08-26T20:08:26.2023117Z * [new tag] ciflow/inductor/161440 -> ciflow/inductor/161440 2025-08-26T20:08:26.2023250Z * [new tag] ciflow/inductor/161447 -> ciflow/inductor/161447 2025-08-26T20:08:26.2023365Z * [new tag] ciflow/inductor/161452 -> ciflow/inductor/161452 2025-08-26T20:08:26.2023751Z * [new tag] ciflow/inductor/161453 -> ciflow/inductor/161453 2025-08-26T20:08:26.2026160Z * [new tag] ciflow/inductor/161458 -> ciflow/inductor/161458 2025-08-26T20:08:26.2026324Z * [new tag] ciflow/inductor/161461 -> ciflow/inductor/161461 2025-08-26T20:08:26.2026456Z * [new tag] ciflow/inductor/161464 -> ciflow/inductor/161464 2025-08-26T20:08:26.2026591Z * [new tag] ciflow/inductor/161466 -> ciflow/inductor/161466 2025-08-26T20:08:26.2026737Z * [new tag] ciflow/inductor/161468 -> ciflow/inductor/161468 2025-08-26T20:08:26.2026873Z * [new tag] ciflow/inductor/161469 -> ciflow/inductor/161469 2025-08-26T20:08:26.2027058Z * [new tag] ciflow/inductor/161474 -> ciflow/inductor/161474 2025-08-26T20:08:26.2027655Z * [new tag] ciflow/inductor/161477 -> ciflow/inductor/161477 2025-08-26T20:08:26.2028089Z * [new tag] ciflow/inductor/161485 -> ciflow/inductor/161485 2025-08-26T20:08:26.2028595Z * [new tag] ciflow/inductor/161486 -> ciflow/inductor/161486 2025-08-26T20:08:26.2029078Z * [new tag] ciflow/inductor/161487 -> ciflow/inductor/161487 2025-08-26T20:08:26.2029456Z * [new tag] ciflow/inductor/161495 -> ciflow/inductor/161495 2025-08-26T20:08:26.2034024Z * [new tag] ciflow/inductor/161497 -> ciflow/inductor/161497 2025-08-26T20:08:26.2034343Z * [new tag] ciflow/inductor/161499 -> ciflow/inductor/161499 2025-08-26T20:08:26.2034510Z * [new tag] ciflow/inductor/161512 -> ciflow/inductor/161512 2025-08-26T20:08:26.2034678Z * [new tag] ciflow/inductor/161521 -> ciflow/inductor/161521 2025-08-26T20:08:26.2034829Z * [new tag] ciflow/inductor/161526 -> ciflow/inductor/161526 2025-08-26T20:08:26.2034971Z * [new tag] ciflow/inductor/161530 -> ciflow/inductor/161530 2025-08-26T20:08:26.2035233Z * [new tag] ciflow/inductor/161534 -> ciflow/inductor/161534 2025-08-26T20:08:26.2035382Z * [new tag] ciflow/inductor/161536 -> ciflow/inductor/161536 2025-08-26T20:08:26.2035521Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-08-26T20:08:26.2035666Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-08-26T20:08:26.2035941Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-08-26T20:08:26.2036107Z * [new tag] ciflow/linux-aarch64/159737 -> ciflow/linux-aarch64/159737 2025-08-26T20:08:26.2036498Z * [new tag] ciflow/linux-aarch64/160078 -> ciflow/linux-aarch64/160078 2025-08-26T20:08:26.2038461Z * [new tag] ciflow/linux-aarch64/160080 -> ciflow/linux-aarch64/160080 2025-08-26T20:08:26.2038646Z * [new tag] ciflow/mps/155923 -> ciflow/mps/155923 2025-08-26T20:08:26.2038772Z * [new tag] ciflow/mps/157553 -> ciflow/mps/157553 2025-08-26T20:08:26.2038902Z * [new tag] ciflow/mps/157635 -> ciflow/mps/157635 2025-08-26T20:08:26.2039578Z * [new tag] ciflow/mps/160839 -> ciflow/mps/160839 2025-08-26T20:08:26.2039814Z * [new tag] ciflow/mps/161511 -> ciflow/mps/161511 2025-08-26T20:08:26.2040299Z * [new tag] ciflow/nightly/158104 -> ciflow/nightly/158104 2025-08-26T20:08:26.2043717Z * [new tag] ciflow/periodic-rocm-mi300/161180 -> ciflow/periodic-rocm-mi300/161180 2025-08-26T20:08:26.2044055Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-08-26T20:08:26.2044478Z * [new tag] ciflow/periodic/0dea191ff7b844352dc2cd5e3b5ef5ea13a76756 -> ciflow/periodic/0dea191ff7b844352dc2cd5e3b5ef5ea13a76756 2025-08-26T20:08:26.2044779Z * [new tag] ciflow/periodic/156491 -> ciflow/periodic/156491 2025-08-26T20:08:26.2045360Z * [new tag] ciflow/periodic/161013 -> ciflow/periodic/161013 2025-08-26T20:08:26.2045525Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-08-26T20:08:26.2045654Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-08-26T20:08:26.2045784Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-08-26T20:08:26.2045952Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-08-26T20:08:26.2046426Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-08-26T20:08:26.2047750Z * [new tag] ciflow/periodic/bc7eaa0d8a1f5ca8ec0eaac461d1df500dcaea84 -> ciflow/periodic/bc7eaa0d8a1f5ca8ec0eaac461d1df500dcaea84 2025-08-26T20:08:26.2048185Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-08-26T20:08:26.2048494Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-08-26T20:08:26.2050414Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-08-26T20:08:26.2050590Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-08-26T20:08:26.2050760Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-08-26T20:08:26.2051098Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-08-26T20:08:26.2053460Z * [new tag] ciflow/rocm-mi300/159158 -> ciflow/rocm-mi300/159158 2025-08-26T20:08:26.2053777Z * [new tag] ciflow/rocm-mi300/161040 -> ciflow/rocm-mi300/161040 2025-08-26T20:08:26.2053952Z * [new tag] ciflow/rocm-mi300/161180 -> ciflow/rocm-mi300/161180 2025-08-26T20:08:26.2054183Z * [new tag] ciflow/rocm-mi300/161225 -> ciflow/rocm-mi300/161225 2025-08-26T20:08:26.2054333Z * [new tag] ciflow/rocm-mi300/161429 -> ciflow/rocm-mi300/161429 2025-08-26T20:08:26.2054457Z * [new tag] ciflow/rocm-mi355/160215 -> ciflow/rocm-mi355/160215 2025-08-26T20:08:26.2054791Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-08-26T20:08:26.2055545Z * [new tag] ciflow/rocm/151845 -> ciflow/rocm/151845 2025-08-26T20:08:26.2055817Z * [new tag] ciflow/rocm/152526 -> ciflow/rocm/152526 2025-08-26T20:08:26.2059536Z * [new tag] ciflow/rocm/154864 -> ciflow/rocm/154864 2025-08-26T20:08:26.2059846Z * [new tag] ciflow/rocm/156491 -> ciflow/rocm/156491 2025-08-26T20:08:26.2059986Z * [new tag] ciflow/rocm/158352 -> ciflow/rocm/158352 2025-08-26T20:08:26.2060232Z * [new tag] ciflow/rocm/159158 -> ciflow/rocm/159158 2025-08-26T20:08:26.2060360Z * [new tag] ciflow/rocm/160215 -> ciflow/rocm/160215 2025-08-26T20:08:26.2060476Z * [new tag] ciflow/rocm/160671 -> ciflow/rocm/160671 2025-08-26T20:08:26.2060714Z * [new tag] ciflow/rocm/160676 -> ciflow/rocm/160676 2025-08-26T20:08:26.2060854Z * [new tag] ciflow/rocm/161180 -> ciflow/rocm/161180 2025-08-26T20:08:26.2060964Z * [new tag] ciflow/rocm/161225 -> ciflow/rocm/161225 2025-08-26T20:08:26.2061495Z * [new tag] ciflow/rocm/161277 -> ciflow/rocm/161277 2025-08-26T20:08:26.2066380Z * [new tag] ciflow/rocm/161429 -> ciflow/rocm/161429 2025-08-26T20:08:26.2068574Z * [new tag] ciflow/rocm/161496 -> ciflow/rocm/161496 2025-08-26T20:08:26.2068806Z * [new tag] ciflow/s390/160893 -> ciflow/s390/160893 2025-08-26T20:08:26.2068999Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-08-26T20:08:26.2069167Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-08-26T20:08:26.2069600Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-08-26T20:08:26.2069808Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-08-26T20:08:26.2076427Z * [new tag] ciflow/slow/161182 -> ciflow/slow/161182 2025-08-26T20:08:26.2076716Z * [new tag] ciflow/slow/161395 -> ciflow/slow/161395 2025-08-26T20:08:26.2077180Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-08-26T20:08:26.2077330Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-08-26T20:08:26.2077715Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-08-26T20:08:26.2083523Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-08-26T20:08:26.2089086Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-08-26T20:08:26.2091305Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-08-26T20:08:26.2091475Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-08-26T20:08:26.2091607Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-08-26T20:08:26.2091937Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-08-26T20:08:26.2092070Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-08-26T20:08:26.2092193Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-08-26T20:08:26.2092319Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-08-26T20:08:26.2092444Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-08-26T20:08:26.2092560Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-08-26T20:08:26.2092888Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-08-26T20:08:26.2093030Z * [new tag] ciflow/torchbench/158137 -> ciflow/torchbench/158137 2025-08-26T20:08:26.2093292Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-08-26T20:08:26.2093425Z * [new tag] ciflow/trunk/151845 -> ciflow/trunk/151845 2025-08-26T20:08:26.2093543Z * [new tag] ciflow/trunk/153784 -> ciflow/trunk/153784 2025-08-26T20:08:26.2093670Z * [new tag] ciflow/trunk/154694 -> ciflow/trunk/154694 2025-08-26T20:08:26.2093790Z * [new tag] ciflow/trunk/154864 -> ciflow/trunk/154864 2025-08-26T20:08:26.2093914Z * [new tag] ciflow/trunk/156418 -> ciflow/trunk/156418 2025-08-26T20:08:26.2094031Z * [new tag] ciflow/trunk/157196 -> ciflow/trunk/157196 2025-08-26T20:08:26.2094145Z * [new tag] ciflow/trunk/157537 -> ciflow/trunk/157537 2025-08-26T20:08:26.2094267Z * [new tag] ciflow/trunk/157767 -> ciflow/trunk/157767 2025-08-26T20:08:26.2094382Z * [new tag] ciflow/trunk/157944 -> ciflow/trunk/157944 2025-08-26T20:08:26.2094502Z * [new tag] ciflow/trunk/158104 -> ciflow/trunk/158104 2025-08-26T20:08:26.2094614Z * [new tag] ciflow/trunk/158541 -> ciflow/trunk/158541 2025-08-26T20:08:26.2094725Z * [new tag] ciflow/trunk/158733 -> ciflow/trunk/158733 2025-08-26T20:08:26.2094849Z * [new tag] ciflow/trunk/158747 -> ciflow/trunk/158747 2025-08-26T20:08:26.2094962Z * [new tag] ciflow/trunk/159158 -> ciflow/trunk/159158 2025-08-26T20:08:26.2095082Z * [new tag] ciflow/trunk/159387 -> ciflow/trunk/159387 2025-08-26T20:08:26.2095194Z * [new tag] ciflow/trunk/159562 -> ciflow/trunk/159562 2025-08-26T20:08:26.2095316Z * [new tag] ciflow/trunk/159786 -> ciflow/trunk/159786 2025-08-26T20:08:26.2095430Z * [new tag] ciflow/trunk/159835 -> ciflow/trunk/159835 2025-08-26T20:08:26.2095547Z * [new tag] ciflow/trunk/159889 -> ciflow/trunk/159889 2025-08-26T20:08:26.2095668Z * [new tag] ciflow/trunk/159923 -> ciflow/trunk/159923 2025-08-26T20:08:26.2095781Z * [new tag] ciflow/trunk/160156 -> ciflow/trunk/160156 2025-08-26T20:08:26.2095903Z * [new tag] ciflow/trunk/160180 -> ciflow/trunk/160180 2025-08-26T20:08:26.2096063Z * [new tag] ciflow/trunk/160198 -> ciflow/trunk/160198 2025-08-26T20:08:26.2096346Z * [new tag] ciflow/trunk/160258 -> ciflow/trunk/160258 2025-08-26T20:08:26.2096483Z * [new tag] ciflow/trunk/160431 -> ciflow/trunk/160431 2025-08-26T20:08:26.2096598Z * [new tag] ciflow/trunk/160448 -> ciflow/trunk/160448 2025-08-26T20:08:26.2096722Z * [new tag] ciflow/trunk/160449 -> ciflow/trunk/160449 2025-08-26T20:08:26.2096841Z * [new tag] ciflow/trunk/160527 -> ciflow/trunk/160527 2025-08-26T20:08:26.2097101Z * [new tag] ciflow/trunk/160532 -> ciflow/trunk/160532 2025-08-26T20:08:26.2099974Z * [new tag] ciflow/trunk/160671 -> ciflow/trunk/160671 2025-08-26T20:08:26.2100226Z * [new tag] ciflow/trunk/160677 -> ciflow/trunk/160677 2025-08-26T20:08:26.2105485Z * [new tag] ciflow/trunk/160692 -> ciflow/trunk/160692 2025-08-26T20:08:26.2110699Z * [new tag] ciflow/trunk/160781 -> ciflow/trunk/160781 2025-08-26T20:08:26.2115053Z * [new tag] ciflow/trunk/160825 -> ciflow/trunk/160825 2025-08-26T20:08:26.2115214Z * [new tag] ciflow/trunk/160836 -> ciflow/trunk/160836 2025-08-26T20:08:26.2115351Z * [new tag] ciflow/trunk/160866 -> ciflow/trunk/160866 2025-08-26T20:08:26.2115722Z * [new tag] ciflow/trunk/160915 -> ciflow/trunk/160915 2025-08-26T20:08:26.2115857Z * [new tag] ciflow/trunk/160991 -> ciflow/trunk/160991 2025-08-26T20:08:26.2115980Z * [new tag] ciflow/trunk/160992 -> ciflow/trunk/160992 2025-08-26T20:08:26.2116131Z * [new tag] ciflow/trunk/161004 -> ciflow/trunk/161004 2025-08-26T20:08:26.2116253Z * [new tag] ciflow/trunk/161016 -> ciflow/trunk/161016 2025-08-26T20:08:26.2116378Z * [new tag] ciflow/trunk/161023 -> ciflow/trunk/161023 2025-08-26T20:08:26.2116495Z * [new tag] ciflow/trunk/161026 -> ciflow/trunk/161026 2025-08-26T20:08:26.2116611Z * [new tag] ciflow/trunk/161032 -> ciflow/trunk/161032 2025-08-26T20:08:26.2116731Z * [new tag] ciflow/trunk/161035 -> ciflow/trunk/161035 2025-08-26T20:08:26.2116843Z * [new tag] ciflow/trunk/161040 -> ciflow/trunk/161040 2025-08-26T20:08:26.2116967Z * [new tag] ciflow/trunk/161094 -> ciflow/trunk/161094 2025-08-26T20:08:26.2117080Z * [new tag] ciflow/trunk/161097 -> ciflow/trunk/161097 2025-08-26T20:08:26.2117195Z * [new tag] ciflow/trunk/161098 -> ciflow/trunk/161098 2025-08-26T20:08:26.2117314Z * [new tag] ciflow/trunk/161100 -> ciflow/trunk/161100 2025-08-26T20:08:26.2117431Z * [new tag] ciflow/trunk/161106 -> ciflow/trunk/161106 2025-08-26T20:08:26.2117551Z * [new tag] ciflow/trunk/161110 -> ciflow/trunk/161110 2025-08-26T20:08:26.2117663Z * [new tag] ciflow/trunk/161114 -> ciflow/trunk/161114 2025-08-26T20:08:26.2117779Z * [new tag] ciflow/trunk/161117 -> ciflow/trunk/161117 2025-08-26T20:08:26.2117902Z * [new tag] ciflow/trunk/161123 -> ciflow/trunk/161123 2025-08-26T20:08:26.2118021Z * [new tag] ciflow/trunk/161124 -> ciflow/trunk/161124 2025-08-26T20:08:26.2118142Z * [new tag] ciflow/trunk/161126 -> ciflow/trunk/161126 2025-08-26T20:08:26.2118257Z * [new tag] ciflow/trunk/161131 -> ciflow/trunk/161131 2025-08-26T20:08:26.2118376Z * [new tag] ciflow/trunk/161143 -> ciflow/trunk/161143 2025-08-26T20:08:26.2118587Z * [new tag] ciflow/trunk/161144 -> ciflow/trunk/161144 2025-08-26T20:08:26.2118702Z * [new tag] ciflow/trunk/161164 -> ciflow/trunk/161164 2025-08-26T20:08:26.2118824Z * [new tag] ciflow/trunk/161180 -> ciflow/trunk/161180 2025-08-26T20:08:26.2118939Z * [new tag] ciflow/trunk/161214 -> ciflow/trunk/161214 2025-08-26T20:08:26.2119060Z * [new tag] ciflow/trunk/161217 -> ciflow/trunk/161217 2025-08-26T20:08:26.2119433Z * [new tag] ciflow/trunk/161225 -> ciflow/trunk/161225 2025-08-26T20:08:26.2119564Z * [new tag] ciflow/trunk/161236 -> ciflow/trunk/161236 2025-08-26T20:08:26.2119692Z * [new tag] ciflow/trunk/161237 -> ciflow/trunk/161237 2025-08-26T20:08:26.2119810Z * [new tag] ciflow/trunk/161241 -> ciflow/trunk/161241 2025-08-26T20:08:26.2119938Z * [new tag] ciflow/trunk/161262 -> ciflow/trunk/161262 2025-08-26T20:08:26.2120059Z * [new tag] ciflow/trunk/161263 -> ciflow/trunk/161263 2025-08-26T20:08:26.2120185Z * [new tag] ciflow/trunk/161279 -> ciflow/trunk/161279 2025-08-26T20:08:26.2120307Z * [new tag] ciflow/trunk/161306 -> ciflow/trunk/161306 2025-08-26T20:08:26.2120427Z * [new tag] ciflow/trunk/161311 -> ciflow/trunk/161311 2025-08-26T20:08:26.2120554Z * [new tag] ciflow/trunk/161354 -> ciflow/trunk/161354 2025-08-26T20:08:26.2120729Z * [new tag] ciflow/trunk/161355 -> ciflow/trunk/161355 2025-08-26T20:08:26.2120847Z * [new tag] ciflow/trunk/161362 -> ciflow/trunk/161362 2025-08-26T20:08:26.2120956Z * [new tag] ciflow/trunk/161363 -> ciflow/trunk/161363 2025-08-26T20:08:26.2121065Z * [new tag] ciflow/trunk/161370 -> ciflow/trunk/161370 2025-08-26T20:08:26.2121185Z * [new tag] ciflow/trunk/161383 -> ciflow/trunk/161383 2025-08-26T20:08:26.2121295Z * [new tag] ciflow/trunk/161385 -> ciflow/trunk/161385 2025-08-26T20:08:26.2121412Z * [new tag] ciflow/trunk/161389 -> ciflow/trunk/161389 2025-08-26T20:08:26.2121519Z * [new tag] ciflow/trunk/161392 -> ciflow/trunk/161392 2025-08-26T20:08:26.2121634Z * [new tag] ciflow/trunk/161395 -> ciflow/trunk/161395 2025-08-26T20:08:26.2121745Z * [new tag] ciflow/trunk/161396 -> ciflow/trunk/161396 2025-08-26T20:08:26.2121854Z * [new tag] ciflow/trunk/161409 -> ciflow/trunk/161409 2025-08-26T20:08:26.2121968Z * [new tag] ciflow/trunk/161410 -> ciflow/trunk/161410 2025-08-26T20:08:26.2122075Z * [new tag] ciflow/trunk/161435 -> ciflow/trunk/161435 2025-08-26T20:08:26.2122194Z * [new tag] ciflow/trunk/161437 -> ciflow/trunk/161437 2025-08-26T20:08:26.2122312Z * [new tag] ciflow/trunk/161451 -> ciflow/trunk/161451 2025-08-26T20:08:26.2122424Z * [new tag] ciflow/trunk/161453 -> ciflow/trunk/161453 2025-08-26T20:08:26.2122534Z * [new tag] ciflow/trunk/161454 -> ciflow/trunk/161454 2025-08-26T20:08:26.2122644Z * [new tag] ciflow/trunk/161489 -> ciflow/trunk/161489 2025-08-26T20:08:26.2122760Z * [new tag] ciflow/trunk/161517 -> ciflow/trunk/161517 2025-08-26T20:08:26.2122877Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-08-26T20:08:26.2123012Z * [new tag] ciflow/win-arm64/158104 -> ciflow/win-arm64/158104 2025-08-26T20:08:26.2123137Z * [new tag] ciflow/win-arm64/159562 -> ciflow/win-arm64/159562 2025-08-26T20:08:26.2128170Z * [new tag] ciflow/win-arm64/160258 -> ciflow/win-arm64/160258 2025-08-26T20:08:26.2134564Z * [new tag] ciflow/win-arm64/161504 -> ciflow/win-arm64/161504 2025-08-26T20:08:26.2134721Z * [new tag] ciflow/xpu/143553 -> ciflow/xpu/143553 2025-08-26T20:08:26.2134852Z * [new tag] ciflow/xpu/158733 -> ciflow/xpu/158733 2025-08-26T20:08:26.2134957Z * [new tag] ciflow/xpu/159473 -> ciflow/xpu/159473 2025-08-26T20:08:26.2135061Z * [new tag] ciflow/xpu/159944 -> ciflow/xpu/159944 2025-08-26T20:08:26.2135187Z * [new tag] ciflow/xpu/160067 -> ciflow/xpu/160067 2025-08-26T20:08:26.2135291Z * [new tag] ciflow/xpu/160158 -> ciflow/xpu/160158 2025-08-26T20:08:26.2135402Z * [new tag] ciflow/xpu/160940 -> ciflow/xpu/160940 2025-08-26T20:08:26.2135506Z * [new tag] ciflow/xpu/161041 -> ciflow/xpu/161041 2025-08-26T20:08:26.2135641Z * [new tag] ciflow/xpu/161045 -> ciflow/xpu/161045 2025-08-26T20:08:26.2135747Z * [new tag] ciflow/xpu/161142 -> ciflow/xpu/161142 2025-08-26T20:08:26.2135848Z * [new tag] ciflow/xpu/161152 -> ciflow/xpu/161152 2025-08-26T20:08:26.2135959Z * [new tag] ciflow/xpu/161246 -> ciflow/xpu/161246 2025-08-26T20:08:26.2136062Z * [new tag] ciflow/xpu/161389 -> ciflow/xpu/161389 2025-08-26T20:08:26.2136171Z * [new tag] ciflow/xpu/161392 -> ciflow/xpu/161392 2025-08-26T20:08:26.2136408Z * [new tag] ciflow/xpu/161397 -> ciflow/xpu/161397 2025-08-26T20:08:26.2136520Z * [new tag] ciflow/xpu/161477 -> ciflow/xpu/161477 2025-08-26T20:08:26.2136631Z * [new tag] ciflow/xpu/161489 -> ciflow/xpu/161489 2025-08-26T20:08:26.2136740Z * [new tag] cslpull75 -> cslpull75 2025-08-26T20:08:26.2136853Z * [new tag] cslpull76 -> cslpull76 2025-08-26T20:08:26.2136949Z * [new tag] cslpull77 -> cslpull77 2025-08-26T20:08:26.2137051Z * [new tag] cslpull78 -> cslpull78 2025-08-26T20:08:26.2137155Z * [new tag] cslpull79 -> cslpull79 2025-08-26T20:08:26.2137247Z * [new tag] cslpull80 -> cslpull80 2025-08-26T20:08:26.2137349Z * [new tag] cslpull81 -> cslpull81 2025-08-26T20:08:26.2137444Z * [new tag] cslpull82 -> cslpull82 2025-08-26T20:08:26.2137543Z * [new tag] cslpull83 -> cslpull83 2025-08-26T20:08:26.2137637Z * [new tag] cslpull84 -> cslpull84 2025-08-26T20:08:26.2137728Z * [new tag] cslpull85 -> cslpull85 2025-08-26T20:08:26.2137840Z * [new tag] cslpull86 -> cslpull86 2025-08-26T20:08:26.2137932Z * [new tag] cslpull87 -> cslpull87 2025-08-26T20:08:26.2138030Z * [new tag] cslpull88 -> cslpull88 2025-08-26T20:08:26.2144988Z * [new tag] cslpull89 -> cslpull89 2025-08-26T20:08:26.2145273Z * [new tag] cslpull90 -> cslpull90 2025-08-26T20:08:26.2145399Z * [new tag] cslpull91 -> cslpull91 2025-08-26T20:08:26.2145517Z * [new tag] cslpull92 -> cslpull92 2025-08-26T20:08:26.2145622Z * [new tag] flight_5 -> flight_5 2025-08-26T20:08:26.2145725Z * [new tag] flight_5.1 -> flight_5.1 2025-08-26T20:08:26.2145956Z * [new tag] flight_5.2 -> flight_5.2 2025-08-26T20:08:26.2146205Z * [new tag] flight_5.3 -> flight_5.3 2025-08-26T20:08:26.2146316Z * [new tag] forpull1 -> forpull1 2025-08-26T20:08:26.2146581Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-08-26T20:08:26.2146723Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-08-26T20:08:26.2146924Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-08-26T20:08:26.2147560Z * [new tag] nightly-binary -> nightly-binary 2025-08-26T20:08:26.2148178Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-08-26T20:08:26.2148330Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-08-26T20:08:26.2148657Z * [new tag] trunk/00efeabc295e072fb9d6e68b008a31fb04201fd1 -> trunk/00efeabc295e072fb9d6e68b008a31fb04201fd1 2025-08-26T20:08:26.2148934Z * [new tag] trunk/037c43d3b24d4db733011cb904c385eaa6e11bcf -> trunk/037c43d3b24d4db733011cb904c385eaa6e11bcf 2025-08-26T20:08:26.2149230Z * [new tag] trunk/0533ff2ccba7e77622ac3c6758f1032bdc10feff -> trunk/0533ff2ccba7e77622ac3c6758f1032bdc10feff 2025-08-26T20:08:26.2154236Z * [new tag] trunk/05e8fac4f374c4dbf0cd0e85e925e9112cf234a2 -> trunk/05e8fac4f374c4dbf0cd0e85e925e9112cf234a2 2025-08-26T20:08:26.2154700Z * [new tag] trunk/089ad1d88bf31ddab769a4f87750b474ed1214c8 -> trunk/089ad1d88bf31ddab769a4f87750b474ed1214c8 2025-08-26T20:08:26.2155244Z * [new tag] trunk/0924304e728b9507a54eced28c812fbd5b13c397 -> trunk/0924304e728b9507a54eced28c812fbd5b13c397 2025-08-26T20:08:26.2155527Z * [new tag] trunk/0a5ab612dd2b9fc5bb2e1281ec7ca8730c5c3c89 -> trunk/0a5ab612dd2b9fc5bb2e1281ec7ca8730c5c3c89 2025-08-26T20:08:26.2155793Z * [new tag] trunk/0d19541284c38212235f78db24e3ac3ae4787e45 -> trunk/0d19541284c38212235f78db24e3ac3ae4787e45 2025-08-26T20:08:26.2156047Z * [new tag] trunk/0d9da384ef76e3ce2e7eaf951252ae9edb922863 -> trunk/0d9da384ef76e3ce2e7eaf951252ae9edb922863 2025-08-26T20:08:26.2156339Z * [new tag] trunk/0dea191ff7b844352dc2cd5e3b5ef5ea13a76756 -> trunk/0dea191ff7b844352dc2cd5e3b5ef5ea13a76756 2025-08-26T20:08:26.2156568Z * [new tag] trunk/0f801a510f5f185543388717241adb7237c3d46a -> trunk/0f801a510f5f185543388717241adb7237c3d46a 2025-08-26T20:08:26.2156816Z * [new tag] trunk/10e67f5ec3834da93fc2022caa7ac69cf97c01f0 -> trunk/10e67f5ec3834da93fc2022caa7ac69cf97c01f0 2025-08-26T20:08:26.2157058Z * [new tag] trunk/1113e7de30da95973c1eac7921601f9a0e94f2db -> trunk/1113e7de30da95973c1eac7921601f9a0e94f2db 2025-08-26T20:08:26.2157307Z * [new tag] trunk/117f11adb4b41a5485b570c4337c22ecc8e00aeb -> trunk/117f11adb4b41a5485b570c4337c22ecc8e00aeb 2025-08-26T20:08:26.2157551Z * [new tag] trunk/1471b20cb3fc502931ef12b1420414e32facd5b0 -> trunk/1471b20cb3fc502931ef12b1420414e32facd5b0 2025-08-26T20:08:26.2157797Z * [new tag] trunk/16e811e0b5073c7b42fe76f650ca2b79e339e053 -> trunk/16e811e0b5073c7b42fe76f650ca2b79e339e053 2025-08-26T20:08:26.2158056Z * [new tag] trunk/17b0263e86aec8aed068bb8b6744b129233e8084 -> trunk/17b0263e86aec8aed068bb8b6744b129233e8084 2025-08-26T20:08:26.2158296Z * [new tag] trunk/18271148d32da3d48897e9e7515de45066fce5bc -> trunk/18271148d32da3d48897e9e7515de45066fce5bc 2025-08-26T20:08:26.2158539Z * [new tag] trunk/19c70c2f3dc345a6555318f5f8b46cd55c42d0b4 -> trunk/19c70c2f3dc345a6555318f5f8b46cd55c42d0b4 2025-08-26T20:08:26.2158776Z * [new tag] trunk/1a566c4909ccf16ace1fbf1f65d90c995b362712 -> trunk/1a566c4909ccf16ace1fbf1f65d90c995b362712 2025-08-26T20:08:26.2159047Z * [new tag] trunk/1d458e294755ff2bfa314c67ddc5cb1dacc2aee8 -> trunk/1d458e294755ff2bfa314c67ddc5cb1dacc2aee8 2025-08-26T20:08:26.2159608Z * [new tag] trunk/1d46aa736fc8870dc88015c729a8c64470fa985c -> trunk/1d46aa736fc8870dc88015c729a8c64470fa985c 2025-08-26T20:08:26.2159888Z * [new tag] trunk/1de4540449ad6b9df8f452ab72da30ce8908af60 -> trunk/1de4540449ad6b9df8f452ab72da30ce8908af60 2025-08-26T20:08:26.2160133Z * [new tag] trunk/1e3fe78a104776cd708f150116348540346dae25 -> trunk/1e3fe78a104776cd708f150116348540346dae25 2025-08-26T20:08:26.2160386Z * [new tag] trunk/1ea918caf990c84bcb4e4ee5eee90f1102815b0a -> trunk/1ea918caf990c84bcb4e4ee5eee90f1102815b0a 2025-08-26T20:08:26.2160650Z * [new tag] trunk/1eccfb157ab9855b3f81872a23502fb15f455e0a -> trunk/1eccfb157ab9855b3f81872a23502fb15f455e0a 2025-08-26T20:08:26.2161039Z * [new tag] trunk/1fbe230b0d82251c6de8b5ae86c4da456b1db05c -> trunk/1fbe230b0d82251c6de8b5ae86c4da456b1db05c 2025-08-26T20:08:26.2161604Z * [new tag] trunk/209143ddeb99b0b075d16525088cee4893be7492 -> trunk/209143ddeb99b0b075d16525088cee4893be7492 2025-08-26T20:08:26.2161987Z * [new tag] trunk/22df59efc0a845b3ff37019029efd07c5a25c456 -> trunk/22df59efc0a845b3ff37019029efd07c5a25c456 2025-08-26T20:08:26.2162311Z * [new tag] trunk/23b033452fb1d4b404216279bbf5b6d06d8570c3 -> trunk/23b033452fb1d4b404216279bbf5b6d06d8570c3 2025-08-26T20:08:26.2162666Z * [new tag] trunk/24e7f3c21c9452c81d72bbd4b0c6b1f96f33536a -> trunk/24e7f3c21c9452c81d72bbd4b0c6b1f96f33536a 2025-08-26T20:08:26.2163006Z * [new tag] trunk/25df65afd8b5e2fffbcaf2b7ed63ef7a1e37ecb9 -> trunk/25df65afd8b5e2fffbcaf2b7ed63ef7a1e37ecb9 2025-08-26T20:08:26.2163409Z * [new tag] trunk/262640fd220236042fbf4443cc163c8838c84c3d -> trunk/262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:08:26.2163861Z * [new tag] trunk/266784ec6ae82f823abe406582e7a91f2ebb564a -> trunk/266784ec6ae82f823abe406582e7a91f2ebb564a 2025-08-26T20:08:26.2164477Z * [new tag] trunk/2835cc5e91eda8cbc4ac59de2ca990fa17107409 -> trunk/2835cc5e91eda8cbc4ac59de2ca990fa17107409 2025-08-26T20:08:26.2164869Z * [new tag] trunk/284b7190054686e68d9cc683b6ce43e45dd22338 -> trunk/284b7190054686e68d9cc683b6ce43e45dd22338 2025-08-26T20:08:26.2165772Z * [new tag] trunk/29afde20203ee6773641b4e3552942a37315316f -> trunk/29afde20203ee6773641b4e3552942a37315316f 2025-08-26T20:08:26.2166009Z * [new tag] trunk/2a7a7ad7116d930fde86cda02f668e624d26ec3e -> trunk/2a7a7ad7116d930fde86cda02f668e624d26ec3e 2025-08-26T20:08:26.2168421Z * [new tag] trunk/2b62ef74208792c7c4bf923f872e54b5f384efc8 -> trunk/2b62ef74208792c7c4bf923f872e54b5f384efc8 2025-08-26T20:08:26.2168879Z * [new tag] trunk/2beffb3311a41589021c121dac543994a7cbdff2 -> trunk/2beffb3311a41589021c121dac543994a7cbdff2 2025-08-26T20:08:26.2169274Z * [new tag] trunk/2c0650a00a0a0dd2bbf25ed22780fdd881bcda54 -> trunk/2c0650a00a0a0dd2bbf25ed22780fdd881bcda54 2025-08-26T20:08:26.2169690Z * [new tag] trunk/2cf69fe0e1bdb1413fe9e802c4b84d8958708421 -> trunk/2cf69fe0e1bdb1413fe9e802c4b84d8958708421 2025-08-26T20:08:26.2170078Z * [new tag] trunk/2cf7ac2fb7ab4067e17cc5ca71034b1c61a4fb10 -> trunk/2cf7ac2fb7ab4067e17cc5ca71034b1c61a4fb10 2025-08-26T20:08:26.2170417Z * [new tag] trunk/2f0cba934de7094a66c6ce68f5e937254f23142a -> trunk/2f0cba934de7094a66c6ce68f5e937254f23142a 2025-08-26T20:08:26.2175445Z * [new tag] trunk/2f0de0ff9361ca4f2b1e6f9edbc600b5fb6abcd6 -> trunk/2f0de0ff9361ca4f2b1e6f9edbc600b5fb6abcd6 2025-08-26T20:08:26.2175762Z * [new tag] trunk/2f50ae7d2022cb096c4156f5a207c291e36ddecf -> trunk/2f50ae7d2022cb096c4156f5a207c291e36ddecf 2025-08-26T20:08:26.2176016Z * [new tag] trunk/2fdd4f918cdc5fc8070e4c9c0d87b9045d316c06 -> trunk/2fdd4f918cdc5fc8070e4c9c0d87b9045d316c06 2025-08-26T20:08:26.2176266Z * [new tag] trunk/30384abcb1d181e774c0ac21b580aa34336a96c6 -> trunk/30384abcb1d181e774c0ac21b580aa34336a96c6 2025-08-26T20:08:26.2176670Z * [new tag] trunk/31a41daff49f2cde941d8b9e35cb2eaeeb606c0d -> trunk/31a41daff49f2cde941d8b9e35cb2eaeeb606c0d 2025-08-26T20:08:26.2176909Z * [new tag] trunk/332fa5b388521c05a19217649745c6edfdc2836d -> trunk/332fa5b388521c05a19217649745c6edfdc2836d 2025-08-26T20:08:26.2177147Z * [new tag] trunk/33346b58148c55592994a43385c321ae8c8808f2 -> trunk/33346b58148c55592994a43385c321ae8c8808f2 2025-08-26T20:08:26.2182142Z * [new tag] trunk/3373b074f5ea5277974fa6e945544fdfb16bb446 -> trunk/3373b074f5ea5277974fa6e945544fdfb16bb446 2025-08-26T20:08:26.2182551Z * [new tag] trunk/33c3794533844236a6e30ba377e0a6802b279fc8 -> trunk/33c3794533844236a6e30ba377e0a6802b279fc8 2025-08-26T20:08:26.2182936Z * [new tag] trunk/35e4d97e047bff8b38fee1dcf6ef6503f0fc9208 -> trunk/35e4d97e047bff8b38fee1dcf6ef6503f0fc9208 2025-08-26T20:08:26.2183292Z * [new tag] trunk/36ac916929ca67b533cc45932970297e9824324e -> trunk/36ac916929ca67b533cc45932970297e9824324e 2025-08-26T20:08:26.2183648Z * [new tag] trunk/371909cfd10e0da1bab1e12fb54a2403c37c5f76 -> trunk/371909cfd10e0da1bab1e12fb54a2403c37c5f76 2025-08-26T20:08:26.2184327Z * [new tag] trunk/373e25c2eb9f882356a9c7a2f18020935ff1d78b -> trunk/373e25c2eb9f882356a9c7a2f18020935ff1d78b 2025-08-26T20:08:26.2184634Z * [new tag] trunk/37a34022b59a6ff2757e5cec0fdc72278418f339 -> trunk/37a34022b59a6ff2757e5cec0fdc72278418f339 2025-08-26T20:08:26.2185038Z * [new tag] trunk/38a492d40d7ebb2856cb120df337c6cdac244528 -> trunk/38a492d40d7ebb2856cb120df337c6cdac244528 2025-08-26T20:08:26.2185300Z * [new tag] trunk/394728bab2de21e8002fc6a47aa4d3acb2d7a728 -> trunk/394728bab2de21e8002fc6a47aa4d3acb2d7a728 2025-08-26T20:08:26.2185544Z * [new tag] trunk/39862acb2e320783245d2a03acfd1b14cae28038 -> trunk/39862acb2e320783245d2a03acfd1b14cae28038 2025-08-26T20:08:26.2185800Z * [new tag] trunk/3a4140bf8e783db3f0094d2a2ce1d8534066432f -> trunk/3a4140bf8e783db3f0094d2a2ce1d8534066432f 2025-08-26T20:08:26.2186040Z * [new tag] trunk/3caddd4daa5b1a167663c07219e065e86247ad76 -> trunk/3caddd4daa5b1a167663c07219e065e86247ad76 2025-08-26T20:08:26.2186288Z * [new tag] trunk/3dacaf0e1eb3286e70bf8d572000ecebf2c1f4c9 -> trunk/3dacaf0e1eb3286e70bf8d572000ecebf2c1f4c9 2025-08-26T20:08:26.2186539Z * [new tag] trunk/3e210f90c2cbd5817aa23d430da10cad200a3ffa -> trunk/3e210f90c2cbd5817aa23d430da10cad200a3ffa 2025-08-26T20:08:26.2186778Z * [new tag] trunk/3e3e83418d0f6b1495f79380f3a3dbc8b2d23062 -> trunk/3e3e83418d0f6b1495f79380f3a3dbc8b2d23062 2025-08-26T20:08:26.2187024Z * [new tag] trunk/3e5b021f217a42ae55dc690083f67a28126808ed -> trunk/3e5b021f217a42ae55dc690083f67a28126808ed 2025-08-26T20:08:26.2187246Z * [new tag] trunk/3ea6cc8c2d443d6104159d50e8328c144f6caa39 -> trunk/3ea6cc8c2d443d6104159d50e8328c144f6caa39 2025-08-26T20:08:26.2187485Z * [new tag] trunk/3f1a97a99cad4cc682b20b43c1178ed9e1b81f24 -> trunk/3f1a97a99cad4cc682b20b43c1178ed9e1b81f24 2025-08-26T20:08:26.2187713Z * [new tag] trunk/3f5a8e2003f2234ca8be19fdc307ba7b995f9be3 -> trunk/3f5a8e2003f2234ca8be19fdc307ba7b995f9be3 2025-08-26T20:08:26.2187942Z * [new tag] trunk/40c0e700a488191cd8f541b30d8e3b9f2c0bc759 -> trunk/40c0e700a488191cd8f541b30d8e3b9f2c0bc759 2025-08-26T20:08:26.2188166Z * [new tag] trunk/419a2dbf5f69cee52382090200b532a81da92c69 -> trunk/419a2dbf5f69cee52382090200b532a81da92c69 2025-08-26T20:08:26.2188383Z * [new tag] trunk/431846a6323c6f1d02da49e311ac694324f386f4 -> trunk/431846a6323c6f1d02da49e311ac694324f386f4 2025-08-26T20:08:26.2188810Z * [new tag] trunk/44549c7146bd6c4166f97e856037babe1b7f4f49 -> trunk/44549c7146bd6c4166f97e856037babe1b7f4f49 2025-08-26T20:08:26.2189378Z * [new tag] trunk/447d34b5f80fb7350f79decd855cb599cab39083 -> trunk/447d34b5f80fb7350f79decd855cb599cab39083 2025-08-26T20:08:26.2189601Z * [new tag] trunk/46429be72323c1807a785234164bd91011f68d08 -> trunk/46429be72323c1807a785234164bd91011f68d08 2025-08-26T20:08:26.2189828Z * [new tag] trunk/4651aaac47ff855e08a74e2fdbfa605bc53afba8 -> trunk/4651aaac47ff855e08a74e2fdbfa605bc53afba8 2025-08-26T20:08:26.2190061Z * [new tag] trunk/46576f5a164fcf95ec7fceaa13516bcb1ca4f6ab -> trunk/46576f5a164fcf95ec7fceaa13516bcb1ca4f6ab 2025-08-26T20:08:26.2190291Z * [new tag] trunk/47d267364cad407b5612bf4a5faa160d2f4a7121 -> trunk/47d267364cad407b5612bf4a5faa160d2f4a7121 2025-08-26T20:08:26.2190527Z * [new tag] trunk/49ff884b1edc3b872eeb2387ec60ef230cae7f24 -> trunk/49ff884b1edc3b872eeb2387ec60ef230cae7f24 2025-08-26T20:08:26.2193197Z * [new tag] trunk/4a1aca11c20cfa29a1513b9f289d75bfe32d05d4 -> trunk/4a1aca11c20cfa29a1513b9f289d75bfe32d05d4 2025-08-26T20:08:26.2193532Z * [new tag] trunk/4acdbb8311f760513556e2e4fdd7bfd88c225e52 -> trunk/4acdbb8311f760513556e2e4fdd7bfd88c225e52 2025-08-26T20:08:26.2193785Z * [new tag] trunk/4c36c8a99463c898190a462300ba7f05b5b3384e -> trunk/4c36c8a99463c898190a462300ba7f05b5b3384e 2025-08-26T20:08:26.2194029Z * [new tag] trunk/4e19c1906a830714c1d9d71361357ce616a034d6 -> trunk/4e19c1906a830714c1d9d71361357ce616a034d6 2025-08-26T20:08:26.2194464Z * [new tag] trunk/4ed3184dee1bf4f775839bfd1448a7a34fe5a898 -> trunk/4ed3184dee1bf4f775839bfd1448a7a34fe5a898 2025-08-26T20:08:26.2194720Z * [new tag] trunk/50cfe76231768ee2c784f68a1eba03369f386019 -> trunk/50cfe76231768ee2c784f68a1eba03369f386019 2025-08-26T20:08:26.2194974Z * [new tag] trunk/510825e5fed8b56eb5e9352c12f0df1feeadb810 -> trunk/510825e5fed8b56eb5e9352c12f0df1feeadb810 2025-08-26T20:08:26.2195228Z * [new tag] trunk/512fc768e94c937df350911aaa4ebce757d1f9df -> trunk/512fc768e94c937df350911aaa4ebce757d1f9df 2025-08-26T20:08:26.2195490Z * [new tag] trunk/517d38d3406abbba35d0694bff259a698cad3ec9 -> trunk/517d38d3406abbba35d0694bff259a698cad3ec9 2025-08-26T20:08:26.2195746Z * [new tag] trunk/5255e65c01bf48bbcd916ecf16ed81cf28d3c6e2 -> trunk/5255e65c01bf48bbcd916ecf16ed81cf28d3c6e2 2025-08-26T20:08:26.2196010Z * [new tag] trunk/543896fcf3312f2053018edf9ee74c0fbb1d28ed -> trunk/543896fcf3312f2053018edf9ee74c0fbb1d28ed 2025-08-26T20:08:26.2196541Z * [new tag] trunk/54c2b66592d168e4a7525f7a58f8ca020517a9cb -> trunk/54c2b66592d168e4a7525f7a58f8ca020517a9cb 2025-08-26T20:08:26.2196795Z * [new tag] trunk/54cc63b467f24242cf0d6538d3e1df39e553daf1 -> trunk/54cc63b467f24242cf0d6538d3e1df39e553daf1 2025-08-26T20:08:26.2197068Z * [new tag] trunk/56ebed627a23eea36190e1ced5024a18ffcedbd7 -> trunk/56ebed627a23eea36190e1ced5024a18ffcedbd7 2025-08-26T20:08:26.2197317Z * [new tag] trunk/576a0e64ed2470abd2c430205d1984a11951ce05 -> trunk/576a0e64ed2470abd2c430205d1984a11951ce05 2025-08-26T20:08:26.2200095Z * [new tag] trunk/5805c4210b477f0a7315d6038078dc4a8be1c8fa -> trunk/5805c4210b477f0a7315d6038078dc4a8be1c8fa 2025-08-26T20:08:26.2200771Z * [new tag] trunk/58f9a3dd6391397e439c5f5075837e8f983735aa -> trunk/58f9a3dd6391397e439c5f5075837e8f983735aa 2025-08-26T20:08:26.2201057Z * [new tag] trunk/595987d28d4c8aee68de83734af919c7710ad58b -> trunk/595987d28d4c8aee68de83734af919c7710ad58b 2025-08-26T20:08:26.2201343Z * [new tag] trunk/599f639ddb8bb45abb2dc305542f38288427183d -> trunk/599f639ddb8bb45abb2dc305542f38288427183d 2025-08-26T20:08:26.2201601Z * [new tag] trunk/5afa4187dfe1e99278f8e372ec09102d5b937572 -> trunk/5afa4187dfe1e99278f8e372ec09102d5b937572 2025-08-26T20:08:26.2201854Z * [new tag] trunk/5d9653d90ee003173dd03f93e09fed236500ef06 -> trunk/5d9653d90ee003173dd03f93e09fed236500ef06 2025-08-26T20:08:26.2204640Z * [new tag] trunk/5dad5b4f57ade4001c0f421dbdad2e418304870e -> trunk/5dad5b4f57ade4001c0f421dbdad2e418304870e 2025-08-26T20:08:26.2204986Z * [new tag] trunk/5ee464db5c4293ac09521f9069fa7d2106680a7f -> trunk/5ee464db5c4293ac09521f9069fa7d2106680a7f 2025-08-26T20:08:26.2210482Z * [new tag] trunk/6096d277c543f5dd40351431ef9a8d556134c74d -> trunk/6096d277c543f5dd40351431ef9a8d556134c74d 2025-08-26T20:08:26.2212593Z * [new tag] trunk/62db8ec39116544ae247f876b3e06753178db49b -> trunk/62db8ec39116544ae247f876b3e06753178db49b 2025-08-26T20:08:26.2212991Z * [new tag] trunk/639b8cc51ddebf10361f3840a6b0a244eb6092a1 -> trunk/639b8cc51ddebf10361f3840a6b0a244eb6092a1 2025-08-26T20:08:26.2218619Z * [new tag] trunk/6443ea337df843681bc558d99efa84a3e5559b7f -> trunk/6443ea337df843681bc558d99efa84a3e5559b7f 2025-08-26T20:08:26.2221555Z * [new tag] trunk/6598f00c18dfcc4fc50427305b6b5724e617246f -> trunk/6598f00c18dfcc4fc50427305b6b5724e617246f 2025-08-26T20:08:26.2226760Z * [new tag] trunk/65d21dae18a34e8bd1b2f0e5aec7144b9dd33611 -> trunk/65d21dae18a34e8bd1b2f0e5aec7144b9dd33611 2025-08-26T20:08:26.2232424Z * [new tag] trunk/660b5656a436dcccb0275ea5421d3eb4f1157b43 -> trunk/660b5656a436dcccb0275ea5421d3eb4f1157b43 2025-08-26T20:08:26.2232753Z * [new tag] trunk/66166cf1e7696bf25f6f7bb815a93df367db48dc -> trunk/66166cf1e7696bf25f6f7bb815a93df367db48dc 2025-08-26T20:08:26.2233331Z * [new tag] trunk/667245dc60242a35ae0a6b0072628eb8e15a6d03 -> trunk/667245dc60242a35ae0a6b0072628eb8e15a6d03 2025-08-26T20:08:26.2233588Z * [new tag] trunk/67b98da1b262317f9c0375d64a4b467c82712548 -> trunk/67b98da1b262317f9c0375d64a4b467c82712548 2025-08-26T20:08:26.2233824Z * [new tag] trunk/67d31f6b281d3b15b205756fc7ebc450cdde1dab -> trunk/67d31f6b281d3b15b205756fc7ebc450cdde1dab 2025-08-26T20:08:26.2234065Z * [new tag] trunk/67fc16c7447f4fc04e7d28bfe201a4a0c78f3ea4 -> trunk/67fc16c7447f4fc04e7d28bfe201a4a0c78f3ea4 2025-08-26T20:08:26.2234349Z * [new tag] trunk/6aef9f3a6906c011a57541c1de7a246222bc9ac9 -> trunk/6aef9f3a6906c011a57541c1de7a246222bc9ac9 2025-08-26T20:08:26.2234633Z * [new tag] trunk/6ea4be1e2eca952ea66090182bd2eede89799a45 -> trunk/6ea4be1e2eca952ea66090182bd2eede89799a45 2025-08-26T20:08:26.2234893Z * [new tag] trunk/7006fd0c8874cb0228d3f2bfd83a989bde4b7021 -> trunk/7006fd0c8874cb0228d3f2bfd83a989bde4b7021 2025-08-26T20:08:26.2235132Z * [new tag] trunk/710514a2a51facaba445d2c188541d778f9fdb59 -> trunk/710514a2a51facaba445d2c188541d778f9fdb59 2025-08-26T20:08:26.2235381Z * [new tag] trunk/7131bfab89c46ffe31b61ea4937a8727e9cf33c1 -> trunk/7131bfab89c46ffe31b61ea4937a8727e9cf33c1 2025-08-26T20:08:26.2235642Z * [new tag] trunk/726dce3c944cbda16e54d3b15cdb4b6ced05af72 -> trunk/726dce3c944cbda16e54d3b15cdb4b6ced05af72 2025-08-26T20:08:26.2235872Z * [new tag] trunk/72e4786d1635681b8d053d0168c7d16b980e5124 -> trunk/72e4786d1635681b8d053d0168c7d16b980e5124 2025-08-26T20:08:26.2236102Z * [new tag] trunk/7376111d59f3170c2814d565c09d09435189692a -> trunk/7376111d59f3170c2814d565c09d09435189692a 2025-08-26T20:08:26.2236348Z * [new tag] trunk/74124d1b46774f2a73aa1aadc2b0874cb523b1c1 -> trunk/74124d1b46774f2a73aa1aadc2b0874cb523b1c1 2025-08-26T20:08:26.2236590Z * [new tag] trunk/74280d091321343b47a2975e17584b973d7c22c4 -> trunk/74280d091321343b47a2975e17584b973d7c22c4 2025-08-26T20:08:26.2236822Z * [new tag] trunk/74c4c758afa8c28162f00a456c185552e1159fd3 -> trunk/74c4c758afa8c28162f00a456c185552e1159fd3 2025-08-26T20:08:26.2237060Z * [new tag] trunk/763053dc536341997641e920d8887b3010901b3b -> trunk/763053dc536341997641e920d8887b3010901b3b 2025-08-26T20:08:26.2237345Z * [new tag] trunk/774b4befa18741b3115802cae71000168a40c384 -> trunk/774b4befa18741b3115802cae71000168a40c384 2025-08-26T20:08:26.2237594Z * [new tag] trunk/77bc959fe122bfd131e339ca36cab445a1860806 -> trunk/77bc959fe122bfd131e339ca36cab445a1860806 2025-08-26T20:08:26.2237844Z * [new tag] trunk/78a8e6a671c5631bc0e89b0e674790a424540547 -> trunk/78a8e6a671c5631bc0e89b0e674790a424540547 2025-08-26T20:08:26.2238100Z * [new tag] trunk/7e4bfa74eafab994b01f8b5501d4d061cbf64808 -> trunk/7e4bfa74eafab994b01f8b5501d4d061cbf64808 2025-08-26T20:08:26.2238392Z * [new tag] trunk/7e6ce41555d595e3fa0d91059491f21cee3eb5ea -> trunk/7e6ce41555d595e3fa0d91059491f21cee3eb5ea 2025-08-26T20:08:26.2238641Z * [new tag] trunk/7f201baf414301b3312576893b7f6f2698acd9ba -> trunk/7f201baf414301b3312576893b7f6f2698acd9ba 2025-08-26T20:08:26.2238916Z * [new tag] trunk/7fcdd8d6afeda6a4c8630816e12bf7cca44b8f8a -> trunk/7fcdd8d6afeda6a4c8630816e12bf7cca44b8f8a 2025-08-26T20:08:26.2239201Z * [new tag] trunk/801851086d09506d081800108c9e214edb3f5b7d -> trunk/801851086d09506d081800108c9e214edb3f5b7d 2025-08-26T20:08:26.2239467Z * [new tag] trunk/8047cde0f3a27f3afa218792b8464d5e0c9d942f -> trunk/8047cde0f3a27f3afa218792b8464d5e0c9d942f 2025-08-26T20:08:26.2239720Z * [new tag] trunk/80df27a612be3433516d7e6dfc8d8be058425d3e -> trunk/80df27a612be3433516d7e6dfc8d8be058425d3e 2025-08-26T20:08:26.2240021Z * [new tag] trunk/818ba434c7de4cd604184b2857d544e0ad95735f -> trunk/818ba434c7de4cd604184b2857d544e0ad95735f 2025-08-26T20:08:26.2240278Z * [new tag] trunk/83283ce7f5a7847b4e561e22be9b0f4530b05527 -> trunk/83283ce7f5a7847b4e561e22be9b0f4530b05527 2025-08-26T20:08:26.2240526Z * [new tag] trunk/85adf80cf15538a7e010fa235036fe8e06f8bede -> trunk/85adf80cf15538a7e010fa235036fe8e06f8bede 2025-08-26T20:08:26.2240794Z * [new tag] trunk/8aad3a60ce16a4acab17a8e46e5df339db2ff740 -> trunk/8aad3a60ce16a4acab17a8e46e5df339db2ff740 2025-08-26T20:08:26.2241037Z * [new tag] trunk/8c442e4fd3310e15f57770944f883ac1d73e77e2 -> trunk/8c442e4fd3310e15f57770944f883ac1d73e77e2 2025-08-26T20:08:26.2241263Z * [new tag] trunk/8c506e6310b9b5295151fb725be479d0f80ce5e8 -> trunk/8c506e6310b9b5295151fb725be479d0f80ce5e8 2025-08-26T20:08:26.2241492Z * [new tag] trunk/8cfc119491f533c4edded4263a78eb0af782a2d5 -> trunk/8cfc119491f533c4edded4263a78eb0af782a2d5 2025-08-26T20:08:26.2241732Z * [new tag] trunk/8dbe7f99bd707ee28ae12ecb9cab54e1785bf13e -> trunk/8dbe7f99bd707ee28ae12ecb9cab54e1785bf13e 2025-08-26T20:08:26.2241960Z * [new tag] trunk/8e1770905565cd67d6c3a91c7afa462f4ef6e6aa -> trunk/8e1770905565cd67d6c3a91c7afa462f4ef6e6aa 2025-08-26T20:08:26.2242187Z * [new tag] trunk/8f31aa97a3e1e17bed29b6cedf9884f0c6b145e9 -> trunk/8f31aa97a3e1e17bed29b6cedf9884f0c6b145e9 2025-08-26T20:08:26.2242415Z * [new tag] trunk/8f766d68397736053883aa281cae0eb46bb233bb -> trunk/8f766d68397736053883aa281cae0eb46bb233bb 2025-08-26T20:08:26.2242641Z * [new tag] trunk/908b0ccb1f70ed2cfa830484e05ee32af13b1836 -> trunk/908b0ccb1f70ed2cfa830484e05ee32af13b1836 2025-08-26T20:08:26.2242878Z * [new tag] trunk/90ea9ccefe3e2d9a9e4840016d1af10c1814d48b -> trunk/90ea9ccefe3e2d9a9e4840016d1af10c1814d48b 2025-08-26T20:08:26.2243115Z * [new tag] trunk/9225c6199412f8a2ee99b7c29f533fb98b9ff62e -> trunk/9225c6199412f8a2ee99b7c29f533fb98b9ff62e 2025-08-26T20:08:26.2243352Z * [new tag] trunk/923bc46122d173a7964c646311a3bea3cd8dd561 -> trunk/923bc46122d173a7964c646311a3bea3cd8dd561 2025-08-26T20:08:26.2243578Z * [new tag] trunk/92ab18482459a63e97f1374e27e8411964da9762 -> trunk/92ab18482459a63e97f1374e27e8411964da9762 2025-08-26T20:08:26.2243811Z * [new tag] trunk/94b9569c4a86e12b944ca66e3125357a14d0eb9e -> trunk/94b9569c4a86e12b944ca66e3125357a14d0eb9e 2025-08-26T20:08:26.2244159Z * [new tag] trunk/957b170d8efe2a51147e0cdb7515acc345ba81da -> trunk/957b170d8efe2a51147e0cdb7515acc345ba81da 2025-08-26T20:08:26.2244394Z * [new tag] trunk/958f9ca88e9a1580de7c94a5a2ca8a750b1335ae -> trunk/958f9ca88e9a1580de7c94a5a2ca8a750b1335ae 2025-08-26T20:08:26.2244640Z * [new tag] trunk/96682103026b5ea27f19e6db9303e17572095b0e -> trunk/96682103026b5ea27f19e6db9303e17572095b0e 2025-08-26T20:08:26.2244871Z * [new tag] trunk/97200c971110d54030feaad999698c7341f8acc7 -> trunk/97200c971110d54030feaad999698c7341f8acc7 2025-08-26T20:08:26.2245125Z * [new tag] trunk/981ac533c6e69a77538aaa7a9747c3d840dfa8be -> trunk/981ac533c6e69a77538aaa7a9747c3d840dfa8be 2025-08-26T20:08:26.2245352Z * [new tag] trunk/995397d47a0e27394ee1010f158e181eb304100a -> trunk/995397d47a0e27394ee1010f158e181eb304100a 2025-08-26T20:08:26.2245592Z * [new tag] trunk/9a41570199155eee92ebd28452a556075e34e1b4 -> trunk/9a41570199155eee92ebd28452a556075e34e1b4 2025-08-26T20:08:26.2245832Z * [new tag] trunk/9b3ebd25acfd2ff4e9b7428079ba364d6f8a14da -> trunk/9b3ebd25acfd2ff4e9b7428079ba364d6f8a14da 2025-08-26T20:08:26.2246083Z * [new tag] trunk/9b4adc4db7494dbc4dbbac5dd85ccbf5babaef44 -> trunk/9b4adc4db7494dbc4dbbac5dd85ccbf5babaef44 2025-08-26T20:08:26.2246331Z * [new tag] trunk/9d18bf01b1661d227f6af41ac07a1e9ef20a9e1a -> trunk/9d18bf01b1661d227f6af41ac07a1e9ef20a9e1a 2025-08-26T20:08:26.2246604Z * [new tag] trunk/9d7cecdd6c44c5421d341bcc359be4097ea9a2f5 -> trunk/9d7cecdd6c44c5421d341bcc359be4097ea9a2f5 2025-08-26T20:08:26.2246855Z * [new tag] trunk/9d882fd9ffc6ad2a292fee548740aabfea745002 -> trunk/9d882fd9ffc6ad2a292fee548740aabfea745002 2025-08-26T20:08:26.2247078Z * [new tag] trunk/9d9cc9897ac44a1a8df38211b03d8342a8af48c3 -> trunk/9d9cc9897ac44a1a8df38211b03d8342a8af48c3 2025-08-26T20:08:26.2247312Z * [new tag] trunk/9e1c9541344b2aa1c946edb779d275072f3b8f4a -> trunk/9e1c9541344b2aa1c946edb779d275072f3b8f4a 2025-08-26T20:08:26.2247536Z * [new tag] trunk/9e491f753ee521a70e6a7e7dbb36f96c9350f5ea -> trunk/9e491f753ee521a70e6a7e7dbb36f96c9350f5ea 2025-08-26T20:08:26.2247764Z * [new tag] trunk/9f6e1b8730d6a7a7d012be90ae08674294aa4933 -> trunk/9f6e1b8730d6a7a7d012be90ae08674294aa4933 2025-08-26T20:08:26.2248001Z * [new tag] trunk/a03cc53e6f6e2fe67316cb8c74c25f5b953f445b -> trunk/a03cc53e6f6e2fe67316cb8c74c25f5b953f445b 2025-08-26T20:08:26.2248228Z * [new tag] trunk/a154c2093c0f2646346f032e1f30012779b3c51d -> trunk/a154c2093c0f2646346f032e1f30012779b3c51d 2025-08-26T20:08:26.2248478Z * [new tag] trunk/a391fa1c42dd32e32a2e5b1cb196bac56daaca88 -> trunk/a391fa1c42dd32e32a2e5b1cb196bac56daaca88 2025-08-26T20:08:26.2248727Z * [new tag] trunk/a3a82e3da85a53afc4bbf3d75bd3d3dcc2e06645 -> trunk/a3a82e3da85a53afc4bbf3d75bd3d3dcc2e06645 2025-08-26T20:08:26.2248969Z * [new tag] trunk/a3fe1ced409d186628ff2975f05ba529a86fae84 -> trunk/a3fe1ced409d186628ff2975f05ba529a86fae84 2025-08-26T20:08:26.2249199Z * [new tag] trunk/a43480d19cdd68e544163b1a07c328a9c54723b8 -> trunk/a43480d19cdd68e544163b1a07c328a9c54723b8 2025-08-26T20:08:26.2249441Z * [new tag] trunk/a445b41e4f11daa82a53a21ec413c15d5079ae77 -> trunk/a445b41e4f11daa82a53a21ec413c15d5079ae77 2025-08-26T20:08:26.2249688Z * [new tag] trunk/a44a0d3671b4ccf2fe915896a8a5204fe79b1e7b -> trunk/a44a0d3671b4ccf2fe915896a8a5204fe79b1e7b 2025-08-26T20:08:26.2249932Z * [new tag] trunk/a6401cb5aa51622045c3f9a03b2cebef236e4182 -> trunk/a6401cb5aa51622045c3f9a03b2cebef236e4182 2025-08-26T20:08:26.2250160Z * [new tag] trunk/a68f63e33161b4665e0f4c399bf8072135a35a57 -> trunk/a68f63e33161b4665e0f4c399bf8072135a35a57 2025-08-26T20:08:26.2250430Z * [new tag] trunk/a72803f1e3c69c780b7d7bcdd9b35360fd98148b -> trunk/a72803f1e3c69c780b7d7bcdd9b35360fd98148b 2025-08-26T20:08:26.2250678Z * [new tag] trunk/a7b5955ea8851d73e35f50a0de5bb0626bae24cb -> trunk/a7b5955ea8851d73e35f50a0de5bb0626bae24cb 2025-08-26T20:08:26.2250909Z * [new tag] trunk/a818fa77e3a72271f144514ef349c5a666313205 -> trunk/a818fa77e3a72271f144514ef349c5a666313205 2025-08-26T20:08:26.2251489Z * [new tag] trunk/a825557ed53507e85ac613862311a81eb88710a4 -> trunk/a825557ed53507e85ac613862311a81eb88710a4 2025-08-26T20:08:26.2251734Z * [new tag] trunk/a85711d565f37b0095af9f7dafa77f392c9aa31e -> trunk/a85711d565f37b0095af9f7dafa77f392c9aa31e 2025-08-26T20:08:26.2251984Z * [new tag] trunk/a941d7ffe54b5f256c1fbd3959ddbf608b7eea88 -> trunk/a941d7ffe54b5f256c1fbd3959ddbf608b7eea88 2025-08-26T20:08:26.2252223Z * [new tag] trunk/a9fabeb012a4b804836a2b8d4b3742b92c9a6b58 -> trunk/a9fabeb012a4b804836a2b8d4b3742b92c9a6b58 2025-08-26T20:08:26.2252481Z * [new tag] trunk/ab7787fb82dd777b2f777ef58bc20dbb7bd8289b -> trunk/ab7787fb82dd777b2f777ef58bc20dbb7bd8289b 2025-08-26T20:08:26.2257473Z * [new tag] trunk/ab8d60f4c86ca19ed00d6e79ae8e6939266f28e6 -> trunk/ab8d60f4c86ca19ed00d6e79ae8e6939266f28e6 2025-08-26T20:08:26.2257758Z * [new tag] trunk/ac8d9418aee4543fa193c86ae0bc3e63707bcd3b -> trunk/ac8d9418aee4543fa193c86ae0bc3e63707bcd3b 2025-08-26T20:08:26.2258200Z * [new tag] trunk/acb00d3ccf5f2d566225f07ed66bd579d5d3e44e -> trunk/acb00d3ccf5f2d566225f07ed66bd579d5d3e44e 2025-08-26T20:08:26.2258452Z * [new tag] trunk/adecb0c9e89e0dfe18d944d292c98c97b686fc83 -> trunk/adecb0c9e89e0dfe18d944d292c98c97b686fc83 2025-08-26T20:08:26.2258688Z * [new tag] trunk/ae8d319fd4a0b0fa7b1372aa07690a36ce823abc -> trunk/ae8d319fd4a0b0fa7b1372aa07690a36ce823abc 2025-08-26T20:08:26.2258931Z * [new tag] trunk/af3265d20f763e5366bfa37e3d4a6307036d0c18 -> trunk/af3265d20f763e5366bfa37e3d4a6307036d0c18 2025-08-26T20:08:26.2259161Z * [new tag] trunk/b0420d24386263f2727fd5714b63cfa6bc89f3e6 -> trunk/b0420d24386263f2727fd5714b63cfa6bc89f3e6 2025-08-26T20:08:26.2259419Z * [new tag] trunk/b1380f434da2fa2de0e5ff6fd70f73082dc08687 -> trunk/b1380f434da2fa2de0e5ff6fd70f73082dc08687 2025-08-26T20:08:26.2259655Z * [new tag] trunk/b2632e79828300302fd11e093d765196c3c0db58 -> trunk/b2632e79828300302fd11e093d765196c3c0db58 2025-08-26T20:08:26.2259902Z * [new tag] trunk/b2e06e0194c3fa8f7578a1b48751cc027394fb67 -> trunk/b2e06e0194c3fa8f7578a1b48751cc027394fb67 2025-08-26T20:08:26.2260947Z * [new tag] trunk/b3e215b864e6ca43b2c4e50ce666673f80feee27 -> trunk/b3e215b864e6ca43b2c4e50ce666673f80feee27 2025-08-26T20:08:26.2261194Z * [new tag] trunk/b708966201811b31ee765ec57715ac21d06ef652 -> trunk/b708966201811b31ee765ec57715ac21d06ef652 2025-08-26T20:08:26.2261458Z * [new tag] trunk/b9e9e92817fd7d1a778f074105603efb07e05004 -> trunk/b9e9e92817fd7d1a778f074105603efb07e05004 2025-08-26T20:08:26.2261719Z * [new tag] trunk/bc7eaa0d8a1f5ca8ec0eaac461d1df500dcaea84 -> trunk/bc7eaa0d8a1f5ca8ec0eaac461d1df500dcaea84 2025-08-26T20:08:26.2261957Z * [new tag] trunk/bcfe1b2d714cbb2716495e09ae010e7c34daf045 -> trunk/bcfe1b2d714cbb2716495e09ae010e7c34daf045 2025-08-26T20:08:26.2262185Z * [new tag] trunk/bd5857a1d6d5455d4f0057c182dff5e8ad2a4c8a -> trunk/bd5857a1d6d5455d4f0057c182dff5e8ad2a4c8a 2025-08-26T20:08:26.2262427Z * [new tag] trunk/be2e6b3158552405acc13ef7829a0217826fb271 -> trunk/be2e6b3158552405acc13ef7829a0217826fb271 2025-08-26T20:08:26.2262676Z * [new tag] trunk/be87f22dfba4488963fcc854699829e2782ee0f2 -> trunk/be87f22dfba4488963fcc854699829e2782ee0f2 2025-08-26T20:08:26.2263018Z * [new tag] trunk/becd6cd744bdf950578519437652a0d1f4b48781 -> trunk/becd6cd744bdf950578519437652a0d1f4b48781 2025-08-26T20:08:26.2263262Z * [new tag] trunk/bf8431ba062efa9ff0cdd5032a3ddf2e007a3216 -> trunk/bf8431ba062efa9ff0cdd5032a3ddf2e007a3216 2025-08-26T20:08:26.2263501Z * [new tag] trunk/c02e26bf31eb3da301158a061aa68527dbfb4d32 -> trunk/c02e26bf31eb3da301158a061aa68527dbfb4d32 2025-08-26T20:08:26.2263763Z * [new tag] trunk/c081481bbebdb568d07ee19cfe2cd3125de6cba7 -> trunk/c081481bbebdb568d07ee19cfe2cd3125de6cba7 2025-08-26T20:08:26.2268759Z * [new tag] trunk/c2390087c34c964ef648addf43efb8c6a34e30c2 -> trunk/c2390087c34c964ef648addf43efb8c6a34e30c2 2025-08-26T20:08:26.2269056Z * [new tag] trunk/c4670e40c9b741d50a79b714e3830149833be908 -> trunk/c4670e40c9b741d50a79b714e3830149833be908 2025-08-26T20:08:26.2269312Z * [new tag] trunk/c5cb255625deb4cdbc5780e6911b73498e17ed5a -> trunk/c5cb255625deb4cdbc5780e6911b73498e17ed5a 2025-08-26T20:08:26.2269573Z * [new tag] trunk/c60dea5261d9648d1da51528a07731966bb6823e -> trunk/c60dea5261d9648d1da51528a07731966bb6823e 2025-08-26T20:08:26.2269823Z * [new tag] trunk/c74e5f60611b7eac4321f53a9e4a15b077fb1bcc -> trunk/c74e5f60611b7eac4321f53a9e4a15b077fb1bcc 2025-08-26T20:08:26.2270074Z * [new tag] trunk/c7a77470c54b28e555319e34048af14d1d66198a -> trunk/c7a77470c54b28e555319e34048af14d1d66198a 2025-08-26T20:08:26.2270324Z * [new tag] trunk/c7fb031706330684fc3a2d8d169bebea874d4e95 -> trunk/c7fb031706330684fc3a2d8d169bebea874d4e95 2025-08-26T20:08:26.2270766Z * [new tag] trunk/c8bb0e4720ddddf3cd1b0b48b336978f763c71ca -> trunk/c8bb0e4720ddddf3cd1b0b48b336978f763c71ca 2025-08-26T20:08:26.2271038Z * [new tag] trunk/ca9fe0107e165a4a4147325ff6d34235ebde447f -> trunk/ca9fe0107e165a4a4147325ff6d34235ebde447f 2025-08-26T20:08:26.2271301Z * [new tag] trunk/caf98fde0d5c47452af45dc77099449edd521579 -> trunk/caf98fde0d5c47452af45dc77099449edd521579 2025-08-26T20:08:26.2271558Z * [new tag] trunk/cb579532150c9e87e7c143adcb020fb7de7cc6b1 -> trunk/cb579532150c9e87e7c143adcb020fb7de7cc6b1 2025-08-26T20:08:26.2271811Z * [new tag] trunk/cc2b65a91ae7773d4ecf9a600dda48fc3e69aa8f -> trunk/cc2b65a91ae7773d4ecf9a600dda48fc3e69aa8f 2025-08-26T20:08:26.2272063Z * [new tag] trunk/cc791d5857f4aa06b8d4e567b1fb2852e3ae963d -> trunk/cc791d5857f4aa06b8d4e567b1fb2852e3ae963d 2025-08-26T20:08:26.2272321Z * [new tag] trunk/cd31be28ec5cd0c4d9cdb6742efe151eee1406ec -> trunk/cd31be28ec5cd0c4d9cdb6742efe151eee1406ec 2025-08-26T20:08:26.2272572Z * [new tag] trunk/cd87f3029582cedb3b88747a3bd7d200b05c1138 -> trunk/cd87f3029582cedb3b88747a3bd7d200b05c1138 2025-08-26T20:08:26.2272803Z * [new tag] trunk/ce048de608180fa88335e5821070472539968b54 -> trunk/ce048de608180fa88335e5821070472539968b54 2025-08-26T20:08:26.2273065Z * [new tag] trunk/ce467df5d1d763d1648aee51c93ce3e9a4699936 -> trunk/ce467df5d1d763d1648aee51c93ce3e9a4699936 2025-08-26T20:08:26.2275492Z * [new tag] trunk/cee72119b2dec7776bc2550dd39a9b1349772751 -> trunk/cee72119b2dec7776bc2550dd39a9b1349772751 2025-08-26T20:08:26.2275757Z * [new tag] trunk/cf94cadbeee31a4d1d46a57f11bce7c9fd1cebc0 -> trunk/cf94cadbeee31a4d1d46a57f11bce7c9fd1cebc0 2025-08-26T20:08:26.2276017Z * [new tag] trunk/cfdaaaaa26d7f34427ba941569eca46f02f79f3e -> trunk/cfdaaaaa26d7f34427ba941569eca46f02f79f3e 2025-08-26T20:08:26.2276269Z * [new tag] trunk/d1faf2ef0476eb60b42c057baee9af0f48ae849a -> trunk/d1faf2ef0476eb60b42c057baee9af0f48ae849a 2025-08-26T20:08:26.2276513Z * [new tag] trunk/d228a776e90368bb693837ae23285ad8fc33def5 -> trunk/d228a776e90368bb693837ae23285ad8fc33def5 2025-08-26T20:08:26.2276751Z * [new tag] trunk/d2b8c0d431e00ad57354c5247e46c1bea0b8cd31 -> trunk/d2b8c0d431e00ad57354c5247e46c1bea0b8cd31 2025-08-26T20:08:26.2277051Z * [new tag] trunk/d2bd55d8de784df439b38378f161271dc43b744c -> trunk/d2bd55d8de784df439b38378f161271dc43b744c 2025-08-26T20:08:26.2277290Z * [new tag] trunk/d4703fb91c3510460d71f648da113177edf593c8 -> trunk/d4703fb91c3510460d71f648da113177edf593c8 2025-08-26T20:08:26.2277529Z * [new tag] trunk/d875d3ca1e5099636c766c9df70ac5888c25215a -> trunk/d875d3ca1e5099636c766c9df70ac5888c25215a 2025-08-26T20:08:26.2277777Z * [new tag] trunk/d8fcb2a4acb506f9c72a1f44fc8b857158bda892 -> trunk/d8fcb2a4acb506f9c72a1f44fc8b857158bda892 2025-08-26T20:08:26.2278348Z * [new tag] trunk/daeb3a6094c62d1881ea68091fcadb02d1dc687e -> trunk/daeb3a6094c62d1881ea68091fcadb02d1dc687e 2025-08-26T20:08:26.2278641Z * [new tag] trunk/db38c44ad639e7ada3e9df2ba026a2cb5e40feb0 -> trunk/db38c44ad639e7ada3e9df2ba026a2cb5e40feb0 2025-08-26T20:08:26.2280471Z * [new tag] trunk/db44de4c0d3e9f1fe5334ff4cc261fb8fe4390c8 -> trunk/db44de4c0d3e9f1fe5334ff4cc261fb8fe4390c8 2025-08-26T20:08:26.2280761Z * [new tag] trunk/dbef6066311a1ce6e60e1f2b6084249d1ad45769 -> trunk/dbef6066311a1ce6e60e1f2b6084249d1ad45769 2025-08-26T20:08:26.2281048Z * [new tag] trunk/df571ae7ad7dacf77ce42c00189cf369d7993387 -> trunk/df571ae7ad7dacf77ce42c00189cf369d7993387 2025-08-26T20:08:26.2281530Z * [new tag] trunk/df6073641079c781e66a905e4f15ee49ac257eb2 -> trunk/df6073641079c781e66a905e4f15ee49ac257eb2 2025-08-26T20:08:26.2285277Z * [new tag] trunk/e1a64b75ff3dc834774a9174c2e7b1c46dea35ec -> trunk/e1a64b75ff3dc834774a9174c2e7b1c46dea35ec 2025-08-26T20:08:26.2285736Z * [new tag] trunk/e20f6d798606f3245686e950c43635bbe526232d -> trunk/e20f6d798606f3245686e950c43635bbe526232d 2025-08-26T20:08:26.2286127Z * [new tag] trunk/e25ee0290ef16503f178e04890c15717f6e9ea44 -> trunk/e25ee0290ef16503f178e04890c15717f6e9ea44 2025-08-26T20:08:26.2286548Z * [new tag] trunk/e34b6a01039df5d8940acdccd8d8989f3cd827aa -> trunk/e34b6a01039df5d8940acdccd8d8989f3cd827aa 2025-08-26T20:08:26.2286821Z * [new tag] trunk/e3d68dfae2dee15e74d3b95beaed7149b6afb94a -> trunk/e3d68dfae2dee15e74d3b95beaed7149b6afb94a 2025-08-26T20:08:26.2287132Z * [new tag] trunk/e3ebf364e6d2fb8008da113a596d3cc426ba9c79 -> trunk/e3ebf364e6d2fb8008da113a596d3cc426ba9c79 2025-08-26T20:08:26.2287366Z * [new tag] trunk/e4839470470168648dee5997f57347bb8541ea2b -> trunk/e4839470470168648dee5997f57347bb8541ea2b 2025-08-26T20:08:26.2287619Z * [new tag] trunk/e63155751825ba026ced3a1fc89563231bc85ccc -> trunk/e63155751825ba026ced3a1fc89563231bc85ccc 2025-08-26T20:08:26.2287877Z * [new tag] trunk/e6aa7287f8c8cac76d792097f20ba1dae6dc8717 -> trunk/e6aa7287f8c8cac76d792097f20ba1dae6dc8717 2025-08-26T20:08:26.2288138Z * [new tag] trunk/e6e45e6ae8452f0bc5e3e258027c42eb9a1394fb -> trunk/e6e45e6ae8452f0bc5e3e258027c42eb9a1394fb 2025-08-26T20:08:26.2288388Z * [new tag] trunk/e795450a35bca909902e12de99245e1c0e7e2872 -> trunk/e795450a35bca909902e12de99245e1c0e7e2872 2025-08-26T20:08:26.2288940Z * [new tag] trunk/e7e270a33a3f368c3ef0c3339950a47fdbfadd71 -> trunk/e7e270a33a3f368c3ef0c3339950a47fdbfadd71 2025-08-26T20:08:26.2289356Z * [new tag] trunk/e836323a23f5750e800abe04ef8ca386b3066b58 -> trunk/e836323a23f5750e800abe04ef8ca386b3066b58 2025-08-26T20:08:26.2292133Z * [new tag] trunk/e83825f91cb2901567fedbf31ba7cc434a897271 -> trunk/e83825f91cb2901567fedbf31ba7cc434a897271 2025-08-26T20:08:26.2292587Z * [new tag] trunk/e9d42b3880dcdbd823bbdc9370c8b0b3af0ba2e3 -> trunk/e9d42b3880dcdbd823bbdc9370c8b0b3af0ba2e3 2025-08-26T20:08:26.2292984Z * [new tag] trunk/eb5549a43164cdf8689cd7d177c03b2508c699f4 -> trunk/eb5549a43164cdf8689cd7d177c03b2508c699f4 2025-08-26T20:08:26.2293851Z * [new tag] trunk/eba1ad09e47b66478f973e03cece7f314ac3b412 -> trunk/eba1ad09e47b66478f973e03cece7f314ac3b412 2025-08-26T20:08:26.2294140Z * [new tag] trunk/eba20d2d748cb17dce9aa26e5513e4567bfd8282 -> trunk/eba20d2d748cb17dce9aa26e5513e4567bfd8282 2025-08-26T20:08:26.2294564Z * [new tag] trunk/ec21cafd85d491d2d220e4e54080fe340a37c4c2 -> trunk/ec21cafd85d491d2d220e4e54080fe340a37c4c2 2025-08-26T20:08:26.2294962Z * [new tag] trunk/ed8bcccf31e1ba01a35e818a4afbb74c333e8dc3 -> trunk/ed8bcccf31e1ba01a35e818a4afbb74c333e8dc3 2025-08-26T20:08:26.2295461Z * [new tag] trunk/eddaaa6c2a66a84e17b17bf8af5131852067b259 -> trunk/eddaaa6c2a66a84e17b17bf8af5131852067b259 2025-08-26T20:08:26.2295942Z * [new tag] trunk/ef761c43538abae5bccc0c4b6ebaf42ff676db7a -> trunk/ef761c43538abae5bccc0c4b6ebaf42ff676db7a 2025-08-26T20:08:26.2297337Z * [new tag] trunk/f085f299584b06a2a7d8855eda2a411313e782ad -> trunk/f085f299584b06a2a7d8855eda2a411313e782ad 2025-08-26T20:08:26.2304014Z * [new tag] trunk/f09458c2e16b4fe7063d73d80fd3e7e354bad3f8 -> trunk/f09458c2e16b4fe7063d73d80fd3e7e354bad3f8 2025-08-26T20:08:26.2304451Z * [new tag] trunk/f0e0a6897ee5cb31ccee10ee8e2d3c01140ff999 -> trunk/f0e0a6897ee5cb31ccee10ee8e2d3c01140ff999 2025-08-26T20:08:26.2304806Z * [new tag] trunk/f30501937738a2440f90988d1d46920529309ba8 -> trunk/f30501937738a2440f90988d1d46920529309ba8 2025-08-26T20:08:26.2305395Z * [new tag] trunk/f391afe9bf8c542fdbb822423d2a1e454b3d9744 -> trunk/f391afe9bf8c542fdbb822423d2a1e454b3d9744 2025-08-26T20:08:26.2305641Z * [new tag] trunk/f521e82a4e80df502fa57e5852af14d8779dcbd1 -> trunk/f521e82a4e80df502fa57e5852af14d8779dcbd1 2025-08-26T20:08:26.2305879Z * [new tag] trunk/f5bf5147ad18994c9a6e0f565d7831362bf5a18a -> trunk/f5bf5147ad18994c9a6e0f565d7831362bf5a18a 2025-08-26T20:08:26.2306096Z * [new tag] trunk/f795e92802c55608ad4f4f198726d250056d0232 -> trunk/f795e92802c55608ad4f4f198726d250056d0232 2025-08-26T20:08:26.2306362Z * [new tag] trunk/f8bd85827d465a8a2a610c27ed9e62a4c27ac07d -> trunk/f8bd85827d465a8a2a610c27ed9e62a4c27ac07d 2025-08-26T20:08:26.2310849Z * [new tag] trunk/f90ccad1651b5a1698b2232acc3e92e2829b7935 -> trunk/f90ccad1651b5a1698b2232acc3e92e2829b7935 2025-08-26T20:08:26.2311335Z * [new tag] trunk/f912c93344caa74e24c8164a2e25fe84a8203073 -> trunk/f912c93344caa74e24c8164a2e25fe84a8203073 2025-08-26T20:08:26.2311743Z * [new tag] trunk/f9875166a953a51bbd454d963ee03d41818a27e8 -> trunk/f9875166a953a51bbd454d963ee03d41818a27e8 2025-08-26T20:08:26.2312159Z * [new tag] trunk/f9df4ec2af0ac19b42f658ae87acf12067e67b36 -> trunk/f9df4ec2af0ac19b42f658ae87acf12067e67b36 2025-08-26T20:08:26.2312435Z * [new tag] trunk/fab5dac734344105ae107e85c08151758a4a9b4d -> trunk/fab5dac734344105ae107e85c08151758a4a9b4d 2025-08-26T20:08:26.2312673Z * [new tag] trunk/fb241d0a448f1dd88471098ac149418124a7c4aa -> trunk/fb241d0a448f1dd88471098ac149418124a7c4aa 2025-08-26T20:08:26.2312923Z * [new tag] trunk/fc0683b1e75fdf3182e0855b3f79e80fe0124ef1 -> trunk/fc0683b1e75fdf3182e0855b3f79e80fe0124ef1 2025-08-26T20:08:26.2313150Z * [new tag] trunk/fc69c2bc67672c3b2d0c62c1821895f09288f1c0 -> trunk/fc69c2bc67672c3b2d0c62c1821895f09288f1c0 2025-08-26T20:08:26.2313412Z * [new tag] trunk/febfc3ec03004116dfd6d504e6853ff02a1dd6e0 -> trunk/febfc3ec03004116dfd6d504e6853ff02a1dd6e0 2025-08-26T20:08:26.2314123Z * [new tag] trunk/fecc5f600110209aaaedead11770a445b3c879e6 -> trunk/fecc5f600110209aaaedead11770a445b3c879e6 2025-08-26T20:08:26.2314746Z * [new tag] trunk/ff4f5dd8ed8e2aaee903c7d30cd4f8bd04d883c8 -> trunk/ff4f5dd8ed8e2aaee903c7d30cd4f8bd04d883c8 2025-08-26T20:08:26.2315200Z * [new tag] trunk/ffa1ce7650766c2ae6eaa96415dfc29e9eb0b3ec -> trunk/ffa1ce7650766c2ae6eaa96415dfc29e9eb0b3ec 2025-08-26T20:08:26.2315839Z * [new tag] v0.1.1 -> v0.1.1 2025-08-26T20:08:26.2317092Z * [new tag] v0.1.10 -> v0.1.10 2025-08-26T20:08:26.2317681Z * [new tag] v0.1.11 -> v0.1.11 2025-08-26T20:08:26.2317847Z * [new tag] v0.1.12 -> v0.1.12 2025-08-26T20:08:26.2318577Z * [new tag] v0.1.2 -> v0.1.2 2025-08-26T20:08:26.2318991Z * [new tag] v0.1.3 -> v0.1.3 2025-08-26T20:08:26.2319843Z * [new tag] v0.1.4 -> v0.1.4 2025-08-26T20:08:26.2322449Z * [new tag] v0.1.5 -> v0.1.5 2025-08-26T20:08:26.2322591Z * [new tag] v0.1.6 -> v0.1.6 2025-08-26T20:08:26.2322692Z * [new tag] v0.1.7 -> v0.1.7 2025-08-26T20:08:26.2322813Z * [new tag] v0.1.8 -> v0.1.8 2025-08-26T20:08:26.2322908Z * [new tag] v0.1.9 -> v0.1.9 2025-08-26T20:08:26.2323321Z * [new tag] v0.2.0 -> v0.2.0 2025-08-26T20:08:26.2329060Z * [new tag] v0.3.0 -> v0.3.0 2025-08-26T20:08:26.2334069Z * [new tag] v0.3.1 -> v0.3.1 2025-08-26T20:08:26.2339631Z * [new tag] v0.4.0 -> v0.4.0 2025-08-26T20:08:26.2339977Z * [new tag] v0.4.1 -> v0.4.1 2025-08-26T20:08:26.2340096Z * [new tag] v1.0.0 -> v1.0.0 2025-08-26T20:08:26.2340209Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-08-26T20:08:26.2340331Z * [new tag] v1.0.1 -> v1.0.1 2025-08-26T20:08:26.2340446Z * [new tag] v1.0rc0 -> v1.0rc0 2025-08-26T20:08:26.2340552Z * [new tag] v1.0rc1 -> v1.0rc1 2025-08-26T20:08:26.2340665Z * [new tag] v1.1.0 -> v1.1.0 2025-08-26T20:08:26.2340774Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-08-26T20:08:26.2340878Z * [new tag] v1.10.0 -> v1.10.0 2025-08-26T20:08:26.2341001Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-08-26T20:08:26.2341109Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-08-26T20:08:26.2341223Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-08-26T20:08:26.2341321Z * [new tag] v1.10.1 -> v1.10.1 2025-08-26T20:08:26.2341430Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-08-26T20:08:26.2341535Z * [new tag] v1.10.2 -> v1.10.2 2025-08-26T20:08:26.2341642Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-08-26T20:08:26.2341738Z * [new tag] v1.11.0 -> v1.11.0 2025-08-26T20:08:26.2341836Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-08-26T20:08:26.2341940Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-08-26T20:08:26.2342035Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-08-26T20:08:26.2342138Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-08-26T20:08:26.2342236Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-08-26T20:08:26.2342337Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-08-26T20:08:26.2342432Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-08-26T20:08:26.2342524Z * [new tag] v1.12.0 -> v1.12.0 2025-08-26T20:08:26.2342675Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-08-26T20:08:26.2343132Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-08-26T20:08:26.2343692Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-08-26T20:08:26.2343952Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-08-26T20:08:26.2344242Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-08-26T20:08:26.2344372Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-08-26T20:08:26.2344491Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-08-26T20:08:26.2344592Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-08-26T20:08:26.2344692Z * [new tag] v1.12.1 -> v1.12.1 2025-08-26T20:08:26.2344789Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-08-26T20:08:26.2347622Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-08-26T20:08:26.2347792Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-08-26T20:08:26.2347908Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-08-26T20:08:26.2348007Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-08-26T20:08:26.2348118Z * [new tag] v1.13.0 -> v1.13.0 2025-08-26T20:08:26.2348213Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-08-26T20:08:26.2348482Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-08-26T20:08:26.2351061Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-08-26T20:08:26.2351216Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-08-26T20:08:26.2351323Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-08-26T20:08:26.2351425Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-08-26T20:08:26.2351572Z * [new tag] v1.13.1 -> v1.13.1 2025-08-26T20:08:26.2351671Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-08-26T20:08:26.2351782Z * [new tag] v1.2.0 -> v1.2.0 2025-08-26T20:08:26.2351883Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-08-26T20:08:26.2351981Z * [new tag] v1.3.0 -> v1.3.0 2025-08-26T20:08:26.2352090Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-08-26T20:08:26.2352704Z * [new tag] v1.3.1 -> v1.3.1 2025-08-26T20:08:26.2353107Z * [new tag] v1.4.0 -> v1.4.0 2025-08-26T20:08:26.2354031Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-08-26T20:08:26.2354237Z * [new tag] v1.4.1 -> v1.4.1 2025-08-26T20:08:26.2354781Z * [new tag] v1.5.0 -> v1.5.0 2025-08-26T20:08:26.2355627Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-08-26T20:08:26.2355894Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-08-26T20:08:26.2356868Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-08-26T20:08:26.2357104Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-08-26T20:08:26.2357425Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-08-26T20:08:26.2358400Z * [new tag] v1.5.1 -> v1.5.1 2025-08-26T20:08:26.2358661Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-08-26T20:08:26.2358784Z * [new tag] v1.6.0 -> v1.6.0 2025-08-26T20:08:26.2360646Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-08-26T20:08:26.2360951Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-08-26T20:08:26.2361056Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-08-26T20:08:26.2361850Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-08-26T20:08:26.2362088Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-08-26T20:08:26.2363016Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-08-26T20:08:26.2363295Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-08-26T20:08:26.2363737Z * [new tag] v1.7.0 -> v1.7.0 2025-08-26T20:08:26.2364563Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-08-26T20:08:26.2364784Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-08-26T20:08:26.2367477Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-08-26T20:08:26.2367629Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-08-26T20:08:26.2367787Z * [new tag] v1.7.1 -> v1.7.1 2025-08-26T20:08:26.2367892Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-08-26T20:08:26.2368001Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-08-26T20:08:26.2368545Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-08-26T20:08:26.2368738Z * [new tag] v1.8.0 -> v1.8.0 2025-08-26T20:08:26.2369530Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-08-26T20:08:26.2370070Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-08-26T20:08:26.2370316Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-08-26T20:08:26.2372883Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-08-26T20:08:26.2373153Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-08-26T20:08:26.2373323Z * [new tag] v1.8.1 -> v1.8.1 2025-08-26T20:08:26.2373445Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-08-26T20:08:26.2373674Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-08-26T20:08:26.2373797Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-08-26T20:08:26.2375248Z * [new tag] v1.8.2 -> v1.8.2 2025-08-26T20:08:26.2375487Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-08-26T20:08:26.2375668Z * [new tag] v1.9.0 -> v1.9.0 2025-08-26T20:08:26.2378045Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-08-26T20:08:26.2378351Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-08-26T20:08:26.2378491Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-08-26T20:08:26.2378614Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-08-26T20:08:26.2378860Z * [new tag] v1.9.1 -> v1.9.1 2025-08-26T20:08:26.2380004Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-08-26T20:08:26.2380288Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-08-26T20:08:26.2380435Z * [new tag] v2.0.0 -> v2.0.0 2025-08-26T20:08:26.2383748Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-08-26T20:08:26.2384069Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-08-26T20:08:26.2384573Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-08-26T20:08:26.2384704Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-08-26T20:08:26.2384807Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-08-26T20:08:26.2385050Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-08-26T20:08:26.2385358Z * [new tag] v2.0.1 -> v2.0.1 2025-08-26T20:08:26.2385621Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-08-26T20:08:26.2385736Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-08-26T20:08:26.2386196Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-08-26T20:08:26.2386573Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-08-26T20:08:26.2390917Z * [new tag] v2.1.0 -> v2.1.0 2025-08-26T20:08:26.2391078Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-08-26T20:08:26.2391195Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-08-26T20:08:26.2391297Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-08-26T20:08:26.2391399Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-08-26T20:08:26.2391692Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-08-26T20:08:26.2391815Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-08-26T20:08:26.2392163Z * [new tag] v2.1.1 -> v2.1.1 2025-08-26T20:08:26.2393111Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-08-26T20:08:26.2393732Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-08-26T20:08:26.2394197Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-08-26T20:08:26.2395080Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-08-26T20:08:26.2395200Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-08-26T20:08:26.2395762Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-08-26T20:08:26.2398830Z * [new tag] v2.1.2 -> v2.1.2 2025-08-26T20:08:26.2399040Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-08-26T20:08:26.2399374Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-08-26T20:08:26.2399537Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-08-26T20:08:26.2399653Z * [new tag] v2.2.0 -> v2.2.0 2025-08-26T20:08:26.2399756Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-08-26T20:08:26.2400194Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-08-26T20:08:26.2400412Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-08-26T20:08:26.2407600Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-08-26T20:08:26.2407880Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-08-26T20:08:26.2408008Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-08-26T20:08:26.2408105Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-08-26T20:08:26.2408350Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-08-26T20:08:26.2408473Z * [new tag] v2.2.1 -> v2.2.1 2025-08-26T20:08:26.2408571Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-08-26T20:08:26.2408795Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-08-26T20:08:26.2408914Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-08-26T20:08:26.2409015Z * [new tag] v2.2.2 -> v2.2.2 2025-08-26T20:08:26.2409115Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-08-26T20:08:26.2409227Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-08-26T20:08:26.2409683Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-08-26T20:08:26.2409817Z * [new tag] v2.3.0 -> v2.3.0 2025-08-26T20:08:26.2410139Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-08-26T20:08:26.2410259Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-08-26T20:08:26.2410359Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-08-26T20:08:26.2415185Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-08-26T20:08:26.2415348Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-08-26T20:08:26.2415455Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-08-26T20:08:26.2415571Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-08-26T20:08:26.2415683Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-08-26T20:08:26.2415782Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-08-26T20:08:26.2415889Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-08-26T20:08:26.2416014Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-08-26T20:08:26.2416121Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-08-26T20:08:26.2416229Z * [new tag] v2.3.1 -> v2.3.1 2025-08-26T20:08:26.2416331Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-08-26T20:08:26.2416436Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-08-26T20:08:26.2416544Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-08-26T20:08:26.2416842Z * [new tag] v2.4.0 -> v2.4.0 2025-08-26T20:08:26.2416953Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-08-26T20:08:26.2417055Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-08-26T20:08:26.2417165Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-08-26T20:08:26.2417274Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-08-26T20:08:26.2417915Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-08-26T20:08:26.2418329Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-08-26T20:08:26.2419693Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-08-26T20:08:26.2420230Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-08-26T20:08:26.2420364Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-08-26T20:08:26.2420671Z * [new tag] v2.4.1 -> v2.4.1 2025-08-26T20:08:26.2421610Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-08-26T20:08:26.2424489Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-08-26T20:08:26.2424634Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-08-26T20:08:26.2424749Z * [new tag] v2.5.0 -> v2.5.0 2025-08-26T20:08:26.2424878Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-08-26T20:08:26.2424998Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-08-26T20:08:26.2430719Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-08-26T20:08:26.2430869Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-08-26T20:08:26.2430975Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-08-26T20:08:26.2431084Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-08-26T20:08:26.2431218Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-08-26T20:08:26.2431330Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-08-26T20:08:26.2431763Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-08-26T20:08:26.2431990Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-08-26T20:08:26.2432282Z * [new tag] v2.5.1 -> v2.5.1 2025-08-26T20:08:26.2432393Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-08-26T20:08:26.2432489Z * [new tag] v2.6.0 -> v2.6.0 2025-08-26T20:08:26.2432590Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-08-26T20:08:26.2432700Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-08-26T20:08:26.2432800Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-08-26T20:08:26.2432915Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-08-26T20:08:26.2433220Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-08-26T20:08:26.2433444Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-08-26T20:08:26.2434486Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-08-26T20:08:26.2434640Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-08-26T20:08:26.2436237Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-08-26T20:08:26.2436521Z * [new tag] v2.7.0 -> v2.7.0 2025-08-26T20:08:26.2446440Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-08-26T20:08:26.2451636Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-08-26T20:08:26.2456107Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-08-26T20:08:26.2461902Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-08-26T20:08:26.2467289Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-08-26T20:08:26.2469337Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-08-26T20:08:26.2469461Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-08-26T20:08:26.2469743Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-08-26T20:08:26.2469959Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-08-26T20:08:26.2470076Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-08-26T20:08:26.2470186Z * [new tag] v2.7.1 -> v2.7.1 2025-08-26T20:08:26.2470284Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-08-26T20:08:26.2470389Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-08-26T20:08:26.2470486Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-08-26T20:08:26.2470597Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-08-26T20:08:26.2470694Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-08-26T20:08:26.2470790Z * [new tag] v2.8.0 -> v2.8.0 2025-08-26T20:08:26.2470892Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-08-26T20:08:26.2470994Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-08-26T20:08:26.2471097Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-08-26T20:08:26.2471194Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-08-26T20:08:26.2471289Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-08-26T20:08:26.2471395Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-08-26T20:08:26.2471490Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-08-26T20:08:26.2471596Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-08-26T20:08:26.2471713Z * [new tag] whc_flight_1 -> whc_flight_1 2025-08-26T20:08:26.2471819Z * [new tag] whc_flight_2 -> whc_flight_2 2025-08-26T20:08:26.2471931Z * [new tag] whc_flight_4 -> whc_flight_4 2025-08-26T20:08:26.2918545Z [command]/usr/bin/git rev-parse --verify --quiet 262640fd220236042fbf4443cc163c8838c84c3d^{object} 2025-08-26T20:08:26.2946022Z 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:08:26.2946998Z ##[endgroup] 2025-08-26T20:08:26.2947335Z ##[group]Determining the checkout info 2025-08-26T20:08:26.2948605Z ##[endgroup] 2025-08-26T20:08:26.2956879Z [command]/usr/bin/git sparse-checkout disable 2025-08-26T20:08:26.2993893Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-08-26T20:08:26.3035741Z ##[group]Checking out the ref 2025-08-26T20:08:26.3039604Z [command]/usr/bin/git checkout --progress --force 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:08:27.3301929Z Updating files: 98% (19085/19395) 2025-08-26T20:08:27.3426463Z Updating files: 99% (19202/19395) 2025-08-26T20:08:27.3426891Z Updating files: 100% (19395/19395) 2025-08-26T20:08:27.3427219Z Updating files: 100% (19395/19395), done. 2025-08-26T20:08:27.3654303Z Note: switching to '262640fd220236042fbf4443cc163c8838c84c3d'. 2025-08-26T20:08:27.3654750Z 2025-08-26T20:08:27.3655108Z You are in 'detached HEAD' state. You can look around, make experimental 2025-08-26T20:08:27.3655485Z changes and commit them, and you can discard any commits you make in this 2025-08-26T20:08:27.3655848Z state without impacting any branches by switching back to a branch. 2025-08-26T20:08:27.3656059Z 2025-08-26T20:08:27.3656224Z If you want to create a new branch to retain commits you create, you may 2025-08-26T20:08:27.3656564Z do so (now or later) by using -c with the switch command. Example: 2025-08-26T20:08:27.3656751Z 2025-08-26T20:08:27.3657260Z git switch -c 2025-08-26T20:08:27.3657408Z 2025-08-26T20:08:27.3657502Z Or undo this operation with: 2025-08-26T20:08:27.3657634Z 2025-08-26T20:08:27.3657704Z git switch - 2025-08-26T20:08:27.3657816Z 2025-08-26T20:08:27.3657983Z Turn off this advice by setting config variable advice.detachedHead to false 2025-08-26T20:08:27.3658217Z 2025-08-26T20:08:27.3658376Z HEAD is now at 262640fd220 [ROCm][CI] restore test_flex_attention tests (#161519) 2025-08-26T20:08:27.4092278Z ##[endgroup] 2025-08-26T20:08:27.4092730Z ##[group]Setting up auth for fetching submodules 2025-08-26T20:08:27.4102290Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-26T20:08:27.4160515Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-08-26T20:08:27.4194426Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-08-26T20:08:27.4220832Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-08-26T20:08:27.4263833Z ##[endgroup] 2025-08-26T20:08:27.4268850Z ##[group]Fetching submodules 2025-08-26T20:08:27.4274245Z [command]/usr/bin/git submodule sync --recursive 2025-08-26T20:08:27.4582868Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-08-26T20:08:27.4899277Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-08-26T20:08:27.4899974Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-08-26T20:08:27.4920799Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-08-26T20:08:27.4926445Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-08-26T20:08:27.4931294Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-08-26T20:08:27.4936307Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-08-26T20:08:27.4941934Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-08-26T20:08:27.4945112Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-08-26T20:08:27.4945696Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-08-26T20:08:27.4958463Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-08-26T20:08:27.4959235Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-08-26T20:08:27.4965247Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-08-26T20:08:27.4965950Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-08-26T20:08:27.4978317Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-08-26T20:08:27.4978973Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-08-26T20:08:27.4981196Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-08-26T20:08:27.4998892Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-08-26T20:08:27.5007299Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-08-26T20:08:27.5007880Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:08:27.5011734Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-08-26T20:08:27.5034775Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-08-26T20:08:27.5035423Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-08-26T20:08:27.5035986Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-08-26T20:08:27.5054211Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-08-26T20:08:27.5054860Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-08-26T20:08:27.5056426Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-08-26T20:08:27.5060705Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-08-26T20:08:27.5076581Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-08-26T20:08:27.5078498Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-08-26T20:08:27.5081993Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-08-26T20:08:27.5096866Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-08-26T20:08:27.5107366Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-08-26T20:08:27.5112513Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-08-26T20:08:27.5115919Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-08-26T20:08:27.5132682Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-08-26T20:08:27.5137590Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-08-26T20:08:27.5139536Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-08-26T20:08:27.5171122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-08-26T20:08:27.7531461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-08-26T20:08:27.7532008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-08-26T20:08:27.7532466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-08-26T20:08:27.7542374Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-08-26T20:08:28.0307177Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-08-26T20:08:28.0307720Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-08-26T20:08:28.0308216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-08-26T20:08:28.0308684Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-08-26T20:08:28.0309136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-08-26T20:08:28.0309862Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-08-26T20:08:28.0310349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-08-26T20:08:28.0314301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-08-26T20:08:28.0315134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-08-26T20:08:28.0316321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-08-26T20:08:28.1308756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-08-26T20:08:28.9797413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-08-26T20:08:28.9798342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-08-26T20:08:28.9800557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-08-26T20:08:28.9801658Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-08-26T20:08:28.9802469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-08-26T20:08:28.9803264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-08-26T20:08:28.9804128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-08-26T20:08:29.0497457Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-08-26T20:08:29.3631862Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-08-26T20:08:29.3632395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-08-26T20:08:29.3845031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-08-26T20:08:41.5764665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-08-26T20:08:41.5765431Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-08-26T20:08:41.5765905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-08-26T20:08:41.5766372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-08-26T20:08:41.5767065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-08-26T20:08:41.5767507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-08-26T20:08:41.5767949Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-08-26T20:08:41.5768387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-08-26T20:08:41.5768846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-08-26T20:08:41.5769272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-08-26T20:08:41.5905238Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-08-26T20:08:41.6024342Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-08-26T20:08:41.6116519Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-08-26T20:08:41.6324099Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-08-26T20:08:41.7021371Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-08-26T20:08:41.7453146Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-08-26T20:08:42.3167738Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-08-26T20:08:42.4546651Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-08-26T20:08:42.4570062Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:08:42.4595385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-08-26T20:08:46.4760205Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-08-26T20:08:46.4973033Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-08-26T20:08:46.7579450Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-08-26T20:08:46.8037478Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c' 2025-08-26T20:08:46.8935674Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-08-26T20:08:46.9335177Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6' 2025-08-26T20:08:47.4715776Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491' 2025-08-26T20:08:47.5908264Z Submodule path 'third_party/fbgemm': checked out '21c7d30c526c0f1ad873ecc632dca6cfa8a69067' 2025-08-26T20:08:47.5933437Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-08-26T20:08:47.5942410Z Submodule 'external/composable_kernel' (https://github.com/jwfromm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:08:47.5944343Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:08:47.5945214Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-08-26T20:08:47.5945944Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-08-26T20:08:47.5946821Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:08:47.5947880Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-08-26T20:08:47.5970318Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-08-26T20:08:48.8184065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-08-26T20:08:48.8185176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-08-26T20:08:48.8186214Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-08-26T20:08:48.8922820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-08-26T20:08:48.9925452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-08-26T20:08:50.1204439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-08-26T20:08:54.2491795Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-08-26T20:08:54.4548909Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9' 2025-08-26T20:08:54.5476567Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-08-26T20:08:55.0724476Z Submodule path 'third_party/fbgemm/external/cutlass': checked out 'b40777404c174b9694a870bff5c13ce6b7f656ad' 2025-08-26T20:08:55.1149563Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-26T20:08:55.1269073Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c' 2025-08-26T20:08:55.2242342Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-08-26T20:08:55.2865199Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-08-26T20:08:55.2874827Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:08:55.2875602Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:08:55.2905947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-08-26T20:08:59.3222047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-08-26T20:08:59.5090618Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-08-26T20:08:59.9887452Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-08-26T20:09:00.1026828Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-08-26T20:09:00.1349080Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-08-26T20:09:00.1698851Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-08-26T20:09:00.1926425Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2' 2025-08-26T20:09:00.2343247Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-26T20:09:00.2470753Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-08-26T20:09:00.2485793Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-08-26T20:09:00.2513774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-08-26T20:09:12.2670654Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-08-26T20:09:12.2859071Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-08-26T20:09:12.3751020Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658' 2025-08-26T20:09:12.3769378Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:09:12.3770423Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:09:12.3771246Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:09:12.3800373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-08-26T20:09:13.0398658Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-08-26T20:09:13.6812391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-08-26T20:09:13.7532839Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-08-26T20:09:13.7548045Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:09:13.7549341Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:09:13.7550044Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:09:13.7555779Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:09:13.7556585Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:09:13.7557421Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:09:13.7558237Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:09:13.7558992Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:09:13.7585466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-08-26T20:09:15.2837225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-08-26T20:09:15.2838046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-08-26T20:09:15.2838825Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-08-26T20:09:15.2839985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-08-26T20:09:15.2840711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-08-26T20:09:15.2841725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-08-26T20:09:15.3840494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-08-26T20:09:20.4118911Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-08-26T20:09:20.4283309Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-08-26T20:09:20.4613873Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-08-26T20:09:20.4749994Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-08-26T20:09:20.4765212Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:09:20.4789185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-08-26T20:09:20.9691396Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-08-26T20:09:20.9874337Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-08-26T20:09:21.0243612Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-08-26T20:09:21.1124657Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-08-26T20:09:21.1280320Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-08-26T20:09:21.1634064Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-08-26T20:09:21.2182136Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-08-26T20:09:21.2565842Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7' 2025-08-26T20:09:21.2917385Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-08-26T20:09:21.3882997Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-08-26T20:09:21.6812108Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-08-26T20:09:21.6840254Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-08-26T20:09:21.6868736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-08-26T20:09:22.8838771Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-08-26T20:09:22.9383541Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-08-26T20:09:22.9404231Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:09:22.9410496Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:09:22.9416836Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:09:22.9418157Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:09:22.9418959Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:09:22.9419800Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:09:22.9420591Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:09:22.9421246Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:09:22.9448523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-08-26T20:09:23.3596845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-08-26T20:09:23.3597659Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-08-26T20:09:23.3598331Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-08-26T20:09:23.3599239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-08-26T20:09:23.4598352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-08-26T20:09:24.0292423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-08-26T20:09:31.1579731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-08-26T20:09:31.4460322Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-08-26T20:09:31.4827565Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-08-26T20:09:31.4995589Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-08-26T20:09:31.5938508Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-08-26T20:09:31.6076805Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-08-26T20:09:31.6205304Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-08-26T20:09:31.6345564Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-08-26T20:09:31.6359600Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:09:31.6360526Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:09:31.6390683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-08-26T20:09:33.3963344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-08-26T20:09:33.6170178Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-08-26T20:09:33.6573770Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-08-26T20:09:33.9974435Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-08-26T20:09:34.0076332Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-08-26T20:09:34.2320032Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-08-26T20:09:34.2333072Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:09:34.2333884Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-08-26T20:09:34.2358712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-08-26T20:09:34.7807367Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-08-26T20:09:35.2031662Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-08-26T20:09:35.2673080Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-08-26T20:09:35.2765020Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-08-26T20:09:35.2877288Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-08-26T20:09:35.3217279Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-08-26T20:09:35.3475586Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-08-26T20:09:35.3869789Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-08-26T20:09:35.4106664Z Submodule path 'third_party/tensorpipe': checked out 'af0118d13e52f5a08841464a768e01a0bf3e3075' 2025-08-26T20:09:35.4125092Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:09:35.4126023Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:09:35.4135502Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:09:35.4136334Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:09:35.4162234Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-08-26T20:09:36.3687591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-08-26T20:09:36.3977139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-08-26T20:09:36.6311606Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-08-26T20:09:36.6843126Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-08-26T20:09:36.6986263Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-08-26T20:09:36.7635910Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-08-26T20:09:36.7894900Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-08-26T20:09:36.7910575Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:09:36.7935482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-08-26T20:09:37.0139980Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-08-26T20:09:37.0175568Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-08-26T20:09:37.0491593Z Entering 'android/libs/fbjni' 2025-08-26T20:09:37.0535936Z Entering 'third_party/FP16' 2025-08-26T20:09:37.0578125Z Entering 'third_party/FXdiv' 2025-08-26T20:09:37.0629072Z Entering 'third_party/NNPACK' 2025-08-26T20:09:37.0669058Z Entering 'third_party/NVTX' 2025-08-26T20:09:37.0709581Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-26T20:09:37.0752734Z Entering 'third_party/XNNPACK' 2025-08-26T20:09:37.0809540Z Entering 'third_party/aiter' 2025-08-26T20:09:37.0848409Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:09:37.0891651Z Entering 'third_party/benchmark' 2025-08-26T20:09:37.0937101Z Entering 'third_party/composable_kernel' 2025-08-26T20:09:37.0984785Z Entering 'third_party/cpp-httplib' 2025-08-26T20:09:37.1023510Z Entering 'third_party/cpuinfo' 2025-08-26T20:09:37.1068494Z Entering 'third_party/cudnn_frontend' 2025-08-26T20:09:37.1112334Z Entering 'third_party/cutlass' 2025-08-26T20:09:37.1158039Z Entering 'third_party/fbgemm' 2025-08-26T20:09:37.1201510Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-26T20:09:37.1239960Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:09:37.1287381Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:09:37.1332523Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-26T20:09:37.1379946Z Entering 'third_party/fbgemm/external/googletest' 2025-08-26T20:09:37.1409812Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:09:37.1446691Z Entering 'third_party/fbgemm/external/json' 2025-08-26T20:09:37.1492828Z Entering 'third_party/flash-attention' 2025-08-26T20:09:37.1531762Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:09:37.1577099Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:09:37.1632635Z Entering 'third_party/flatbuffers' 2025-08-26T20:09:37.1671726Z Entering 'third_party/fmt' 2025-08-26T20:09:37.1715288Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:09:37.1755333Z Entering 'third_party/gloo' 2025-08-26T20:09:37.1793037Z Entering 'third_party/googletest' 2025-08-26T20:09:37.1833966Z Entering 'third_party/ideep' 2025-08-26T20:09:37.1873534Z Entering 'third_party/ideep/mkl-dnn' 2025-08-26T20:09:37.1916396Z Entering 'third_party/ittapi' 2025-08-26T20:09:37.1962864Z Entering 'third_party/kineto' 2025-08-26T20:09:37.2004801Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:09:37.2046268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:09:37.2082059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:09:37.2121787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:09:37.2162067Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:09:37.2204101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:09:37.2251382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:09:37.2289286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:09:37.2336089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:09:37.2376440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:09:37.2425169Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:09:37.2465034Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:09:37.2508406Z Entering 'third_party/kleidiai' 2025-08-26T20:09:37.2546553Z Entering 'third_party/mimalloc' 2025-08-26T20:09:37.2587252Z Entering 'third_party/nlohmann' 2025-08-26T20:09:37.2632213Z Entering 'third_party/onnx' 2025-08-26T20:09:37.2686234Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-26T20:09:37.2734673Z Entering 'third_party/opentelemetry-cpp' 2025-08-26T20:09:37.2775203Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:09:37.2815064Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:09:37.2853618Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:09:37.2895923Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:09:37.2939751Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:09:37.2977552Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:09:37.3018253Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:09:37.3053577Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:09:37.3095850Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:09:37.3137568Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:09:37.3189126Z Entering 'third_party/pocketfft' 2025-08-26T20:09:37.3236809Z Entering 'third_party/protobuf' 2025-08-26T20:09:37.3280767Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:09:37.3325972Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-26T20:09:37.3367187Z Entering 'third_party/psimd' 2025-08-26T20:09:37.3406743Z Entering 'third_party/pthreadpool' 2025-08-26T20:09:37.3444913Z Entering 'third_party/pybind11' 2025-08-26T20:09:37.3480722Z Entering 'third_party/python-peachpy' 2025-08-26T20:09:37.3531462Z Entering 'third_party/sleef' 2025-08-26T20:09:37.3571578Z Entering 'third_party/tensorpipe' 2025-08-26T20:09:37.3610777Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:09:37.3656217Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:09:37.3689956Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:09:37.3733753Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:09:37.3768770Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:09:37.3827226Z ##[endgroup] 2025-08-26T20:09:37.3827726Z ##[group]Persisting credentials for submodules 2025-08-26T20:09:37.3833001Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-08-26T20:09:37.4151439Z Entering 'android/libs/fbjni' 2025-08-26T20:09:37.4211273Z Entering 'third_party/FP16' 2025-08-26T20:09:37.4268891Z Entering 'third_party/FXdiv' 2025-08-26T20:09:37.4326392Z Entering 'third_party/NNPACK' 2025-08-26T20:09:37.4380556Z Entering 'third_party/NVTX' 2025-08-26T20:09:37.4439723Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-26T20:09:37.4490819Z Entering 'third_party/XNNPACK' 2025-08-26T20:09:37.4555457Z Entering 'third_party/aiter' 2025-08-26T20:09:37.4610912Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:09:37.4673264Z Entering 'third_party/benchmark' 2025-08-26T20:09:37.4732710Z Entering 'third_party/composable_kernel' 2025-08-26T20:09:37.4796749Z Entering 'third_party/cpp-httplib' 2025-08-26T20:09:37.4853801Z Entering 'third_party/cpuinfo' 2025-08-26T20:09:37.4912527Z Entering 'third_party/cudnn_frontend' 2025-08-26T20:09:37.4966827Z Entering 'third_party/cutlass' 2025-08-26T20:09:37.5027369Z Entering 'third_party/fbgemm' 2025-08-26T20:09:37.5080860Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-26T20:09:37.5140464Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:09:37.5193415Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:09:37.5250813Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-26T20:09:37.5313832Z Entering 'third_party/fbgemm/external/googletest' 2025-08-26T20:09:37.5372419Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:09:37.5433812Z Entering 'third_party/fbgemm/external/json' 2025-08-26T20:09:37.5479792Z Entering 'third_party/flash-attention' 2025-08-26T20:09:37.5543079Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:09:37.5595569Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:09:37.5663247Z Entering 'third_party/flatbuffers' 2025-08-26T20:09:37.5714446Z Entering 'third_party/fmt' 2025-08-26T20:09:37.5771431Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:09:37.5829738Z Entering 'third_party/gloo' 2025-08-26T20:09:37.5879608Z Entering 'third_party/googletest' 2025-08-26T20:09:37.5941482Z Entering 'third_party/ideep' 2025-08-26T20:09:37.5986203Z Entering 'third_party/ideep/mkl-dnn' 2025-08-26T20:09:37.6049989Z Entering 'third_party/ittapi' 2025-08-26T20:09:37.6108168Z Entering 'third_party/kineto' 2025-08-26T20:09:37.6160004Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:09:37.6210865Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:09:37.6267431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:09:37.6321685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:09:37.6375889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:09:37.6431268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:09:37.6488686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:09:37.6552232Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:09:37.6608060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:09:37.6661470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:09:37.6714483Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:09:37.6770301Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:09:37.6831426Z Entering 'third_party/kleidiai' 2025-08-26T20:09:37.6880090Z Entering 'third_party/mimalloc' 2025-08-26T20:09:37.6936536Z Entering 'third_party/nlohmann' 2025-08-26T20:09:37.6990871Z Entering 'third_party/onnx' 2025-08-26T20:09:37.7067370Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-26T20:09:37.7122273Z Entering 'third_party/opentelemetry-cpp' 2025-08-26T20:09:37.7176788Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:09:37.7237905Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:09:37.7291591Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:09:37.7349186Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:09:37.7410653Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:09:37.7460355Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:09:37.7513562Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:09:37.7563493Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:09:37.7619006Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:09:37.7676584Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:09:37.7748422Z Entering 'third_party/pocketfft' 2025-08-26T20:09:37.7805970Z Entering 'third_party/protobuf' 2025-08-26T20:09:37.7869547Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:09:37.7930356Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-26T20:09:37.7981106Z Entering 'third_party/psimd' 2025-08-26T20:09:37.8032548Z Entering 'third_party/pthreadpool' 2025-08-26T20:09:37.8085715Z Entering 'third_party/pybind11' 2025-08-26T20:09:37.8140129Z Entering 'third_party/python-peachpy' 2025-08-26T20:09:37.8191416Z Entering 'third_party/sleef' 2025-08-26T20:09:37.8249134Z Entering 'third_party/tensorpipe' 2025-08-26T20:09:37.8302635Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:09:37.8355002Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:09:37.8405767Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:09:37.8462610Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:09:37.8510339Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:09:37.8585313Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-08-26T20:09:37.8906615Z Entering 'android/libs/fbjni' 2025-08-26T20:09:37.8965985Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-08-26T20:09:37.8981091Z Entering 'third_party/FP16' 2025-08-26T20:09:37.9034282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-08-26T20:09:37.9046761Z Entering 'third_party/FXdiv' 2025-08-26T20:09:37.9096650Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-08-26T20:09:37.9112827Z Entering 'third_party/NNPACK' 2025-08-26T20:09:37.9161679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-08-26T20:09:37.9180361Z Entering 'third_party/NVTX' 2025-08-26T20:09:37.9227834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-08-26T20:09:37.9250193Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-26T20:09:37.9292657Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-08-26T20:09:37.9312704Z Entering 'third_party/XNNPACK' 2025-08-26T20:09:37.9358226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-08-26T20:09:37.9385458Z Entering 'third_party/aiter' 2025-08-26T20:09:37.9438113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-08-26T20:09:37.9456274Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:09:37.9501185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-08-26T20:09:37.9532795Z Entering 'third_party/benchmark' 2025-08-26T20:09:37.9576571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-08-26T20:09:37.9593533Z Entering 'third_party/composable_kernel' 2025-08-26T20:09:37.9645067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-08-26T20:09:37.9667551Z Entering 'third_party/cpp-httplib' 2025-08-26T20:09:37.9720820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-08-26T20:09:37.9735556Z Entering 'third_party/cpuinfo' 2025-08-26T20:09:37.9787026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-08-26T20:09:37.9807186Z Entering 'third_party/cudnn_frontend' 2025-08-26T20:09:37.9854220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-08-26T20:09:37.9876101Z Entering 'third_party/cutlass' 2025-08-26T20:09:37.9926619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-08-26T20:09:37.9946373Z Entering 'third_party/fbgemm' 2025-08-26T20:09:37.9995216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-08-26T20:09:38.0013152Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-26T20:09:38.0059152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-08-26T20:09:38.0080691Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:09:38.0129199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-08-26T20:09:38.0156000Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:09:38.0210221Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-08-26T20:09:38.0223649Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-26T20:09:38.0272923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-08-26T20:09:38.0300428Z Entering 'third_party/fbgemm/external/googletest' 2025-08-26T20:09:38.0357812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-08-26T20:09:38.0358473Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:09:38.0410616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-08-26T20:09:38.0428422Z Entering 'third_party/fbgemm/external/json' 2025-08-26T20:09:38.0475287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-08-26T20:09:38.0496837Z Entering 'third_party/flash-attention' 2025-08-26T20:09:38.0542207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-08-26T20:09:38.0560016Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:09:38.0611011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-08-26T20:09:38.0637080Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:09:38.0691357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-08-26T20:09:38.0712776Z Entering 'third_party/flatbuffers' 2025-08-26T20:09:38.0761615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-08-26T20:09:38.0780683Z Entering 'third_party/fmt' 2025-08-26T20:09:38.0832993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-08-26T20:09:38.0849942Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:09:38.0897978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-08-26T20:09:38.0917954Z Entering 'third_party/gloo' 2025-08-26T20:09:38.0964045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-08-26T20:09:38.0981675Z Entering 'third_party/googletest' 2025-08-26T20:09:38.1030043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-08-26T20:09:38.1049021Z Entering 'third_party/ideep' 2025-08-26T20:09:38.1098293Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-08-26T20:09:38.1110240Z Entering 'third_party/ideep/mkl-dnn' 2025-08-26T20:09:38.1161499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-08-26T20:09:38.1183187Z Entering 'third_party/ittapi' 2025-08-26T20:09:38.1229735Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-08-26T20:09:38.1252065Z Entering 'third_party/kineto' 2025-08-26T20:09:38.1301714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-08-26T20:09:38.1320399Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:09:38.1371091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-08-26T20:09:38.1384872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:09:38.1435677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-08-26T20:09:38.1453163Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:09:38.1506103Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-08-26T20:09:38.1519996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:09:38.1567584Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-08-26T20:09:38.1585242Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:09:38.1633630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-08-26T20:09:38.1650641Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:09:38.1695753Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-08-26T20:09:38.1716711Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:09:38.1763591Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-08-26T20:09:38.1779022Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:09:38.1832809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-08-26T20:09:38.1855233Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:09:38.1906287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-08-26T20:09:38.1936012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:09:38.1976216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-08-26T20:09:38.1993646Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:09:38.2046909Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-08-26T20:09:38.2069482Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:09:38.2115683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-08-26T20:09:38.2143834Z Entering 'third_party/kleidiai' 2025-08-26T20:09:38.2188677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-08-26T20:09:38.2213979Z Entering 'third_party/mimalloc' 2025-08-26T20:09:38.2263769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-08-26T20:09:38.2276612Z Entering 'third_party/nlohmann' 2025-08-26T20:09:38.2327372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-08-26T20:09:38.2345694Z Entering 'third_party/onnx' 2025-08-26T20:09:38.2396672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-08-26T20:09:38.2428740Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-26T20:09:38.2475887Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-08-26T20:09:38.2495099Z Entering 'third_party/opentelemetry-cpp' 2025-08-26T20:09:38.2549649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-08-26T20:09:38.2565081Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:09:38.2625779Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-08-26T20:09:38.2640911Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:09:38.2686430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-08-26T20:09:38.2708152Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:09:38.2754950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-08-26T20:09:38.2774139Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:09:38.2823463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-08-26T20:09:38.2841138Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:09:38.2893321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-08-26T20:09:38.2911628Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:09:38.2958900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-08-26T20:09:38.2975089Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:09:38.3025999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-08-26T20:09:38.3034991Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:09:38.3086068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-08-26T20:09:38.3102957Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:09:38.3153428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-08-26T20:09:38.3174639Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:09:38.3221474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-08-26T20:09:38.3262470Z Entering 'third_party/pocketfft' 2025-08-26T20:09:38.3312377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-08-26T20:09:38.3332489Z Entering 'third_party/protobuf' 2025-08-26T20:09:38.3378205Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-08-26T20:09:38.3396588Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:09:38.3446561Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-08-26T20:09:38.3464087Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-26T20:09:38.3509611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-08-26T20:09:38.3530501Z Entering 'third_party/psimd' 2025-08-26T20:09:38.3584878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-08-26T20:09:38.3599617Z Entering 'third_party/pthreadpool' 2025-08-26T20:09:38.3649914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-08-26T20:09:38.3662436Z Entering 'third_party/pybind11' 2025-08-26T20:09:38.3713262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-08-26T20:09:38.3730638Z Entering 'third_party/python-peachpy' 2025-08-26T20:09:38.3781933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-08-26T20:09:38.3799581Z Entering 'third_party/sleef' 2025-08-26T20:09:38.3853198Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-08-26T20:09:38.3869018Z Entering 'third_party/tensorpipe' 2025-08-26T20:09:38.3922805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-08-26T20:09:38.3938585Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:09:38.3984145Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-08-26T20:09:38.3999574Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:09:38.4048405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-08-26T20:09:38.4061468Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:09:38.4115060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-08-26T20:09:38.4128070Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:09:38.4177204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-08-26T20:09:38.4188466Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:09:38.4236836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-08-26T20:09:38.5293960Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-08-26T20:09:38.5612402Z Entering 'android/libs/fbjni' 2025-08-26T20:09:38.5656632Z Entering 'third_party/FP16' 2025-08-26T20:09:38.5695691Z Entering 'third_party/FXdiv' 2025-08-26T20:09:38.5741464Z Entering 'third_party/NNPACK' 2025-08-26T20:09:38.5778642Z Entering 'third_party/NVTX' 2025-08-26T20:09:38.5824521Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-26T20:09:38.5866589Z Entering 'third_party/XNNPACK' 2025-08-26T20:09:38.5917302Z Entering 'third_party/aiter' 2025-08-26T20:09:38.5953811Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:09:38.6007233Z Entering 'third_party/benchmark' 2025-08-26T20:09:38.6047883Z Entering 'third_party/composable_kernel' 2025-08-26T20:09:38.6097474Z Entering 'third_party/cpp-httplib' 2025-08-26T20:09:38.6145053Z Entering 'third_party/cpuinfo' 2025-08-26T20:09:38.6181038Z Entering 'third_party/cudnn_frontend' 2025-08-26T20:09:38.6225477Z Entering 'third_party/cutlass' 2025-08-26T20:09:38.6274161Z Entering 'third_party/fbgemm' 2025-08-26T20:09:38.6314890Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-26T20:09:38.6358722Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:09:38.6408544Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:09:38.6447019Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-26T20:09:38.6491552Z Entering 'third_party/fbgemm/external/googletest' 2025-08-26T20:09:38.6537410Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:09:38.6581007Z Entering 'third_party/fbgemm/external/json' 2025-08-26T20:09:38.6619869Z Entering 'third_party/flash-attention' 2025-08-26T20:09:38.6663284Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:09:38.6709810Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:09:38.6754715Z Entering 'third_party/flatbuffers' 2025-08-26T20:09:38.6795714Z Entering 'third_party/fmt' 2025-08-26T20:09:38.6836109Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:09:38.6875493Z Entering 'third_party/gloo' 2025-08-26T20:09:38.6918323Z Entering 'third_party/googletest' 2025-08-26T20:09:38.6959989Z Entering 'third_party/ideep' 2025-08-26T20:09:38.6997785Z Entering 'third_party/ideep/mkl-dnn' 2025-08-26T20:09:38.7045655Z Entering 'third_party/ittapi' 2025-08-26T20:09:38.7085167Z Entering 'third_party/kineto' 2025-08-26T20:09:38.7126074Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:09:38.7163468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:09:38.7209176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:09:38.7251628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:09:38.7291847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:09:38.7333673Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:09:38.7380304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:09:38.7424073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:09:38.7464274Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:09:38.7504279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:09:38.7542439Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:09:38.7581336Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:09:38.7625281Z Entering 'third_party/kleidiai' 2025-08-26T20:09:38.7667459Z Entering 'third_party/mimalloc' 2025-08-26T20:09:38.7706148Z Entering 'third_party/nlohmann' 2025-08-26T20:09:38.7753526Z Entering 'third_party/onnx' 2025-08-26T20:09:38.7810946Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-26T20:09:38.7854154Z Entering 'third_party/opentelemetry-cpp' 2025-08-26T20:09:38.7895248Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:09:38.7941861Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:09:38.7977872Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:09:38.8016500Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:09:38.8055912Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:09:38.8090741Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:09:38.8130645Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:09:38.8173567Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:09:38.8213329Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:09:38.8250725Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:09:38.8309511Z Entering 'third_party/pocketfft' 2025-08-26T20:09:38.8347988Z Entering 'third_party/protobuf' 2025-08-26T20:09:38.8390786Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:09:38.8432401Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-26T20:09:38.8472591Z Entering 'third_party/psimd' 2025-08-26T20:09:38.8509411Z Entering 'third_party/pthreadpool' 2025-08-26T20:09:38.8548894Z Entering 'third_party/pybind11' 2025-08-26T20:09:38.8587004Z Entering 'third_party/python-peachpy' 2025-08-26T20:09:38.8630552Z Entering 'third_party/sleef' 2025-08-26T20:09:38.8667250Z Entering 'third_party/tensorpipe' 2025-08-26T20:09:38.8713840Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:09:38.8747857Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:09:38.8784641Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:09:38.8824909Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:09:38.8860101Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:09:38.8924728Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-08-26T20:09:38.9233054Z Entering 'android/libs/fbjni' 2025-08-26T20:09:38.9269230Z Entering 'third_party/FP16' 2025-08-26T20:09:38.9310610Z Entering 'third_party/FXdiv' 2025-08-26T20:09:38.9352084Z Entering 'third_party/NNPACK' 2025-08-26T20:09:38.9389528Z Entering 'third_party/NVTX' 2025-08-26T20:09:38.9430042Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-26T20:09:38.9467474Z Entering 'third_party/XNNPACK' 2025-08-26T20:09:38.9526252Z Entering 'third_party/aiter' 2025-08-26T20:09:38.9568801Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:09:38.9616195Z Entering 'third_party/benchmark' 2025-08-26T20:09:38.9653689Z Entering 'third_party/composable_kernel' 2025-08-26T20:09:38.9703476Z Entering 'third_party/cpp-httplib' 2025-08-26T20:09:38.9743506Z Entering 'third_party/cpuinfo' 2025-08-26T20:09:38.9785504Z Entering 'third_party/cudnn_frontend' 2025-08-26T20:09:38.9833857Z Entering 'third_party/cutlass' 2025-08-26T20:09:38.9881999Z Entering 'third_party/fbgemm' 2025-08-26T20:09:38.9927491Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-26T20:09:38.9971579Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:09:39.0017477Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:09:39.0054873Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-26T20:09:39.0101263Z Entering 'third_party/fbgemm/external/googletest' 2025-08-26T20:09:39.0145093Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:09:39.0182514Z Entering 'third_party/fbgemm/external/json' 2025-08-26T20:09:39.0233872Z Entering 'third_party/flash-attention' 2025-08-26T20:09:39.0271198Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:09:39.0314107Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:09:39.0365496Z Entering 'third_party/flatbuffers' 2025-08-26T20:09:39.0411203Z Entering 'third_party/fmt' 2025-08-26T20:09:39.0453380Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:09:39.0496735Z Entering 'third_party/gloo' 2025-08-26T20:09:39.0535024Z Entering 'third_party/googletest' 2025-08-26T20:09:39.0574072Z Entering 'third_party/ideep' 2025-08-26T20:09:39.0613478Z Entering 'third_party/ideep/mkl-dnn' 2025-08-26T20:09:39.0664089Z Entering 'third_party/ittapi' 2025-08-26T20:09:39.0704012Z Entering 'third_party/kineto' 2025-08-26T20:09:39.0742375Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:09:39.0782767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:09:39.0827688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:09:39.0872319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:09:39.0913786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:09:39.0952997Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:09:39.0992241Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:09:39.1035112Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:09:39.1080766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:09:39.1119823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:09:39.1160834Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:09:39.1200796Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:09:39.1252004Z Entering 'third_party/kleidiai' 2025-08-26T20:09:39.1286183Z Entering 'third_party/mimalloc' 2025-08-26T20:09:39.1329129Z Entering 'third_party/nlohmann' 2025-08-26T20:09:39.1372091Z Entering 'third_party/onnx' 2025-08-26T20:09:39.1425692Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-26T20:09:39.1469362Z Entering 'third_party/opentelemetry-cpp' 2025-08-26T20:09:39.1512231Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:09:39.1552443Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:09:39.1589174Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:09:39.1634310Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:09:39.1675761Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:09:39.1716768Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:09:39.1755328Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:09:39.1794942Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:09:39.1839678Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:09:39.1884578Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:09:39.1943517Z Entering 'third_party/pocketfft' 2025-08-26T20:09:39.1983470Z Entering 'third_party/protobuf' 2025-08-26T20:09:39.2030202Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:09:39.2072861Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-26T20:09:39.2114924Z Entering 'third_party/psimd' 2025-08-26T20:09:39.2152315Z Entering 'third_party/pthreadpool' 2025-08-26T20:09:39.2195288Z Entering 'third_party/pybind11' 2025-08-26T20:09:39.2237780Z Entering 'third_party/python-peachpy' 2025-08-26T20:09:39.2276986Z Entering 'third_party/sleef' 2025-08-26T20:09:39.2324513Z Entering 'third_party/tensorpipe' 2025-08-26T20:09:39.2363646Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:09:39.2400256Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:09:39.2437938Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:09:39.2479087Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:09:39.2517314Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:09:39.2575805Z ##[endgroup] 2025-08-26T20:09:39.2618729Z [command]/usr/bin/git log -1 --format=%H 2025-08-26T20:09:39.2645685Z 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:09:39.2820960Z Prepare all required actions 2025-08-26T20:09:39.2821595Z Getting action download info 2025-08-26T20:09:39.4300662Z ##[group]Run ./.github/actions/setup-linux 2025-08-26T20:09:39.4300910Z env: 2025-08-26T20:09:39.4301079Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:39.4301264Z ##[endgroup] 2025-08-26T20:09:39.4341716Z ##[group]Run set -euo pipefail 2025-08-26T20:09:39.4341999Z set -euo pipefail 2025-08-26T20:09:39.4342213Z function get_ec2_metadata() { 2025-08-26T20:09:39.4342471Z  # Pulled from instance metadata endpoint for EC2 2025-08-26T20:09:39.4342899Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-08-26T20:09:39.4343256Z  category=$1 2025-08-26T20:09:39.4343504Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-08-26T20:09:39.4343780Z  runner_name_str=i-04c468ba96b53884f 2025-08-26T20:09:39.4344062Z  if [[ -f /.inarc ]]; then 2025-08-26T20:09:39.4344300Z  echo "ARC Runner, no info on ec2 metadata" 2025-08-26T20:09:39.4344563Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-08-26T20:09:39.4344865Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-08-26T20:09:39.4345134Z  else 2025-08-26T20:09:39.4345667Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-08-26T20:09:39.4346200Z  fi 2025-08-26T20:09:39.4346352Z } 2025-08-26T20:09:39.4346546Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-08-26T20:09:39.4346826Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-08-26T20:09:39.4347131Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-08-26T20:09:39.4347394Z echo "system info $(uname -a)" 2025-08-26T20:09:39.4353638Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:39.4353898Z env: 2025-08-26T20:09:39.4354072Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:39.4354262Z ##[endgroup] 2025-08-26T20:09:39.4491439Z ami-id: ami-05ffe3c48a9991133 2025-08-26T20:09:39.4599614Z instance-id: i-04c468ba96b53884f 2025-08-26T20:09:39.4694132Z instance-type: m7i-flex.8xlarge 2025-08-26T20:09:39.4708762Z system info Linux ip-10-0-58-230.ec2.internal 6.1.141-155.222.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 17 10:29:47 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-08-26T20:09:39.4735169Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-26T20:09:39.4735745Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-26T20:09:39.4740557Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:39.4740818Z env: 2025-08-26T20:09:39.4740967Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:39.4741181Z ##[endgroup] 2025-08-26T20:09:39.4797706Z ##[group]Run if systemctl is-active --quiet docker; then 2025-08-26T20:09:39.4798037Z if systemctl is-active --quiet docker; then 2025-08-26T20:09:39.4798316Z  echo "Docker daemon is running..."; 2025-08-26T20:09:39.4798546Z else 2025-08-26T20:09:39.4798798Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-08-26T20:09:39.4799085Z fi 2025-08-26T20:09:39.4803474Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:39.4803735Z env: 2025-08-26T20:09:39.4803902Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:39.4804092Z ##[endgroup] 2025-08-26T20:09:39.4877273Z Docker daemon is running... 2025-08-26T20:09:39.4915388Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-26T20:09:39.4915621Z with: 2025-08-26T20:09:39.4915778Z shell: bash 2025-08-26T20:09:39.4916083Z timeout_minutes: 5 2025-08-26T20:09:39.4916270Z max_attempts: 3 2025-08-26T20:09:39.4916562Z retry_wait_seconds: 30 2025-08-26T20:09:39.4918064Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-08-26T20:09:39.4919776Z polling_interval_seconds: 1 2025-08-26T20:09:39.4920000Z warning_on_retry: true 2025-08-26T20:09:39.4920199Z continue_on_error: false 2025-08-26T20:09:39.4920390Z env: 2025-08-26T20:09:39.4920566Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:39.4920768Z AWS_RETRY_MODE: standard 2025-08-26T20:09:39.4920968Z AWS_MAX_ATTEMPTS: 5 2025-08-26T20:09:39.4921163Z AWS_DEFAULT_REGION: us-east-1 2025-08-26T20:09:39.4921373Z ##[endgroup] 2025-08-26T20:09:40.4470411Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-26T20:09:40.4473206Z Configure a credential helper to remove this warning. See 2025-08-26T20:09:40.4473660Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-26T20:09:40.4473927Z 2025-08-26T20:09:40.4474008Z Login Succeeded 2025-08-26T20:09:40.5634590Z Command completed after 1 attempt(s). 2025-08-26T20:09:40.5693087Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-26T20:09:40.5693477Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-26T20:09:40.5693766Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-26T20:09:40.5700489Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:40.5700732Z env: 2025-08-26T20:09:40.5700909Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:40.5701095Z ##[endgroup] 2025-08-26T20:09:40.5794069Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-26T20:09:40.5794481Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-26T20:09:40.5794772Z # shellcheck disable=SC2046 2025-08-26T20:09:40.5795015Z docker stop $(docker ps -q) || true 2025-08-26T20:09:40.5795252Z # Prune all of the docker images 2025-08-26T20:09:40.5795485Z docker system prune -af 2025-08-26T20:09:40.5801200Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:40.5801473Z env: 2025-08-26T20:09:40.5801642Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:40.5801846Z ##[endgroup] 2025-08-26T20:09:40.6250419Z "docker stop" requires at least 1 argument. 2025-08-26T20:09:40.6250767Z See 'docker stop --help'. 2025-08-26T20:09:40.6250936Z 2025-08-26T20:09:40.6251064Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-08-26T20:09:40.6251265Z 2025-08-26T20:09:40.6251350Z Stop one or more running containers 2025-08-26T20:09:40.6471576Z Total reclaimed space: 0B 2025-08-26T20:09:40.6503470Z ##[group]Run set +e 2025-08-26T20:09:40.6503695Z set +e 2025-08-26T20:09:40.6503872Z set -x 2025-08-26T20:09:40.6504067Z  2025-08-26T20:09:40.6504246Z PT_DOMAIN=download.pytorch.org 2025-08-26T20:09:40.6504639Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2025-08-26T20:09:40.6505124Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2025-08-26T20:09:40.6505465Z # one is returned at random 2025-08-26T20:09:40.6505739Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2025-08-26T20:09:40.6505986Z  2025-08-26T20:09:40.6506283Z if [ -z "${RESOLVED_IP}" ]; then 2025-08-26T20:09:40.6506584Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2025-08-26T20:09:40.6507033Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2025-08-26T20:09:40.6507289Z  2025-08-26T20:09:40.6507458Z  if [ -z "${RESOLVED_IP}" ]; then 2025-08-26T20:09:40.6507714Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2025-08-26T20:09:40.6507942Z  exit 1 2025-08-26T20:09:40.6508098Z  fi 2025-08-26T20:09:40.6508253Z fi 2025-08-26T20:09:40.6508400Z  2025-08-26T20:09:40.6508581Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2025-08-26T20:09:40.6508813Z  # Clean up any old records first 2025-08-26T20:09:40.6509055Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2025-08-26T20:09:40.6509276Z fi 2025-08-26T20:09:40.6509427Z  2025-08-26T20:09:40.6509642Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2025-08-26T20:09:40.6509904Z cat /etc/hosts 2025-08-26T20:09:40.6514682Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:40.6514939Z env: 2025-08-26T20:09:40.6515110Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:40.6515297Z ##[endgroup] 2025-08-26T20:09:40.6544351Z + PT_DOMAIN=download.pytorch.org 2025-08-26T20:09:40.6549674Z ++ tail -n1 2025-08-26T20:09:40.6549933Z ++ dig -4 +short download.pytorch.org 2025-08-26T20:09:40.7182983Z + RESOLVED_IP=18.160.10.36 2025-08-26T20:09:40.7183437Z + '[' -z 18.160.10.36 ']' 2025-08-26T20:09:40.7183775Z + grep -r download.pytorch.org /etc/hosts 2025-08-26T20:09:40.7203655Z + echo '18.160.10.36 download.pytorch.org' 2025-08-26T20:09:40.7204238Z + sudo tee -a /etc/hosts 2025-08-26T20:09:40.9814269Z 18.160.10.36 download.pytorch.org 2025-08-26T20:09:40.9832386Z + cat /etc/hosts 2025-08-26T20:09:40.9844001Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2025-08-26T20:09:40.9851868Z ::1 localhost6 localhost6.localdomain6 2025-08-26T20:09:40.9852151Z 18.160.10.36 download.pytorch.org 2025-08-26T20:09:40.9965998Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-08-26T20:09:40.9966330Z with: 2025-08-26T20:09:40.9966924Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:40.9967617Z use-custom-docker-registry: true 2025-08-26T20:09:40.9967845Z docker-build-dir: .ci/docker 2025-08-26T20:09:40.9968065Z docker-build-script: ./build.sh 2025-08-26T20:09:40.9968283Z working-directory: . 2025-08-26T20:09:40.9968540Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:40.9968816Z force-push: false 2025-08-26T20:09:40.9968997Z env: 2025-08-26T20:09:40.9969162Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:40.9969357Z ##[endgroup] 2025-08-26T20:09:40.9987418Z ##[group]Run set -ex 2025-08-26T20:09:40.9987672Z set -ex 2025-08-26T20:09:40.9987851Z  2025-08-26T20:09:40.9988192Z # If the docker build directory or the build script doesn't exist, the action will 2025-08-26T20:09:40.9988647Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-08-26T20:09:40.9989030Z # job could then download the pre-built image as usual 2025-08-26T20:09:40.9989501Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-08-26T20:09:40.9989935Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9990176Z else 2025-08-26T20:09:40.9990380Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9990692Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9990979Z  2025-08-26T20:09:40.9991371Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-08-26T20:09:40.9991897Z  exit 0 2025-08-26T20:09:40.9992065Z fi 2025-08-26T20:09:40.9992232Z  2025-08-26T20:09:40.9992489Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-08-26T20:09:40.9992901Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-08-26T20:09:40.9993274Z  # use it as it is, but first let's extract the tag 2025-08-26T20:09:40.9993606Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-08-26T20:09:40.9993963Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9994308Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9994590Z else 2025-08-26T20:09:40.9994804Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-08-26T20:09:40.9995075Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-08-26T20:09:40.9995367Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-08-26T20:09:40.9995607Z  fi 2025-08-26T20:09:40.9995930Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-08-26T20:09:40.9996593Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9997032Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9997501Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:40.9997797Z fi 2025-08-26T20:09:41.0005839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:41.0006081Z env: 2025-08-26T20:09:41.0006250Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:41.0006442Z REPO_NAME: pytorch 2025-08-26T20:09:41.0008162Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.0008763Z DOCKER_BUILD_DIR: .ci/docker 2025-08-26T20:09:41.0008957Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-08-26T20:09:41.0009219Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:41.0009486Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-08-26T20:09:41.0009688Z CUSTOM_TAG_PREFIX: 2025-08-26T20:09:41.0009857Z ##[endgroup] 2025-08-26T20:09:41.0034641Z + [[ -d .ci/docker ]] 2025-08-26T20:09:41.0035438Z + [[ -f .ci/docker/./build.sh ]] 2025-08-26T20:09:41.0035698Z + [[ true == \t\r\u\e ]] 2025-08-26T20:09:41.0035898Z + echo skip=false 2025-08-26T20:09:41.0036655Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-08-26T20:09:41.0042049Z ++ awk -F '[:,]' '{print $2}' 2025-08-26T20:09:41.0043237Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.0069006Z + DOCKER_TAG=pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.0069816Z + echo docker-tag=pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.0070618Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.0095533Z ##[group]Run set +e 2025-08-26T20:09:41.0095780Z set +e 2025-08-26T20:09:41.0095961Z set -x 2025-08-26T20:09:41.0096123Z  2025-08-26T20:09:41.0096481Z login() { 2025-08-26T20:09:41.0096836Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-26T20:09:41.0097364Z } 2025-08-26T20:09:41.0097522Z  2025-08-26T20:09:41.0097684Z retry () { 2025-08-26T20:09:41.0097894Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-26T20:09:41.0098123Z } 2025-08-26T20:09:41.0098276Z  2025-08-26T20:09:41.0098454Z retry login "${DOCKER_REGISTRY}" 2025-08-26T20:09:41.0098675Z  2025-08-26T20:09:41.0098842Z START_TIME=$(date +%s) 2025-08-26T20:09:41.0099057Z # Wait up to 120 minutes 2025-08-26T20:09:41.0099328Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-08-26T20:09:41.0099667Z  # Check if image already exists, if it does then skip building it 2025-08-26T20:09:41.0100004Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-08-26T20:09:41.0100250Z  exit 0 2025-08-26T20:09:41.0100425Z  fi 2025-08-26T20:09:41.0100591Z  2025-08-26T20:09:41.0100869Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-08-26T20:09:41.0101309Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-08-26T20:09:41.0101741Z  # latter, it will wait for the Docker images to become available before continuing 2025-08-26T20:09:41.0102089Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-08-26T20:09:41.0102363Z  # It's a Docker build job, let's build the image 2025-08-26T20:09:41.0102600Z  break 2025-08-26T20:09:41.0102768Z  else 2025-08-26T20:09:41.0102997Z  # It's a regular build job, wait for the image to become available 2025-08-26T20:09:41.0103297Z  sleep 300 2025-08-26T20:09:41.0103466Z  fi 2025-08-26T20:09:41.0103613Z done 2025-08-26T20:09:41.0103762Z  2025-08-26T20:09:41.0103992Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-08-26T20:09:41.0104430Z # be empty. The default action would be to continue rebuild the image 2025-08-26T20:09:41.0104753Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-08-26T20:09:41.0105034Z  # if we're on the base branch then use the parent commit 2025-08-26T20:09:41.0105292Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-08-26T20:09:41.0105497Z else 2025-08-26T20:09:41.0105712Z  # otherwise we're on a PR, so use the most recent base commit 2025-08-26T20:09:41.0106004Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-08-26T20:09:41.0106231Z fi 2025-08-26T20:09:41.0106382Z  2025-08-26T20:09:41.0106546Z if [[ -z "${MERGE_BASE}" ]]; then 2025-08-26T20:09:41.0106782Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:41.0106994Z  2025-08-26T20:09:41.0107289Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-08-26T20:09:41.0107618Z  exit 0 2025-08-26T20:09:41.0107773Z fi 2025-08-26T20:09:41.0107910Z  2025-08-26T20:09:41.0108116Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-08-26T20:09:41.0108549Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-08-26T20:09:41.0108941Z  exit 1 2025-08-26T20:09:41.0109089Z fi 2025-08-26T20:09:41.0109237Z  2025-08-26T20:09:41.0109474Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-08-26T20:09:41.0109873Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-08-26T20:09:41.0110231Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-08-26T20:09:41.0110639Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-08-26T20:09:41.0111147Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-08-26T20:09:41.0111426Z fi 2025-08-26T20:09:41.0111570Z  2025-08-26T20:09:41.0111747Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-26T20:09:41.0116769Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:41.0117036Z env: 2025-08-26T20:09:41.0117204Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:41.0117406Z DOCKER_BUILD_DIR: .ci/docker 2025-08-26T20:09:41.0117647Z BASE_REVISION: 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:09:41.0118274Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.0119072Z DOCKER_TAG: pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.0119942Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:41.0120226Z DOCKER_PUSH: 2025-08-26T20:09:41.0120397Z ##[endgroup] 2025-08-26T20:09:41.0144408Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:41.0144766Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:41.0145075Z + aws ecr get-login-password --region us-east-1 2025-08-26T20:09:41.0149914Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:41.4571409Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-26T20:09:41.4571900Z Configure a credential helper to remove this warning. See 2025-08-26T20:09:41.4572337Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-26T20:09:41.4572598Z 2025-08-26T20:09:41.4572732Z Login Succeeded 2025-08-26T20:09:41.4592512Z ++ date +%s 2025-08-26T20:09:41.4604188Z + START_TIME=1756238981 2025-08-26T20:09:41.4608215Z ++ date +%s 2025-08-26T20:09:41.4617272Z + [[ 1756231781 -lt 1756238981 ]] 2025-08-26T20:09:41.4622460Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:41.6692039Z { 2025-08-26T20:09:41.6692466Z "schemaVersion": 2, 2025-08-26T20:09:41.6699557Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-08-26T20:09:41.6699925Z "config": { 2025-08-26T20:09:41.6700191Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-08-26T20:09:41.6700684Z "size": 30011, 2025-08-26T20:09:41.6700998Z "digest": "sha256:932da535a977ab0b0008738bb22f52295068c34f7c3dedec486c588f4545c297" 2025-08-26T20:09:41.6701467Z }, 2025-08-26T20:09:41.6702169Z "layers": [ 2025-08-26T20:09:41.6702733Z { 2025-08-26T20:09:41.6703109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6703430Z "size": 30448173, 2025-08-26T20:09:41.6703795Z "digest": "sha256:660ffc76f83b006444a5731b215acc2e35138d8be5cac8ed1ffd40f947117495" 2025-08-26T20:09:41.6704113Z }, 2025-08-26T20:09:41.6704256Z { 2025-08-26T20:09:41.6704539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6704835Z "size": 1553, 2025-08-26T20:09:41.6705133Z "digest": "sha256:4d54b123ef9dd9869d9af411fedf1d5b1db8e69f8333d323cc767ea27f368335" 2025-08-26T20:09:41.6705444Z }, 2025-08-26T20:09:41.6705588Z { 2025-08-26T20:09:41.6705822Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6706105Z "size": 314206045, 2025-08-26T20:09:41.6706399Z "digest": "sha256:9c55cec78f8406fa713d2fc186d15d0715894b44b0f44c5405e0b0063a3f0b35" 2025-08-26T20:09:41.6706714Z }, 2025-08-26T20:09:41.6706851Z { 2025-08-26T20:09:41.6707082Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6707358Z "size": 791, 2025-08-26T20:09:41.6707656Z "digest": "sha256:9bc7d79f3fbb3f75440f59f706b65c619d51400f6713a7401ffda0dcaf302bb9" 2025-08-26T20:09:41.6708349Z }, 2025-08-26T20:09:41.6708499Z { 2025-08-26T20:09:41.6708732Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6709021Z "size": 106, 2025-08-26T20:09:41.6709320Z "digest": "sha256:aafdf104ff63b060f0439c67965849e39ee9201be5223ae1f47d0ebb26c3b325" 2025-08-26T20:09:41.6709677Z }, 2025-08-26T20:09:41.6710021Z { 2025-08-26T20:09:41.6710309Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6710592Z "size": 703, 2025-08-26T20:09:41.6710872Z "digest": "sha256:82c8895e8a78ab52a51c545848284071a28cc32bea40e6fdfb13397379736b4a" 2025-08-26T20:09:41.6711176Z }, 2025-08-26T20:09:41.6711378Z { 2025-08-26T20:09:41.6711604Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6711878Z "size": 1218, 2025-08-26T20:09:41.6712166Z "digest": "sha256:0f8cef21ed4173eec4d4181571521b7800b00cf43ce0ca11e9c9fe88d35f579a" 2025-08-26T20:09:41.6712496Z }, 2025-08-26T20:09:41.6712631Z { 2025-08-26T20:09:41.6712854Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6713127Z "size": 485, 2025-08-26T20:09:41.6713406Z "digest": "sha256:23a46894eb0acedf8723b47155c8b47190e9518f68cdacc3b98cbf44adb6d567" 2025-08-26T20:09:41.6713730Z }, 2025-08-26T20:09:41.6713872Z { 2025-08-26T20:09:41.6714106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6714382Z "size": 110343865, 2025-08-26T20:09:41.6714678Z "digest": "sha256:71763536b7fe8d33e2a4a8ca5d37e9398128db4909b9df72878dfaa11e5fc01c" 2025-08-26T20:09:41.6715003Z }, 2025-08-26T20:09:41.6715145Z { 2025-08-26T20:09:41.6715372Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6715643Z "size": 4788, 2025-08-26T20:09:41.6715938Z "digest": "sha256:2d20cafec487317b3f84d0dafee9ba5dfac753ffd39b33da488c28bbe5f49e11" 2025-08-26T20:09:41.6716387Z }, 2025-08-26T20:09:41.6716533Z { 2025-08-26T20:09:41.6716758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6717042Z "size": 1709, 2025-08-26T20:09:41.6717324Z "digest": "sha256:24343f5bbf5c9b05649870203050524a23cf8203b02ed938517d81427cfba66f" 2025-08-26T20:09:41.6717654Z }, 2025-08-26T20:09:41.6717790Z { 2025-08-26T20:09:41.6718022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6718303Z "size": 724, 2025-08-26T20:09:41.6718598Z "digest": "sha256:13e40e4a2eab659a96b75462cbf452eb8f13dccf5ab0ff09302a846bd5543ddf" 2025-08-26T20:09:41.6718929Z }, 2025-08-26T20:09:41.6719076Z { 2025-08-26T20:09:41.6719383Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6719670Z "size": 544, 2025-08-26T20:09:41.6719949Z "digest": "sha256:f6116ea55ac16326cfa791b3966b4545df025e8766eff08811885d86b5c7aed3" 2025-08-26T20:09:41.6720260Z }, 2025-08-26T20:09:41.6720410Z { 2025-08-26T20:09:41.6720646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6720923Z "size": 3392839691, 2025-08-26T20:09:41.6721229Z "digest": "sha256:734ff9c3d9c1b471933cb23484266c74d0f6f14bca5d0ea678a2af748168df48" 2025-08-26T20:09:41.6721548Z }, 2025-08-26T20:09:41.6721693Z { 2025-08-26T20:09:41.6721909Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6722184Z "size": 32, 2025-08-26T20:09:41.6722470Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6722785Z }, 2025-08-26T20:09:41.6722923Z { 2025-08-26T20:09:41.6723154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6723435Z "size": 381, 2025-08-26T20:09:41.6723827Z "digest": "sha256:7962f79fd05ce7dabec3601db761338db0aa6ef16a4f3ad13782cc78cd70c652" 2025-08-26T20:09:41.6724140Z }, 2025-08-26T20:09:41.6724287Z { 2025-08-26T20:09:41.6724584Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6724882Z "size": 234845, 2025-08-26T20:09:41.6725161Z "digest": "sha256:38d0e1be877ec4bf812a7be88f0661a8cf934c1f242a73a9542b2a280865a029" 2025-08-26T20:09:41.6725472Z }, 2025-08-26T20:09:41.6725614Z { 2025-08-26T20:09:41.6725837Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6726102Z "size": 230, 2025-08-26T20:09:41.6726377Z "digest": "sha256:950bd866dd7860d8ac6c5ffbb1782f2690054713f28d3b709037fc9730da03d9" 2025-08-26T20:09:41.6726678Z }, 2025-08-26T20:09:41.6726816Z { 2025-08-26T20:09:41.6727032Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6727303Z "size": 3301737, 2025-08-26T20:09:41.6727580Z "digest": "sha256:6795befe89abaf06f5eb31de15bd2bc27124f975105cdd05fc224a19b76ff5bb" 2025-08-26T20:09:41.6727873Z }, 2025-08-26T20:09:41.6727999Z { 2025-08-26T20:09:41.6728213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6728472Z "size": 1480, 2025-08-26T20:09:41.6728736Z "digest": "sha256:e7d56d45e721154b379ff640d26cec682314d71fc8e21022770cab7dd3cc5e19" 2025-08-26T20:09:41.6729021Z }, 2025-08-26T20:09:41.6729162Z { 2025-08-26T20:09:41.6729386Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6729657Z "size": 483, 2025-08-26T20:09:41.6729888Z + exit 0 2025-08-26T20:09:41.6730152Z "digest": "sha256:01f006747b5d902996f805ec7d7170a04598825aa7a4b06bc6fa385505089121" 2025-08-26T20:09:41.6730452Z }, 2025-08-26T20:09:41.6730595Z { 2025-08-26T20:09:41.6730813Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6731094Z "size": 195, 2025-08-26T20:09:41.6731385Z "digest": "sha256:c507b40a0297efab4e0a38e9d52ea7e201c791f03465ba75bd79181fd63775f6" 2025-08-26T20:09:41.6731715Z }, 2025-08-26T20:09:41.6731847Z { 2025-08-26T20:09:41.6732132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6732407Z "size": 608, 2025-08-26T20:09:41.6732691Z "digest": "sha256:4e1b4afdbeeaeb2217f375fae67bcfe5c7b3c92b11212829a9948928fef1692a" 2025-08-26T20:09:41.6733000Z }, 2025-08-26T20:09:41.6733136Z { 2025-08-26T20:09:41.6733357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6733630Z "size": 226, 2025-08-26T20:09:41.6733895Z "digest": "sha256:b64445d67ed16d1080d0742a1888d3edf0470bf3b6435f2425be095f835627c6" 2025-08-26T20:09:41.6734199Z }, 2025-08-26T20:09:41.6734346Z { 2025-08-26T20:09:41.6734569Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6734834Z "size": 802, 2025-08-26T20:09:41.6735104Z "digest": "sha256:f48ac6c01ba8292699169c761c26346d25917821734ff02aaf7c2307c86e25d9" 2025-08-26T20:09:41.6735404Z }, 2025-08-26T20:09:41.6735546Z { 2025-08-26T20:09:41.6735768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6736049Z "size": 32, 2025-08-26T20:09:41.6736334Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6736643Z }, 2025-08-26T20:09:41.6736783Z { 2025-08-26T20:09:41.6736996Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6737268Z "size": 104, 2025-08-26T20:09:41.6737541Z "digest": "sha256:b9fb5bb581578935aca59a4e9209cea159b83ed497fc2767d20bce961704000e" 2025-08-26T20:09:41.6737849Z }, 2025-08-26T20:09:41.6737983Z { 2025-08-26T20:09:41.6738211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6738492Z "size": 1495, 2025-08-26T20:09:41.6738775Z "digest": "sha256:f593871a9673982aa4794d09514195b314a62efd7826816cd11cfba859714f73" 2025-08-26T20:09:41.6739067Z }, 2025-08-26T20:09:41.6739206Z { 2025-08-26T20:09:41.6739431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6739704Z "size": 453909723, 2025-08-26T20:09:41.6740058Z "digest": "sha256:7320abfeca74acff3f57d30078a8f75bbf3d898e92fe4d03c460cbfe4a527ce4" 2025-08-26T20:09:41.6742658Z }, 2025-08-26T20:09:41.6742816Z { 2025-08-26T20:09:41.6743047Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6743316Z "size": 163, 2025-08-26T20:09:41.6743591Z "digest": "sha256:d35702d52345ea7ce1043851059af4f0164587a2a1a4c37f88a55e555fed8d3f" 2025-08-26T20:09:41.6743905Z }, 2025-08-26T20:09:41.6744054Z { 2025-08-26T20:09:41.6744279Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6744557Z "size": 347, 2025-08-26T20:09:41.6744852Z "digest": "sha256:1e9e58db0ee671d3308e0545371b2ae914cd3d0e3bcb65abd24b0a530a9ba448" 2025-08-26T20:09:41.6745174Z }, 2025-08-26T20:09:41.6745314Z { 2025-08-26T20:09:41.6745542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6745825Z "size": 32, 2025-08-26T20:09:41.6746122Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6746441Z }, 2025-08-26T20:09:41.6746589Z { 2025-08-26T20:09:41.6746825Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6747111Z "size": 106, 2025-08-26T20:09:41.6747383Z "digest": "sha256:425283ca33f47a6819851b20b812539935074ebbc15ed9a9b38f3143731def23" 2025-08-26T20:09:41.6747696Z }, 2025-08-26T20:09:41.6747843Z { 2025-08-26T20:09:41.6748080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6748355Z "size": 425, 2025-08-26T20:09:41.6748640Z "digest": "sha256:009019a99c9c70bcc68db6f65668ecbebe198cb561981de0e7116306dad6ec03" 2025-08-26T20:09:41.6748956Z }, 2025-08-26T20:09:41.6749106Z { 2025-08-26T20:09:41.6749327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6749615Z "size": 19309417, 2025-08-26T20:09:41.6749916Z "digest": "sha256:2fd6dfbf16970d024d4e40c7a670f9af439856d8e4827b53d52cdd9f87b429e0" 2025-08-26T20:09:41.6750371Z }, 2025-08-26T20:09:41.6750515Z { 2025-08-26T20:09:41.6750745Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6751031Z "size": 108, 2025-08-26T20:09:41.6751382Z "digest": "sha256:fd1110e8b53efb8dfaf7a4acba2ade582165d9b0570acf3aa720832ebc05fac6" 2025-08-26T20:09:41.6751715Z }, 2025-08-26T20:09:41.6751862Z { 2025-08-26T20:09:41.6752092Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6752369Z "size": 639, 2025-08-26T20:09:41.6752649Z "digest": "sha256:fbdbed5f255002792fbfdd3e4e23f56286e327484c5260d4cbf23eb5990df3fe" 2025-08-26T20:09:41.6752969Z }, 2025-08-26T20:09:41.6753113Z { 2025-08-26T20:09:41.6753341Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6753614Z "size": 724, 2025-08-26T20:09:41.6753901Z "digest": "sha256:13e40e4a2eab659a96b75462cbf452eb8f13dccf5ab0ff09302a846bd5543ddf" 2025-08-26T20:09:41.6754228Z }, 2025-08-26T20:09:41.6754375Z { 2025-08-26T20:09:41.6754595Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6754878Z "size": 148, 2025-08-26T20:09:41.6755165Z "digest": "sha256:e9de589665efac6d8d64d93edb0b5b6d50aee6ad8f4003ca0be710092c87cc22" 2025-08-26T20:09:41.6755493Z }, 2025-08-26T20:09:41.6755633Z { 2025-08-26T20:09:41.6755859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6756142Z "size": 134, 2025-08-26T20:09:41.6756428Z "digest": "sha256:43d965cdf5c886269294d9335906d2ab6be1b7344af01bd2d7501ca5aba4e745" 2025-08-26T20:09:41.6756747Z }, 2025-08-26T20:09:41.6756897Z { 2025-08-26T20:09:41.6757130Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6757410Z "size": 139, 2025-08-26T20:09:41.6757690Z "digest": "sha256:c87e96cbf6b3e41fdea9351f65572cfe5b09619ace9f85e44ad6dd06fe147dea" 2025-08-26T20:09:41.6758013Z }, 2025-08-26T20:09:41.6758158Z { 2025-08-26T20:09:41.6758499Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6758781Z "size": 18559842793, 2025-08-26T20:09:41.6759100Z "digest": "sha256:6cefe6cfb1b53e946c0c89339815f540b13eab3d0f50daf25c8f1a61fa50cee1" 2025-08-26T20:09:41.6759926Z }, 2025-08-26T20:09:41.6760131Z { 2025-08-26T20:09:41.6760481Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6760943Z "size": 223, 2025-08-26T20:09:41.6761404Z "digest": "sha256:479676d38bf240769a82c4cbae1a1051bf293efb8f1277e5569dea568c94d1f0" 2025-08-26T20:09:41.6761764Z }, 2025-08-26T20:09:41.6761903Z { 2025-08-26T20:09:41.6762134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6762423Z "size": 274477481, 2025-08-26T20:09:41.6762724Z "digest": "sha256:67c58b456bc9996bc671445140efba403a9d5d3e03536bbaf2c98137b44631b8" 2025-08-26T20:09:41.6763036Z }, 2025-08-26T20:09:41.6763186Z { 2025-08-26T20:09:41.6763423Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6763711Z "size": 6440372376, 2025-08-26T20:09:41.6764001Z "digest": "sha256:308a33031a2e65123173cbf992a394f8fba0a5f3af0905f7ad1577e27bebf819" 2025-08-26T20:09:41.6764319Z }, 2025-08-26T20:09:41.6764465Z { 2025-08-26T20:09:41.6764697Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6764976Z "size": 129, 2025-08-26T20:09:41.6765263Z "digest": "sha256:3e9dbc0952173a3527715a472ca0596fb0b78bc933de0e39f9d054ec8bf22a46" 2025-08-26T20:09:41.6765580Z }, 2025-08-26T20:09:41.6765723Z { 2025-08-26T20:09:41.6765943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6766220Z "size": 777, 2025-08-26T20:09:41.6766509Z "digest": "sha256:469773c071e58c1493f13afaf6fb5fe2d2d578c1ddea126135316db3cb5162ff" 2025-08-26T20:09:41.6766829Z }, 2025-08-26T20:09:41.6766967Z { 2025-08-26T20:09:41.6767192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6767589Z "size": 724, 2025-08-26T20:09:41.6767890Z "digest": "sha256:13e40e4a2eab659a96b75462cbf452eb8f13dccf5ab0ff09302a846bd5543ddf" 2025-08-26T20:09:41.6768205Z }, 2025-08-26T20:09:41.6768356Z { 2025-08-26T20:09:41.6768587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6768874Z "size": 140, 2025-08-26T20:09:41.6769161Z "digest": "sha256:ec16a4f2a115c65620d19bf0f3c6b30e7bda76c7bab0e0419d92c471e0494910" 2025-08-26T20:09:41.6769493Z }, 2025-08-26T20:09:41.6769637Z { 2025-08-26T20:09:41.6769870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6770148Z "size": 32, 2025-08-26T20:09:41.6770440Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6770761Z }, 2025-08-26T20:09:41.6770904Z { 2025-08-26T20:09:41.6771122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6771405Z "size": 158, 2025-08-26T20:09:41.6771678Z "digest": "sha256:01968fc264193aa85381c10fa67b9c79ce6214b60f43705715cb6f83d9e5dc64" 2025-08-26T20:09:41.6772008Z }, 2025-08-26T20:09:41.6772142Z { 2025-08-26T20:09:41.6772376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6772651Z "size": 1010, 2025-08-26T20:09:41.6772937Z "digest": "sha256:1b890f6e6c9420a8709d391e62b53cda920b40f251889194b24172ba0dd83927" 2025-08-26T20:09:41.6773260Z }, 2025-08-26T20:09:41.6773410Z { 2025-08-26T20:09:41.6773644Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6773926Z "size": 724, 2025-08-26T20:09:41.6774211Z "digest": "sha256:13e40e4a2eab659a96b75462cbf452eb8f13dccf5ab0ff09302a846bd5543ddf" 2025-08-26T20:09:41.6774554Z }, 2025-08-26T20:09:41.6774698Z { 2025-08-26T20:09:41.6774928Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6775199Z "size": 135, 2025-08-26T20:09:41.6775562Z "digest": "sha256:f8db46864fc761918b649267a6a00baeee4f34a0eea9cea5b750685c66b17bdb" 2025-08-26T20:09:41.6775885Z }, 2025-08-26T20:09:41.6776032Z { 2025-08-26T20:09:41.6776251Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6776527Z "size": 32, 2025-08-26T20:09:41.6776819Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6777149Z }, 2025-08-26T20:09:41.6777283Z { 2025-08-26T20:09:41.6777510Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6777791Z "size": 157, 2025-08-26T20:09:41.6778071Z "digest": "sha256:cdf1ab518f852096bfe97d860fd98a61713a73bb65739554cebfe8299d130fd1" 2025-08-26T20:09:41.6778392Z }, 2025-08-26T20:09:41.6778529Z { 2025-08-26T20:09:41.6778755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6779032Z "size": 1369, 2025-08-26T20:09:41.6779314Z "digest": "sha256:0061302d04951a0b439d4dfbdfa03f500877faf187f3940a655b065d53b1fc94" 2025-08-26T20:09:41.6779625Z }, 2025-08-26T20:09:41.6779765Z { 2025-08-26T20:09:41.6779989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6780258Z "size": 32, 2025-08-26T20:09:41.6780548Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6780888Z }, 2025-08-26T20:09:41.6781042Z { 2025-08-26T20:09:41.6781272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6781557Z "size": 136, 2025-08-26T20:09:41.6781844Z "digest": "sha256:7efd21765c9f354e41a6820f89addc3ce5a16766a203740a389ac5e9ce80d6fc" 2025-08-26T20:09:41.6782161Z }, 2025-08-26T20:09:41.6782298Z { 2025-08-26T20:09:41.6782528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6782810Z "size": 380, 2025-08-26T20:09:41.6783098Z "digest": "sha256:47eb26c12113434e1c0cf8da7273e9040511b9884eccf4a3dcbea2d5249a023d" 2025-08-26T20:09:41.6783480Z }, 2025-08-26T20:09:41.6783624Z { 2025-08-26T20:09:41.6783853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6784134Z "size": 32, 2025-08-26T20:09:41.6784423Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6784735Z }, 2025-08-26T20:09:41.6784879Z { 2025-08-26T20:09:41.6785108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6785390Z "size": 104, 2025-08-26T20:09:41.6785669Z "digest": "sha256:27c4d270ca26645b622a0d54fe5017b1e9885c1fc8798cc40732d4cd3520b3d7" 2025-08-26T20:09:41.6786081Z }, 2025-08-26T20:09:41.6786228Z { 2025-08-26T20:09:41.6786451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6786741Z "size": 407, 2025-08-26T20:09:41.6787036Z "digest": "sha256:4bc6ef31bd523602cfd4f80f47fe9cf84cf74986fce29a393c69a2008ee3c78f" 2025-08-26T20:09:41.6787358Z }, 2025-08-26T20:09:41.6787504Z { 2025-08-26T20:09:41.6787727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6788005Z "size": 32, 2025-08-26T20:09:41.6788283Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6788595Z }, 2025-08-26T20:09:41.6788729Z { 2025-08-26T20:09:41.6788950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6789234Z "size": 109, 2025-08-26T20:09:41.6789524Z "digest": "sha256:5ce18bf5b45dbcad8afcd0fde76d5d0a674d53e510d99cceff0587e6b948e52e" 2025-08-26T20:09:41.6789840Z }, 2025-08-26T20:09:41.6789978Z { 2025-08-26T20:09:41.6790199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6790483Z "size": 1896, 2025-08-26T20:09:41.6790766Z "digest": "sha256:285c81af4b9d0b1429be21e8a67cdb35d3b2ce9652ba5bd3e7b4fe003b1da902" 2025-08-26T20:09:41.6791086Z }, 2025-08-26T20:09:41.6791227Z { 2025-08-26T20:09:41.6791517Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6791787Z "size": 242997127, 2025-08-26T20:09:41.6792084Z "digest": "sha256:bb07dd0d561f2f9bd5c780aefc22a73058368931be47e4e3af1a539c4d267ae2" 2025-08-26T20:09:41.6792396Z }, 2025-08-26T20:09:41.6792537Z { 2025-08-26T20:09:41.6792752Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6793027Z "size": 106, 2025-08-26T20:09:41.6793315Z "digest": "sha256:50a5a3b68ee3894de8898a04c6a4bfdd0ac1859da95b5a8ceab7ccb6a9a620da" 2025-08-26T20:09:41.6793628Z }, 2025-08-26T20:09:41.6793763Z { 2025-08-26T20:09:41.6793987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6794259Z "size": 164, 2025-08-26T20:09:41.6794536Z "digest": "sha256:79eecebd063f6be16d85070bb148f969679d1d8df9c0dee228e3595118fcd6f0" 2025-08-26T20:09:41.6794835Z }, 2025-08-26T20:09:41.6794976Z { 2025-08-26T20:09:41.6795206Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6795484Z "size": 7943, 2025-08-26T20:09:41.6795756Z "digest": "sha256:1d221336b6cfad31c20882c181f430a4282a12a9be9028b5bd8280aadca44c42" 2025-08-26T20:09:41.6796061Z }, 2025-08-26T20:09:41.6796461Z { 2025-08-26T20:09:41.6796707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6796985Z "size": 8077, 2025-08-26T20:09:41.6797289Z "digest": "sha256:bb9b1ab32155a02a0f4cd73062f3e3f649e281a8439037f7b94cc6df74dc4fba" 2025-08-26T20:09:41.6797597Z }, 2025-08-26T20:09:41.6797739Z { 2025-08-26T20:09:41.6797961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6798248Z "size": 303, 2025-08-26T20:09:41.6798539Z "digest": "sha256:0867b7fa637cd840cab46ab9d0f60674db13df921cabda8599405045604f5b2d" 2025-08-26T20:09:41.6798849Z }, 2025-08-26T20:09:41.6798986Z { 2025-08-26T20:09:41.6799317Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6799908Z "size": 32, 2025-08-26T20:09:41.6800390Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6800858Z }, 2025-08-26T20:09:41.6801052Z { 2025-08-26T20:09:41.6801398Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6801889Z "size": 108, 2025-08-26T20:09:41.6802317Z "digest": "sha256:0edddbef40dd990542474976dcfe8db6dca1fee30e5d623ff3c442b93de5a701" 2025-08-26T20:09:41.6802740Z }, 2025-08-26T20:09:41.6802883Z { 2025-08-26T20:09:41.6803108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6803377Z "size": 54145663, 2025-08-26T20:09:41.6803673Z "digest": "sha256:bf05e37245a2b1f993edf59c7d829c700ba42dde0686edb2dc0b990fe7a97573" 2025-08-26T20:09:41.6803981Z }, 2025-08-26T20:09:41.6804123Z { 2025-08-26T20:09:41.6804338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-26T20:09:41.6804611Z "size": 32, 2025-08-26T20:09:41.6804907Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-26T20:09:41.6805217Z } 2025-08-26T20:09:41.6805354Z ] 2025-08-26T20:09:41.6805501Z } 2025-08-26T20:09:41.6834971Z ##[group]Run set -eux 2025-08-26T20:09:41.6835186Z set -eux 2025-08-26T20:09:41.6835469Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-08-26T20:09:41.6836210Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-08-26T20:09:41.6842816Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:41.6843058Z env: 2025-08-26T20:09:41.6843216Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:41.6843399Z ##[endgroup] 2025-08-26T20:09:41.6869973Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-08-26T20:09:41.6870839Z + jq -r .docker_hub_readonly_token 2025-08-26T20:09:41.6871094Z + jq --raw-output .SecretString 2025-08-26T20:09:41.6873936Z + docker login --username pytorchbot --password-stdin 2025-08-26T20:09:42.1805815Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-26T20:09:42.1806227Z Login Succeeded 2025-08-26T20:09:42.1809487Z Configure a credential helper to remove this warning. See 2025-08-26T20:09:42.1809916Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-26T20:09:42.1810176Z 2025-08-26T20:09:42.1884847Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-08-26T20:09:42.1885128Z tag=${ECR_DOCKER_IMAGE##*:} 2025-08-26T20:09:42.1885408Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-08-26T20:09:42.1890011Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:42.1890267Z env: 2025-08-26T20:09:42.1890434Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:42.1891000Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:42.1891567Z ##[endgroup] 2025-08-26T20:09:42.1914946Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:42.1956642Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-08-26T20:09:42.1956997Z with: 2025-08-26T20:09:42.1957678Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:42.1958467Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:42.1958792Z env: 2025-08-26T20:09:42.1958986Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:42.1959604Z ##[endgroup] 2025-08-26T20:09:42.1975042Z ##[group]Run set -x 2025-08-26T20:09:42.1975285Z set -x 2025-08-26T20:09:42.1975495Z set +e 2025-08-26T20:09:42.1975682Z  2025-08-26T20:09:42.1975861Z login() { 2025-08-26T20:09:42.1976248Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-26T20:09:42.1976630Z } 2025-08-26T20:09:42.1976805Z  2025-08-26T20:09:42.1977008Z retry () { 2025-08-26T20:09:42.1977229Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-26T20:09:42.1977466Z } 2025-08-26T20:09:42.1977635Z  2025-08-26T20:09:42.1977822Z retry login "${DOCKER_REGISTRY}" 2025-08-26T20:09:42.1978058Z  2025-08-26T20:09:42.1978411Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-08-26T20:09:42.1978891Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-08-26T20:09:42.1979171Z  2025-08-26T20:09:42.1979339Z set -e 2025-08-26T20:09:42.1979608Z # ignore output since only exit code is used for conditional 2025-08-26T20:09:42.1979970Z # only pull docker image if it's not available locally 2025-08-26T20:09:42.1980368Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-08-26T20:09:42.1980749Z  retry docker pull "${DOCKER_IMAGE}" 2025-08-26T20:09:42.1980995Z fi 2025-08-26T20:09:42.1985406Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:09:42.1985673Z env: 2025-08-26T20:09:42.1985851Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:09:42.1986416Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:42.1987075Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:42.1987337Z ##[endgroup] 2025-08-26T20:09:42.2008781Z + set +e 2025-08-26T20:09:42.2009214Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:42.2009907Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:42.2012235Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-26T20:09:42.2012800Z + aws ecr get-login-password --region us-east-1 2025-08-26T20:09:42.6398455Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-26T20:09:42.6398927Z Configure a credential helper to remove this warning. See 2025-08-26T20:09:42.6399451Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-26T20:09:42.6399716Z 2025-08-26T20:09:42.6400246Z Login Succeeded 2025-08-26T20:09:42.6426376Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:42.6427121Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-08-26T20:09:42.8913330Z + IMAGE_SIZE=28511.53002166748 2025-08-26T20:09:42.8913645Z Compressed size of image in MB: 28511.53002166748 2025-08-26T20:09:42.8914120Z + echo 'Compressed size of image in MB: 28511.53002166748' 2025-08-26T20:09:42.8915031Z + set -e 2025-08-26T20:09:42.8916167Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:42.9030530Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:42.9031872Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:09:43.1882295Z pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6: Pulling from pytorch/ci-image 2025-08-26T20:09:43.1883012Z 660ffc76f83b: Pulling fs layer 2025-08-26T20:09:43.1883257Z 4d54b123ef9d: Pulling fs layer 2025-08-26T20:09:43.1883466Z 9c55cec78f84: Pulling fs layer 2025-08-26T20:09:43.1883665Z 9bc7d79f3fbb: Pulling fs layer 2025-08-26T20:09:43.1883877Z aafdf104ff63: Pulling fs layer 2025-08-26T20:09:43.1884076Z 82c8895e8a78: Pulling fs layer 2025-08-26T20:09:43.1884294Z 0f8cef21ed41: Pulling fs layer 2025-08-26T20:09:43.1884481Z 23a46894eb0a: Pulling fs layer 2025-08-26T20:09:43.1884678Z 71763536b7fe: Pulling fs layer 2025-08-26T20:09:43.1884881Z 2d20cafec487: Pulling fs layer 2025-08-26T20:09:43.1885081Z 24343f5bbf5c: Pulling fs layer 2025-08-26T20:09:43.1885274Z 13e40e4a2eab: Pulling fs layer 2025-08-26T20:09:43.1885472Z f6116ea55ac1: Pulling fs layer 2025-08-26T20:09:43.1885669Z aafdf104ff63: Waiting 2025-08-26T20:09:43.1885860Z 734ff9c3d9c1: Pulling fs layer 2025-08-26T20:09:43.1886046Z 82c8895e8a78: Waiting 2025-08-26T20:09:43.1886229Z 4f4fb700ef54: Pulling fs layer 2025-08-26T20:09:43.1886427Z 7962f79fd05c: Pulling fs layer 2025-08-26T20:09:43.1886628Z 38d0e1be877e: Pulling fs layer 2025-08-26T20:09:43.1886815Z 950bd866dd78: Pulling fs layer 2025-08-26T20:09:43.1887014Z 6795befe89ab: Pulling fs layer 2025-08-26T20:09:43.1887227Z e7d56d45e721: Pulling fs layer 2025-08-26T20:09:43.1887428Z 01f006747b5d: Pulling fs layer 2025-08-26T20:09:43.1887615Z c507b40a0297: Pulling fs layer 2025-08-26T20:09:43.1887906Z 4e1b4afdbeea: Pulling fs layer 2025-08-26T20:09:43.1888102Z b64445d67ed1: Pulling fs layer 2025-08-26T20:09:43.1888297Z f48ac6c01ba8: Pulling fs layer 2025-08-26T20:09:43.1888489Z b9fb5bb58157: Pulling fs layer 2025-08-26T20:09:43.1888686Z f593871a9673: Pulling fs layer 2025-08-26T20:09:43.1888879Z 7320abfeca74: Pulling fs layer 2025-08-26T20:09:43.1889074Z d35702d52345: Pulling fs layer 2025-08-26T20:09:43.1889252Z 0f8cef21ed41: Waiting 2025-08-26T20:09:43.1889436Z 1e9e58db0ee6: Pulling fs layer 2025-08-26T20:09:43.1889632Z 425283ca33f4: Pulling fs layer 2025-08-26T20:09:43.1889853Z 23a46894eb0a: Waiting 2025-08-26T20:09:43.1890314Z 009019a99c9c: Pulling fs layer 2025-08-26T20:09:43.1890503Z 2fd6dfbf1697: Pulling fs layer 2025-08-26T20:09:43.1890697Z 71763536b7fe: Waiting 2025-08-26T20:09:43.1890876Z fd1110e8b53e: Pulling fs layer 2025-08-26T20:09:43.1891074Z 2d20cafec487: Waiting 2025-08-26T20:09:43.1891272Z fbdbed5f2550: Pulling fs layer 2025-08-26T20:09:43.1891481Z e9de589665ef: Pulling fs layer 2025-08-26T20:09:43.1891676Z 43d965cdf5c8: Pulling fs layer 2025-08-26T20:09:43.1891877Z c87e96cbf6b3: Pulling fs layer 2025-08-26T20:09:43.1892062Z 9bc7d79f3fbb: Waiting 2025-08-26T20:09:43.1892248Z 6cefe6cfb1b5: Pulling fs layer 2025-08-26T20:09:43.1892440Z 425283ca33f4: Waiting 2025-08-26T20:09:43.1892614Z 6795befe89ab: Waiting 2025-08-26T20:09:43.1892786Z 479676d38bf2: Pulling fs layer 2025-08-26T20:09:43.1892977Z 009019a99c9c: Waiting 2025-08-26T20:09:43.1893156Z 67c58b456bc9: Pulling fs layer 2025-08-26T20:09:43.1893362Z e7d56d45e721: Waiting 2025-08-26T20:09:43.1893528Z d35702d52345: Waiting 2025-08-26T20:09:43.1893706Z 308a33031a2e: Pulling fs layer 2025-08-26T20:09:43.1893892Z 01f006747b5d: Waiting 2025-08-26T20:09:43.1894065Z 3e9dbc095217: Pulling fs layer 2025-08-26T20:09:43.1894249Z 24343f5bbf5c: Waiting 2025-08-26T20:09:43.1894424Z 469773c071e5: Pulling fs layer 2025-08-26T20:09:43.1894611Z 1e9e58db0ee6: Waiting 2025-08-26T20:09:43.1894907Z ec16a4f2a115: Pulling fs layer 2025-08-26T20:09:43.1895096Z f6116ea55ac1: Waiting 2025-08-26T20:09:43.1895267Z 13e40e4a2eab: Waiting 2025-08-26T20:09:43.1895444Z 734ff9c3d9c1: Waiting 2025-08-26T20:09:43.1895614Z 4f4fb700ef54: Waiting 2025-08-26T20:09:43.1895775Z c507b40a0297: Waiting 2025-08-26T20:09:43.1895944Z 7962f79fd05c: Waiting 2025-08-26T20:09:43.1896117Z 4e1b4afdbeea: Waiting 2025-08-26T20:09:43.1896551Z 01968fc26419: Pulling fs layer 2025-08-26T20:09:43.1896746Z 1b890f6e6c94: Pulling fs layer 2025-08-26T20:09:43.1896937Z 2fd6dfbf1697: Waiting 2025-08-26T20:09:43.1897117Z f8db46864fc7: Pulling fs layer 2025-08-26T20:09:43.1897299Z fd1110e8b53e: Waiting 2025-08-26T20:09:43.1897472Z 950bd866dd78: Waiting 2025-08-26T20:09:43.1897642Z fbdbed5f2550: Waiting 2025-08-26T20:09:43.1897814Z 38d0e1be877e: Waiting 2025-08-26T20:09:43.1897977Z e9de589665ef: Waiting 2025-08-26T20:09:43.1898155Z cdf1ab518f85: Pulling fs layer 2025-08-26T20:09:43.1898344Z b64445d67ed1: Waiting 2025-08-26T20:09:43.1898519Z 0061302d0495: Pulling fs layer 2025-08-26T20:09:43.1898698Z 43d965cdf5c8: Waiting 2025-08-26T20:09:43.1898868Z c87e96cbf6b3: Waiting 2025-08-26T20:09:43.1899045Z 7efd21765c9f: Pulling fs layer 2025-08-26T20:09:43.1899238Z 47eb26c12113: Pulling fs layer 2025-08-26T20:09:43.1899421Z 7320abfeca74: Waiting 2025-08-26T20:09:43.1899588Z f593871a9673: Waiting 2025-08-26T20:09:43.1899754Z 479676d38bf2: Waiting 2025-08-26T20:09:43.1899928Z 27c4d270ca26: Pulling fs layer 2025-08-26T20:09:43.1900116Z 4bc6ef31bd52: Pulling fs layer 2025-08-26T20:09:43.1900305Z 1b890f6e6c94: Waiting 2025-08-26T20:09:43.1900481Z 5ce18bf5b45d: Pulling fs layer 2025-08-26T20:09:43.1900676Z 285c81af4b9d: Pulling fs layer 2025-08-26T20:09:43.1900860Z 67c58b456bc9: Waiting 2025-08-26T20:09:43.1901031Z 308a33031a2e: Waiting 2025-08-26T20:09:43.1901201Z 6cefe6cfb1b5: Waiting 2025-08-26T20:09:43.1901380Z bb07dd0d561f: Pulling fs layer 2025-08-26T20:09:43.1901568Z 50a5a3b68ee3: Pulling fs layer 2025-08-26T20:09:43.1901756Z cdf1ab518f85: Waiting 2025-08-26T20:09:43.1901933Z f8db46864fc7: Waiting 2025-08-26T20:09:43.1902114Z 79eecebd063f: Pulling fs layer 2025-08-26T20:09:43.1902300Z 0061302d0495: Waiting 2025-08-26T20:09:43.1902479Z 1d221336b6cf: Pulling fs layer 2025-08-26T20:09:43.1902693Z bb9b1ab32155: Pulling fs layer 2025-08-26T20:09:43.1902888Z 0867b7fa637c: Pulling fs layer 2025-08-26T20:09:43.1903067Z 7efd21765c9f: Waiting 2025-08-26T20:09:43.1903240Z 47eb26c12113: Waiting 2025-08-26T20:09:43.1903425Z 0edddbef40dd: Pulling fs layer 2025-08-26T20:09:43.1903618Z 27c4d270ca26: Waiting 2025-08-26T20:09:43.1903792Z bf05e37245a2: Pulling fs layer 2025-08-26T20:09:43.1903985Z 5ce18bf5b45d: Waiting 2025-08-26T20:09:43.1904253Z 3e9dbc095217: Waiting 2025-08-26T20:09:43.1904424Z 469773c071e5: Waiting 2025-08-26T20:09:43.1904589Z 0867b7fa637c: Waiting 2025-08-26T20:09:43.1904764Z 0edddbef40dd: Waiting 2025-08-26T20:09:43.1904940Z 285c81af4b9d: Waiting 2025-08-26T20:09:43.1905106Z bf05e37245a2: Waiting 2025-08-26T20:09:43.1905289Z 50a5a3b68ee3: Waiting 2025-08-26T20:09:43.1905463Z bb07dd0d561f: Waiting 2025-08-26T20:09:43.1905634Z 79eecebd063f: Waiting 2025-08-26T20:09:43.1905797Z bb9b1ab32155: Waiting 2025-08-26T20:09:43.1905968Z ec16a4f2a115: Waiting 2025-08-26T20:09:43.1906138Z f48ac6c01ba8: Waiting 2025-08-26T20:09:43.1906305Z 4bc6ef31bd52: Waiting 2025-08-26T20:09:43.1906474Z 01968fc26419: Waiting 2025-08-26T20:09:43.1906641Z 1d221336b6cf: Waiting 2025-08-26T20:09:43.1906809Z b9fb5bb58157: Waiting 2025-08-26T20:09:43.2863326Z 4d54b123ef9d: Verifying Checksum 2025-08-26T20:09:43.2864781Z 4d54b123ef9d: Download complete 2025-08-26T20:09:43.3787698Z 9bc7d79f3fbb: Download complete 2025-08-26T20:09:43.4490313Z aafdf104ff63: Verifying Checksum 2025-08-26T20:09:43.4490673Z aafdf104ff63: Download complete 2025-08-26T20:09:43.5472808Z 82c8895e8a78: Download complete 2025-08-26T20:09:43.5674215Z 660ffc76f83b: Verifying Checksum 2025-08-26T20:09:43.5674538Z 660ffc76f83b: Download complete 2025-08-26T20:09:43.6299126Z 0f8cef21ed41: Verifying Checksum 2025-08-26T20:09:43.6299693Z 0f8cef21ed41: Download complete 2025-08-26T20:09:43.6784490Z 23a46894eb0a: Verifying Checksum 2025-08-26T20:09:43.6784997Z 23a46894eb0a: Download complete 2025-08-26T20:09:43.7550135Z 2d20cafec487: Verifying Checksum 2025-08-26T20:09:43.7550494Z 2d20cafec487: Download complete 2025-08-26T20:09:43.8303586Z 24343f5bbf5c: Download complete 2025-08-26T20:09:43.9065825Z 13e40e4a2eab: Verifying Checksum 2025-08-26T20:09:43.9066161Z 13e40e4a2eab: Download complete 2025-08-26T20:09:43.9749947Z f6116ea55ac1: Verifying Checksum 2025-08-26T20:09:43.9750206Z f6116ea55ac1: Download complete 2025-08-26T20:09:44.7586551Z 660ffc76f83b: Pull complete 2025-08-26T20:09:44.7718929Z 4d54b123ef9d: Pull complete 2025-08-26T20:09:44.7972413Z 71763536b7fe: Download complete 2025-08-26T20:09:44.8062207Z 4f4fb700ef54: Verifying Checksum 2025-08-26T20:09:44.8062681Z 4f4fb700ef54: Download complete 2025-08-26T20:09:44.9133251Z 7962f79fd05c: Verifying Checksum 2025-08-26T20:09:44.9133555Z 7962f79fd05c: Download complete 2025-08-26T20:09:45.0085117Z 38d0e1be877e: Verifying Checksum 2025-08-26T20:09:45.0085453Z 38d0e1be877e: Download complete 2025-08-26T20:09:45.0982338Z 950bd866dd78: Verifying Checksum 2025-08-26T20:09:45.0984571Z 950bd866dd78: Download complete 2025-08-26T20:09:45.2093397Z 6795befe89ab: Verifying Checksum 2025-08-26T20:09:45.2093909Z 6795befe89ab: Download complete 2025-08-26T20:09:45.2878399Z e7d56d45e721: Verifying Checksum 2025-08-26T20:09:45.2878717Z e7d56d45e721: Download complete 2025-08-26T20:09:45.3787794Z 01f006747b5d: Verifying Checksum 2025-08-26T20:09:45.3788350Z 01f006747b5d: Download complete 2025-08-26T20:09:45.4580310Z c507b40a0297: Verifying Checksum 2025-08-26T20:09:45.4586894Z c507b40a0297: Download complete 2025-08-26T20:09:45.5278680Z 4e1b4afdbeea: Download complete 2025-08-26T20:09:45.6140353Z b64445d67ed1: Download complete 2025-08-26T20:09:45.7231226Z f48ac6c01ba8: Verifying Checksum 2025-08-26T20:09:45.7231544Z f48ac6c01ba8: Download complete 2025-08-26T20:09:45.8470471Z b9fb5bb58157: Verifying Checksum 2025-08-26T20:09:45.8470894Z b9fb5bb58157: Download complete 2025-08-26T20:09:46.4081060Z 9c55cec78f84: Download complete 2025-08-26T20:09:46.4904412Z d35702d52345: Verifying Checksum 2025-08-26T20:09:46.4904710Z d35702d52345: Download complete 2025-08-26T20:09:46.5686608Z 1e9e58db0ee6: Verifying Checksum 2025-08-26T20:09:46.5686913Z 1e9e58db0ee6: Download complete 2025-08-26T20:09:46.6476039Z 425283ca33f4: Verifying Checksum 2025-08-26T20:09:46.6478877Z 425283ca33f4: Download complete 2025-08-26T20:09:46.7305640Z 009019a99c9c: Verifying Checksum 2025-08-26T20:09:46.7309404Z 009019a99c9c: Download complete 2025-08-26T20:09:46.9838902Z 2fd6dfbf1697: Verifying Checksum 2025-08-26T20:09:46.9839597Z 2fd6dfbf1697: Download complete 2025-08-26T20:09:47.0593288Z fd1110e8b53e: Download complete 2025-08-26T20:09:47.1418223Z fbdbed5f2550: Download complete 2025-08-26T20:09:47.2383151Z e9de589665ef: Download complete 2025-08-26T20:09:47.3111572Z 43d965cdf5c8: Verifying Checksum 2025-08-26T20:09:47.3112086Z 43d965cdf5c8: Download complete 2025-08-26T20:09:47.3831253Z c87e96cbf6b3: Verifying Checksum 2025-08-26T20:09:47.3831579Z c87e96cbf6b3: Download complete 2025-08-26T20:09:50.5148228Z 7320abfeca74: Verifying Checksum 2025-08-26T20:09:50.5148551Z 7320abfeca74: Download complete 2025-08-26T20:09:50.5911630Z 479676d38bf2: Verifying Checksum 2025-08-26T20:09:50.5912118Z 479676d38bf2: Download complete 2025-08-26T20:09:53.3878399Z 67c58b456bc9: Verifying Checksum 2025-08-26T20:09:53.3878718Z 67c58b456bc9: Download complete 2025-08-26T20:09:57.8728977Z 9c55cec78f84: Pull complete 2025-08-26T20:09:58.1828255Z 9bc7d79f3fbb: Pull complete 2025-08-26T20:09:58.4982949Z aafdf104ff63: Pull complete 2025-08-26T20:09:58.9287292Z 82c8895e8a78: Pull complete 2025-08-26T20:09:59.2554637Z 0f8cef21ed41: Pull complete 2025-08-26T20:09:59.5154877Z 23a46894eb0a: Pull complete 2025-08-26T20:10:03.0946058Z 71763536b7fe: Pull complete 2025-08-26T20:10:03.3713354Z 2d20cafec487: Pull complete 2025-08-26T20:10:03.6757794Z 24343f5bbf5c: Pull complete 2025-08-26T20:10:03.9528216Z 13e40e4a2eab: Pull complete 2025-08-26T20:10:04.1223041Z f6116ea55ac1: Pull complete 2025-08-26T20:10:17.9638400Z 734ff9c3d9c1: Verifying Checksum 2025-08-26T20:10:17.9638913Z 734ff9c3d9c1: Download complete 2025-08-26T20:10:18.0487921Z 3e9dbc095217: Verifying Checksum 2025-08-26T20:10:18.0491978Z 3e9dbc095217: Download complete 2025-08-26T20:10:18.1540190Z 469773c071e5: Verifying Checksum 2025-08-26T20:10:18.1540650Z 469773c071e5: Download complete 2025-08-26T20:10:18.2421979Z ec16a4f2a115: Verifying Checksum 2025-08-26T20:10:18.2422473Z ec16a4f2a115: Download complete 2025-08-26T20:10:18.3176961Z 01968fc26419: Verifying Checksum 2025-08-26T20:10:18.3183063Z 01968fc26419: Download complete 2025-08-26T20:10:18.4019965Z 1b890f6e6c94: Verifying Checksum 2025-08-26T20:10:18.4020493Z 1b890f6e6c94: Download complete 2025-08-26T20:10:18.4667231Z f8db46864fc7: Verifying Checksum 2025-08-26T20:10:18.4667532Z f8db46864fc7: Download complete 2025-08-26T20:10:18.5524419Z cdf1ab518f85: Download complete 2025-08-26T20:10:18.6619488Z 0061302d0495: Verifying Checksum 2025-08-26T20:10:18.6619816Z 0061302d0495: Download complete 2025-08-26T20:10:18.6983163Z 7efd21765c9f: Verifying Checksum 2025-08-26T20:10:18.6983484Z 7efd21765c9f: Download complete 2025-08-26T20:10:18.7556124Z 47eb26c12113: Download complete 2025-08-26T20:10:18.8332080Z 27c4d270ca26: Download complete 2025-08-26T20:10:18.9216038Z 4bc6ef31bd52: Download complete 2025-08-26T20:10:18.9785874Z 5ce18bf5b45d: Verifying Checksum 2025-08-26T20:10:18.9786397Z 5ce18bf5b45d: Download complete 2025-08-26T20:10:19.0675438Z 285c81af4b9d: Download complete 2025-08-26T20:10:21.5561722Z bb07dd0d561f: Verifying Checksum 2025-08-26T20:10:21.5562054Z bb07dd0d561f: Download complete 2025-08-26T20:10:21.6317912Z 50a5a3b68ee3: Verifying Checksum 2025-08-26T20:10:21.6318220Z 50a5a3b68ee3: Download complete 2025-08-26T20:10:21.7208001Z 79eecebd063f: Verifying Checksum 2025-08-26T20:10:21.7208513Z 79eecebd063f: Download complete 2025-08-26T20:10:21.8138228Z 1d221336b6cf: Verifying Checksum 2025-08-26T20:10:21.8138532Z 1d221336b6cf: Download complete 2025-08-26T20:10:21.8899437Z bb9b1ab32155: Verifying Checksum 2025-08-26T20:10:21.8905082Z bb9b1ab32155: Download complete 2025-08-26T20:10:21.9678806Z 0867b7fa637c: Verifying Checksum 2025-08-26T20:10:21.9679185Z 0867b7fa637c: Download complete 2025-08-26T20:10:22.0505203Z 0edddbef40dd: Verifying Checksum 2025-08-26T20:10:22.0505734Z 0edddbef40dd: Download complete 2025-08-26T20:10:22.6480140Z bf05e37245a2: Verifying Checksum 2025-08-26T20:10:22.6480679Z bf05e37245a2: Download complete 2025-08-26T20:10:57.8358581Z 308a33031a2e: Verifying Checksum 2025-08-26T20:10:57.8359456Z 308a33031a2e: Download complete 2025-08-26T20:11:36.0090883Z 734ff9c3d9c1: Pull complete 2025-08-26T20:11:36.2571558Z 4f4fb700ef54: Pull complete 2025-08-26T20:11:36.4345121Z 7962f79fd05c: Pull complete 2025-08-26T20:11:36.6616594Z 38d0e1be877e: Pull complete 2025-08-26T20:11:36.8547115Z 950bd866dd78: Pull complete 2025-08-26T20:11:37.2053511Z 6795befe89ab: Pull complete 2025-08-26T20:11:37.3968961Z e7d56d45e721: Pull complete 2025-08-26T20:11:37.6953706Z 01f006747b5d: Pull complete 2025-08-26T20:11:37.9871815Z c507b40a0297: Pull complete 2025-08-26T20:11:38.3304872Z 4e1b4afdbeea: Pull complete 2025-08-26T20:11:38.8320212Z b64445d67ed1: Pull complete 2025-08-26T20:11:39.2172661Z f48ac6c01ba8: Pull complete 2025-08-26T20:11:39.8380711Z b9fb5bb58157: Pull complete 2025-08-26T20:11:40.1878669Z f593871a9673: Pull complete 2025-08-26T20:11:51.5421012Z 7320abfeca74: Pull complete 2025-08-26T20:11:51.7274088Z d35702d52345: Pull complete 2025-08-26T20:11:51.9999051Z 1e9e58db0ee6: Pull complete 2025-08-26T20:11:52.9390043Z 425283ca33f4: Pull complete 2025-08-26T20:11:53.2853557Z 009019a99c9c: Pull complete 2025-08-26T20:11:54.1638959Z 2fd6dfbf1697: Pull complete 2025-08-26T20:11:54.6986017Z fd1110e8b53e: Pull complete 2025-08-26T20:11:55.1071009Z fbdbed5f2550: Pull complete 2025-08-26T20:11:55.8555847Z e9de589665ef: Pull complete 2025-08-26T20:11:56.2455694Z 43d965cdf5c8: Pull complete 2025-08-26T20:11:56.5801764Z c87e96cbf6b3: Pull complete 2025-08-26T20:12:53.0325339Z 6cefe6cfb1b5: Verifying Checksum 2025-08-26T20:12:53.0330831Z 6cefe6cfb1b5: Download complete 2025-08-26T20:17:01.6702199Z 6cefe6cfb1b5: Pull complete 2025-08-26T20:17:02.0289262Z 479676d38bf2: Pull complete 2025-08-26T20:17:04.6360881Z 67c58b456bc9: Pull complete 2025-08-26T20:19:31.7078122Z 308a33031a2e: Pull complete 2025-08-26T20:19:31.7344440Z 3e9dbc095217: Pull complete 2025-08-26T20:19:31.7621398Z 469773c071e5: Pull complete 2025-08-26T20:19:31.8166033Z ec16a4f2a115: Pull complete 2025-08-26T20:19:31.8695697Z 01968fc26419: Pull complete 2025-08-26T20:19:31.8954369Z 1b890f6e6c94: Pull complete 2025-08-26T20:19:31.9509837Z f8db46864fc7: Pull complete 2025-08-26T20:19:32.0059279Z cdf1ab518f85: Pull complete 2025-08-26T20:19:32.0303922Z 0061302d0495: Pull complete 2025-08-26T20:19:32.0747004Z 7efd21765c9f: Pull complete 2025-08-26T20:19:32.0991775Z 47eb26c12113: Pull complete 2025-08-26T20:19:32.1512094Z 27c4d270ca26: Pull complete 2025-08-26T20:19:32.1774505Z 4bc6ef31bd52: Pull complete 2025-08-26T20:19:32.2289948Z 5ce18bf5b45d: Pull complete 2025-08-26T20:19:32.2520309Z 285c81af4b9d: Pull complete 2025-08-26T20:19:41.6220294Z bb07dd0d561f: Pull complete 2025-08-26T20:19:42.0990787Z 50a5a3b68ee3: Pull complete 2025-08-26T20:19:42.5797618Z 79eecebd063f: Pull complete 2025-08-26T20:19:42.9403965Z 1d221336b6cf: Pull complete 2025-08-26T20:19:43.3991255Z bb9b1ab32155: Pull complete 2025-08-26T20:19:43.8835984Z 0867b7fa637c: Pull complete 2025-08-26T20:19:44.6042943Z 0edddbef40dd: Pull complete 2025-08-26T20:19:47.2497951Z bf05e37245a2: Pull complete 2025-08-26T20:19:47.9564154Z Digest: sha256:acbbd4ce4ca5911beba428e48e3c25069f341e6f142804bf943d333ccc654c8c 2025-08-26T20:19:48.0622654Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:19:48.0950197Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:19:48.1005905Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-26T20:19:48.1006541Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-26T20:19:48.1015467Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:19:48.1015721Z env: 2025-08-26T20:19:48.1015886Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:19:48.1016354Z ##[endgroup] 2025-08-26T20:19:48.1092423Z Prepare all required actions 2025-08-26T20:19:48.1343301Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-08-26T20:19:48.1343555Z with: 2025-08-26T20:19:48.1344189Z github-token: *** 2025-08-26T20:19:48.1344352Z env: 2025-08-26T20:19:48.1344511Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:19:48.1344697Z ##[endgroup] 2025-08-26T20:19:48.1488200Z ##[group]Run set -eux 2025-08-26T20:19:48.1488572Z set -eux 2025-08-26T20:19:48.1488919Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-26T20:19:48.1493820Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:19:48.1494087Z env: 2025-08-26T20:19:48.1494260Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:19:48.1494700Z GITHUB_TOKEN: *** 2025-08-26T20:19:48.1494886Z ##[endgroup] 2025-08-26T20:19:48.1519718Z + python3 .github/scripts/get_workflow_job_id.py 17248463670 i-04c468ba96b53884f 2025-08-26T20:19:49.1527673Z Setting output job-id=48946862580 2025-08-26T20:19:49.1528666Z Setting output job-name=linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:19:49.1724871Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-08-26T20:19:49.1725340Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-08-26T20:19:49.1725920Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-08-26T20:19:49.1726439Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:19:49.1731014Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:19:49.1731269Z env: 2025-08-26T20:19:49.1731431Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:19:49.1731612Z JOB_ID: 48946862580 2025-08-26T20:19:49.1731960Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:19:49.1732365Z WORKFLOW_NAME: inductor 2025-08-26T20:19:49.1732547Z WORKFLOW_RUN_ID: 17248463670 2025-08-26T20:19:49.1732765Z MONITOR_LOG_INTERVAL: 5 2025-08-26T20:19:49.1732948Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-08-26T20:19:49.1733150Z ##[endgroup] 2025-08-26T20:19:49.7120794Z Defaulting to user installation because normal site-packages is not writeable 2025-08-26T20:19:50.0391422Z Collecting psutil==5.9.8 2025-08-26T20:19:50.0544419Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-08-26T20:19:50.1988317Z Collecting dataclasses_json==0.6.7 2025-08-26T20:19:50.2022624Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-08-26T20:19:50.2568380Z Collecting nvidia-ml-py==11.525.84 2025-08-26T20:19:50.2599667Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-08-26T20:19:50.3352781Z Collecting typing-inspect<1,>=0.4.0 2025-08-26T20:19:50.3384984Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-08-26T20:19:50.5010720Z Collecting marshmallow<4.0.0,>=3.18.0 2025-08-26T20:19:50.5047632Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-08-26T20:19:50.5976800Z Collecting packaging>=17.0 2025-08-26T20:19:50.6020910Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-08-26T20:19:50.7128549Z Collecting typing-extensions>=3.7.4 2025-08-26T20:19:50.7156931Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-08-26T20:19:50.8437794Z Collecting mypy-extensions>=0.3.0 2025-08-26T20:19:50.8477239Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-08-26T20:19:51.1460301Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-08-26T20:19:51.8380150Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-08-26T20:19:52.1118026Z Prepare all required actions 2025-08-26T20:19:52.1118374Z Getting action download info 2025-08-26T20:19:52.2613317Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-08-26T20:19:52.7963799Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-08-26T20:19:55.2242359Z ##[group]Run ./.github/actions/download-build-artifacts 2025-08-26T20:19:55.2242613Z with: 2025-08-26T20:19:55.2242793Z name: linux-jammy-py3.9-gcc11-build 2025-08-26T20:19:55.2243003Z s3-bucket: gha-artifacts 2025-08-26T20:19:55.2243188Z env: 2025-08-26T20:19:55.2243345Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:19:55.2243522Z ##[endgroup] 2025-08-26T20:19:55.2400615Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-26T20:19:55.2400873Z with: 2025-08-26T20:19:55.2401068Z name: linux-jammy-py3.9-gcc11-build 2025-08-26T20:19:55.2401332Z s3-bucket: gha-artifacts 2025-08-26T20:19:55.2401582Z region: us-east-1 2025-08-26T20:19:55.2401748Z env: 2025-08-26T20:19:55.2401910Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:19:55.2402098Z ##[endgroup] 2025-08-26T20:19:56.0248153Z (node:48611) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-26T20:19:56.0248794Z 2025-08-26T20:19:56.0249055Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-26T20:19:56.0249453Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-26T20:19:56.0249893Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-26T20:19:57.1430613Z Found 1 objects with prefix pytorch/pytorch/17248463670/linux-jammy-py3.9-gcc11-build/ 2025-08-26T20:19:57.1431350Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-26T20:20:01.7948869Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-26T20:20:01.7954718Z Artifact download has finished successfully 2025-08-26T20:20:01.8147139Z ##[group]Run unzip -o artifacts.zip 2025-08-26T20:20:01.8147394Z unzip -o artifacts.zip 2025-08-26T20:20:01.8152569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:01.8152839Z env: 2025-08-26T20:20:01.8153008Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:01.8153200Z ##[endgroup] 2025-08-26T20:20:01.8230215Z Archive: artifacts.zip 2025-08-26T20:20:01.8234549Z creating: dist/ 2025-08-26T20:20:02.9475975Z inflating: dist/torch-2.9.0a0+git262640f-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:02.9476612Z creating: dist/vision/ 2025-08-26T20:20:02.9556373Z inflating: dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:02.9557364Z creating: dist/audio/ 2025-08-26T20:20:02.9588262Z inflating: dist/audio/torchaudio-2.8.0a0+10a5002-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:02.9590725Z creating: dist/ao/ 2025-08-26T20:20:02.9630617Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-08-26T20:20:02.9747115Z inflating: dist/.ninja_log 2025-08-26T20:20:02.9752936Z creating: build/custom_test_artifacts/ 2025-08-26T20:20:02.9755635Z creating: build/custom_test_artifacts/custom-op-build/ 2025-08-26T20:20:02.9756042Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-08-26T20:20:02.9756460Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-08-26T20:20:02.9756933Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-26T20:20:02.9757381Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/ 2025-08-26T20:20:02.9757820Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-26T20:20:02.9758284Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-26T20:20:02.9759514Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-26T20:20:02.9760050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-26T20:20:02.9760579Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-26T20:20:02.9761070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-26T20:20:02.9761517Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-26T20:20:02.9761979Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-26T20:20:02.9762473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-26T20:20:02.9762972Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-26T20:20:02.9763437Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-26T20:20:02.9763924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-26T20:20:02.9764450Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-26T20:20:02.9764915Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-08-26T20:20:02.9765312Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-08-26T20:20:02.9765716Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-08-26T20:20:02.9766148Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-08-26T20:20:02.9766645Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-08-26T20:20:02.9767131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-08-26T20:20:02.9767569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-08-26T20:20:02.9768019Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-08-26T20:20:02.9768472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-08-26T20:20:02.9768960Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-08-26T20:20:02.9769407Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-08-26T20:20:02.9769882Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-08-26T20:20:02.9784744Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-08-26T20:20:02.9957371Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-08-26T20:20:02.9958021Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-08-26T20:20:02.9958562Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-08-26T20:20:02.9959144Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-08-26T20:20:02.9960102Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-08-26T20:20:02.9960701Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-08-26T20:20:02.9961265Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-08-26T20:20:02.9961796Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-08-26T20:20:02.9962794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-08-26T20:20:02.9963333Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-08-26T20:20:02.9963847Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-08-26T20:20:02.9980622Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-08-26T20:20:03.0056455Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-08-26T20:20:03.0057081Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-26T20:20:03.0057602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-08-26T20:20:03.0058072Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-08-26T20:20:03.0058705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-08-26T20:20:03.0059624Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-08-26T20:20:03.0060157Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json 2025-08-26T20:20:03.0060619Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-08-26T20:20:03.0061008Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-08-26T20:20:03.0061395Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-08-26T20:20:03.0220720Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-08-26T20:20:03.0270635Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-08-26T20:20:03.0275188Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-08-26T20:20:03.0275614Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-08-26T20:20:03.0276150Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-08-26T20:20:03.0276618Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-26T20:20:03.0277067Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/ 2025-08-26T20:20:03.0277502Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-26T20:20:03.0277961Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-26T20:20:03.0278407Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-26T20:20:03.0278913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-26T20:20:03.0279742Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-26T20:20:03.0280248Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-26T20:20:03.0280785Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-26T20:20:03.0281241Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-26T20:20:03.0281794Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-26T20:20:03.0282378Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-26T20:20:03.0282875Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-26T20:20:03.0283390Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-26T20:20:03.0288612Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-26T20:20:03.0292341Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-08-26T20:20:03.0293463Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-08-26T20:20:03.0293928Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-08-26T20:20:03.0294458Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-08-26T20:20:03.0295078Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-08-26T20:20:03.0295617Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-08-26T20:20:03.0296375Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-08-26T20:20:03.0296896Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-08-26T20:20:03.0297418Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-08-26T20:20:03.0297937Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-08-26T20:20:03.0298501Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-08-26T20:20:03.0298998Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-08-26T20:20:03.0308975Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-08-26T20:20:03.0367929Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-08-26T20:20:03.0374975Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-26T20:20:03.0381214Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-08-26T20:20:03.0387867Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-08-26T20:20:03.0393119Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-08-26T20:20:03.0393768Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-08-26T20:20:03.0394264Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json 2025-08-26T20:20:03.0394709Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-08-26T20:20:03.0395096Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-08-26T20:20:03.0395474Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-08-26T20:20:03.0408888Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-08-26T20:20:03.0409368Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-08-26T20:20:03.0409737Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-08-26T20:20:03.0410185Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-08-26T20:20:03.0410689Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-26T20:20:03.0411305Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/ 2025-08-26T20:20:03.0412214Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-26T20:20:03.0412763Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-26T20:20:03.0413257Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-26T20:20:03.0413890Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-26T20:20:03.0416487Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-26T20:20:03.0420172Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-26T20:20:03.0420924Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-26T20:20:03.0421454Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-26T20:20:03.0422020Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-26T20:20:03.0422603Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-26T20:20:03.0423123Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-26T20:20:03.0423678Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-26T20:20:03.0424284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-26T20:20:03.0424815Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-08-26T20:20:03.0425258Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-08-26T20:20:03.0425695Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-08-26T20:20:03.0426231Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-08-26T20:20:03.0426923Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-08-26T20:20:03.0432301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-08-26T20:20:03.0432963Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-08-26T20:20:03.0433527Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-08-26T20:20:03.0434151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-08-26T20:20:03.0434713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-08-26T20:20:03.0435269Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-08-26T20:20:03.0435805Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-08-26T20:20:03.0436422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-08-26T20:20:03.0544180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-08-26T20:20:03.0547765Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-08-26T20:20:03.0553879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-08-26T20:20:03.0554525Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-08-26T20:20:03.0555131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-08-26T20:20:03.0555708Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-08-26T20:20:03.0556318Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-08-26T20:20:03.0556899Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-08-26T20:20:03.0557480Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-08-26T20:20:03.0558483Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-08-26T20:20:03.0559126Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-08-26T20:20:03.0564952Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-08-26T20:20:03.0616303Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-08-26T20:20:03.0621339Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-26T20:20:03.0627005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-08-26T20:20:03.0629209Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-08-26T20:20:03.0629834Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-08-26T20:20:03.0633112Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-08-26T20:20:03.0633842Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json 2025-08-26T20:20:03.0634312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-08-26T20:20:03.0634735Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-08-26T20:20:03.0635141Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-08-26T20:20:03.0711794Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-08-26T20:20:03.0753092Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-08-26T20:20:03.0754857Z creating: build/lib/ 2025-08-26T20:20:03.0828388Z inflating: build/lib/libprotobuf-lite.a 2025-08-26T20:20:03.1249777Z inflating: build/lib/libprotobuf.a 2025-08-26T20:20:03.1709421Z inflating: build/lib/libprotoc.a 2025-08-26T20:20:03.1717631Z inflating: build/lib/libpthreadpool.a 2025-08-26T20:20:03.1728268Z inflating: build/lib/libcpuinfo.a 2025-08-26T20:20:03.1738142Z inflating: build/lib/libcpuinfo_internals.a 2025-08-26T20:20:03.1744628Z inflating: build/lib/libclog.a 2025-08-26T20:20:03.1753928Z inflating: build/lib/libpytorch_qnnpack.a 2025-08-26T20:20:03.1754272Z inflating: build/lib/libnnpack_reference_layers.a 2025-08-26T20:20:03.1932561Z inflating: build/lib/libmicrokernels-prod.a 2025-08-26T20:20:03.1950542Z inflating: build/lib/libnnpack.a 2025-08-26T20:20:03.2768660Z inflating: build/lib/libmicrokernels-all.a 2025-08-26T20:20:03.2833591Z inflating: build/lib/libgtest.a 2025-08-26T20:20:03.2852153Z inflating: build/lib/libgmock.a 2025-08-26T20:20:03.2857850Z inflating: build/lib/libgtest_main.a 2025-08-26T20:20:03.2934996Z inflating: build/lib/libXNNPACK.a 2025-08-26T20:20:03.2935534Z inflating: build/lib/libgmock_main.a 2025-08-26T20:20:03.3006069Z inflating: build/lib/libbenchmark.a 2025-08-26T20:20:03.3006376Z inflating: build/lib/libbenchmark_main.a 2025-08-26T20:20:03.3069759Z inflating: build/lib/libasmjit.a 2025-08-26T20:20:03.3076501Z inflating: build/lib/libittnotify.a 2025-08-26T20:20:03.3076833Z inflating: build/lib/libjitprofiling.a 2025-08-26T20:20:03.4160775Z inflating: build/lib/libfbgemm.a 2025-08-26T20:20:03.4190461Z inflating: build/lib/libtensorpipe_uv.a 2025-08-26T20:20:03.4703696Z inflating: build/lib/libtensorpipe.a 2025-08-26T20:20:03.4822360Z inflating: build/lib/libgloo.a 2025-08-26T20:20:03.4868886Z inflating: build/lib/libonnx_proto.a 2025-08-26T20:20:03.5551563Z inflating: build/lib/libonnx.a 2025-08-26T20:20:04.5045987Z inflating: build/lib/libdnnl.a 2025-08-26T20:20:04.5066152Z inflating: build/lib/libfmt.a 2025-08-26T20:20:04.5318202Z inflating: build/lib/libkineto.a 2025-08-26T20:20:04.5424039Z inflating: build/lib/libc10.so 2025-08-26T20:20:04.5428124Z inflating: build/lib/libtorch_global_deps.so 2025-08-26T20:20:07.3752292Z inflating: build/lib/libtorch_cpu.so 2025-08-26T20:20:07.3752621Z inflating: build/lib/libtorch.so 2025-08-26T20:20:07.3820482Z inflating: build/lib/libtorchbind_test.so 2025-08-26T20:20:07.3835459Z inflating: build/lib/libjitbackend_test.so 2025-08-26T20:20:07.3863164Z inflating: build/lib/libbackend_with_compiler.so 2025-08-26T20:20:07.3885522Z inflating: build/lib/libaoti_custom_ops.so 2025-08-26T20:20:07.3887262Z inflating: build/lib/libshm.so 2025-08-26T20:20:07.5848429Z inflating: build/lib/libtorch_python.so 2025-08-26T20:20:07.5880905Z inflating: build/lib/libnnapi_backend.so 2025-08-26T20:20:07.5881351Z creating: build/bin/ 2025-08-26T20:20:07.5881649Z creating: build/bin/CMakeFiles/ 2025-08-26T20:20:07.5882016Z inflating: build/bin/cmake_install.cmake 2025-08-26T20:20:07.5882366Z inflating: build/bin/CTestTestfile.cmake 2025-08-26T20:20:07.6318950Z inflating: build/bin/protoc-3.13.0.0 2025-08-26T20:20:07.6757319Z inflating: build/bin/protoc 2025-08-26T20:20:07.6813016Z inflating: build/bin/c10_AllocatorConfig_test 2025-08-26T20:20:07.6865422Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-08-26T20:20:07.6924652Z inflating: build/bin/c10_DeviceGuard_test 2025-08-26T20:20:07.6978526Z inflating: build/bin/c10_Device_test 2025-08-26T20:20:07.7046190Z inflating: build/bin/c10_DispatchKeySet_test 2025-08-26T20:20:07.7095438Z inflating: build/bin/c10_StreamGuard_test 2025-08-26T20:20:07.7155825Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-08-26T20:20:07.7210483Z inflating: build/bin/c10_SymInt_test 2025-08-26T20:20:07.7265515Z inflating: build/bin/c10_Scalar_test 2025-08-26T20:20:07.7317491Z inflating: build/bin/c10_ConstexprCrc_test 2025-08-26T20:20:07.7379055Z inflating: build/bin/c10_InlineStreamGuard_test 2025-08-26T20:20:07.7451163Z inflating: build/bin/c10_cow_test 2025-08-26T20:20:07.7508930Z inflating: build/bin/c10_SizesAndStrides_test 2025-08-26T20:20:07.7565007Z inflating: build/bin/c10_Bitset_test 2025-08-26T20:20:07.7616598Z inflating: build/bin/c10_ArrayRef_test 2025-08-26T20:20:07.7671035Z inflating: build/bin/c10_DeadlockDetection_test 2025-08-26T20:20:07.7732471Z inflating: build/bin/c10_Enumerate_test 2025-08-26T20:20:07.7784772Z inflating: build/bin/c10_IntrusiveList_test 2025-08-26T20:20:07.7845421Z inflating: build/bin/c10_LeftRight_test 2025-08-26T20:20:07.7900183Z inflating: build/bin/c10_Half_test 2025-08-26T20:20:07.7960910Z inflating: build/bin/c10_Metaprogramming_test 2025-08-26T20:20:07.8020627Z inflating: build/bin/c10_NetworkFlow_test 2025-08-26T20:20:07.8073654Z inflating: build/bin/c10_Semaphore_test 2025-08-26T20:20:07.8128298Z inflating: build/bin/c10_Synchronized_test 2025-08-26T20:20:07.8186255Z inflating: build/bin/c10_ThreadLocal_test 2025-08-26T20:20:07.8245085Z inflating: build/bin/c10_TypeIndex_test 2025-08-26T20:20:07.8297646Z inflating: build/bin/c10_TypeList_test 2025-08-26T20:20:07.8349209Z inflating: build/bin/c10_TypeTraits_test 2025-08-26T20:20:07.8406704Z inflating: build/bin/c10_accumulate_test 2025-08-26T20:20:07.8463463Z inflating: build/bin/c10_bfloat16_test 2025-08-26T20:20:07.8522350Z inflating: build/bin/c10_complex_math_test 2025-08-26T20:20:07.8577179Z inflating: build/bin/c10_bit_cast_test 2025-08-26T20:20:07.8630599Z inflating: build/bin/c10_error_test 2025-08-26T20:20:07.8691759Z inflating: build/bin/c10_complex_test 2025-08-26T20:20:07.8743378Z inflating: build/bin/c10_exception_test 2025-08-26T20:20:07.8795521Z inflating: build/bin/c10_flags_test 2025-08-26T20:20:07.8848998Z inflating: build/bin/c10_generic_math_test 2025-08-26T20:20:07.8905389Z inflating: build/bin/c10_irange_test 2025-08-26T20:20:07.9067847Z inflating: build/bin/c10_intrusive_ptr_test 2025-08-26T20:20:07.9125576Z inflating: build/bin/c10_lazy_test 2025-08-26T20:20:07.9187323Z inflating: build/bin/c10_logging_test 2025-08-26T20:20:07.9245303Z inflating: build/bin/c10_registry_test 2025-08-26T20:20:07.9317966Z inflating: build/bin/c10_optional_test 2025-08-26T20:20:07.9472079Z inflating: build/bin/c10_small_vector_test 2025-08-26T20:20:07.9537362Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-08-26T20:20:07.9587248Z inflating: build/bin/c10_ssize_test 2025-08-26T20:20:07.9648510Z inflating: build/bin/c10_string_util_test 2025-08-26T20:20:07.9702447Z inflating: build/bin/c10_tempfile_test 2025-08-26T20:20:07.9750837Z inflating: build/bin/c10_string_view_test 2025-08-26T20:20:07.9794918Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-08-26T20:20:07.9859774Z inflating: build/bin/c10_typeid_test 2025-08-26T20:20:08.0427216Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-08-26T20:20:08.1016550Z inflating: build/bin/vec_test_all_types_AVX512 2025-08-26T20:20:08.1619100Z inflating: build/bin/vec_test_all_types_AVX2 2025-08-26T20:20:08.1678092Z inflating: build/bin/static_runtime_bench 2025-08-26T20:20:08.1927154Z inflating: build/bin/static_runtime_test 2025-08-26T20:20:08.2001889Z inflating: build/bin/Dict_test 2025-08-26T20:20:08.2059954Z inflating: build/bin/Dimname_test 2025-08-26T20:20:08.2130374Z inflating: build/bin/MaybeOwned_test 2025-08-26T20:20:08.2186263Z inflating: build/bin/NamedTensor_test 2025-08-26T20:20:08.2246140Z inflating: build/bin/apply_utils_test 2025-08-26T20:20:08.2309402Z inflating: build/bin/atest 2025-08-26T20:20:08.2374802Z inflating: build/bin/basic 2025-08-26T20:20:08.2432606Z inflating: build/bin/broadcast_test 2025-08-26T20:20:08.2484373Z inflating: build/bin/cpu_allocator_test 2025-08-26T20:20:08.2548311Z inflating: build/bin/cpu_generator_test 2025-08-26T20:20:08.2601335Z inflating: build/bin/cpu_profiling_allocator_test 2025-08-26T20:20:08.2698666Z inflating: build/bin/cpu_rng_test 2025-08-26T20:20:08.2754927Z inflating: build/bin/dlconvertor_test 2025-08-26T20:20:08.2817589Z inflating: build/bin/extension_backend_test 2025-08-26T20:20:08.2876309Z inflating: build/bin/half_test 2025-08-26T20:20:08.2976037Z inflating: build/bin/ivalue_test 2025-08-26T20:20:08.3031075Z inflating: build/bin/lazy_tensor_test 2025-08-26T20:20:08.3083916Z inflating: build/bin/math_kernel_test 2025-08-26T20:20:08.3145838Z inflating: build/bin/memory_format_test 2025-08-26T20:20:08.3199037Z inflating: build/bin/memory_overlapping_test 2025-08-26T20:20:08.3255084Z inflating: build/bin/mobile_memory_cleanup 2025-08-26T20:20:08.3314340Z inflating: build/bin/native_test 2025-08-26T20:20:08.3370726Z inflating: build/bin/operator_name_test 2025-08-26T20:20:08.3425954Z inflating: build/bin/operators_test 2025-08-26T20:20:08.3480093Z inflating: build/bin/packedtensoraccessor_test 2025-08-26T20:20:08.3551372Z inflating: build/bin/pow_test 2025-08-26T20:20:08.3615605Z inflating: build/bin/quantized_test 2025-08-26T20:20:08.3672454Z inflating: build/bin/reduce_ops_test 2025-08-26T20:20:08.3725984Z inflating: build/bin/reportMemoryUsage_test 2025-08-26T20:20:08.3788015Z inflating: build/bin/scalar_tensor_test 2025-08-26T20:20:08.3849511Z inflating: build/bin/scalar_test 2025-08-26T20:20:08.3905579Z inflating: build/bin/StorageUtils_test 2025-08-26T20:20:08.3960665Z inflating: build/bin/stride_properties_test 2025-08-26T20:20:08.4042214Z inflating: build/bin/tensor_iterator_test 2025-08-26T20:20:08.4100591Z inflating: build/bin/test_parallel 2025-08-26T20:20:08.4155304Z inflating: build/bin/thread_init_test 2025-08-26T20:20:08.4212245Z inflating: build/bin/type_ptr_test 2025-08-26T20:20:08.4274093Z inflating: build/bin/type_test 2025-08-26T20:20:08.4335400Z inflating: build/bin/undefined_tensor_test 2025-08-26T20:20:08.4383671Z inflating: build/bin/verify_api_visibility 2025-08-26T20:20:08.4460286Z inflating: build/bin/legacy_vmap_test 2025-08-26T20:20:08.4514051Z inflating: build/bin/weakref_test 2025-08-26T20:20:08.4571725Z inflating: build/bin/wrapdim_test 2025-08-26T20:20:08.4622088Z inflating: build/bin/xla_tensor_test 2025-08-26T20:20:08.4686335Z inflating: build/bin/IListRef_test 2025-08-26T20:20:08.4792700Z inflating: build/bin/List_test 2025-08-26T20:20:08.4863187Z inflating: build/bin/KernelFunction_test 2025-08-26T20:20:08.4985245Z inflating: build/bin/kernel_function_legacy_test 2025-08-26T20:20:08.5081575Z inflating: build/bin/kernel_function_test 2025-08-26T20:20:08.5209013Z inflating: build/bin/kernel_lambda_legacy_test 2025-08-26T20:20:08.5312342Z inflating: build/bin/kernel_lambda_test 2025-08-26T20:20:08.5375644Z inflating: build/bin/kernel_stackbased_test 2025-08-26T20:20:08.5471400Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-08-26T20:20:08.5524805Z inflating: build/bin/CppSignature_test 2025-08-26T20:20:08.5582740Z inflating: build/bin/backend_fallback_test 2025-08-26T20:20:08.5635476Z inflating: build/bin/op_allowlist_test 2025-08-26T20:20:08.5941611Z inflating: build/bin/op_registration_test 2025-08-26T20:20:08.6012481Z inflating: build/bin/inline_container_test 2025-08-26T20:20:08.7097581Z inflating: build/bin/test_jit 2025-08-26T20:20:08.7149510Z inflating: build/bin/BackoffTest 2025-08-26T20:20:08.7208742Z inflating: build/bin/FileStoreTest 2025-08-26T20:20:08.7269015Z inflating: build/bin/TCPStoreTest 2025-08-26T20:20:08.7325040Z inflating: build/bin/HashStoreTest 2025-08-26T20:20:08.7697650Z inflating: build/bin/test_nativert 2025-08-26T20:20:08.7699919Z inflating: build/bin/example_allreduce 2025-08-26T20:20:08.7768624Z inflating: build/bin/ProcessGroupGlooTest 2025-08-26T20:20:08.7831867Z inflating: build/bin/test_dist_autograd 2025-08-26T20:20:08.7903697Z inflating: build/bin/test_cpp_rpc 2025-08-26T20:20:08.9012070Z inflating: build/bin/test_api 2025-08-26T20:20:08.9012654Z inflating: build/bin/parallel_benchmark 2025-08-26T20:20:08.9351383Z inflating: build/bin/test_lazy 2025-08-26T20:20:08.9352907Z inflating: build/bin/torch_shm_manager 2025-08-26T20:20:08.9353170Z creating: .additional_ci_files/ 2025-08-26T20:20:08.9428444Z inflating: .additional_ci_files/test-times.json 2025-08-26T20:20:08.9728325Z inflating: .additional_ci_files/test-class-times.json 2025-08-26T20:20:08.9811863Z ##[group]Run rm artifacts.zip 2025-08-26T20:20:08.9812100Z rm artifacts.zip 2025-08-26T20:20:08.9817068Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:08.9817321Z env: 2025-08-26T20:20:08.9817484Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:08.9817668Z ##[endgroup] 2025-08-26T20:20:09.0120358Z ##[group]Run df -H 2025-08-26T20:20:09.0120582Z df -H 2025-08-26T20:20:09.0125560Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:09.0125845Z env: 2025-08-26T20:20:09.0126032Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:09.0126262Z ##[endgroup] 2025-08-26T20:20:09.0171730Z Filesystem Size Used Avail Use% Mounted on 2025-08-26T20:20:09.0172240Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-08-26T20:20:09.0172624Z tmpfs 67G 0 67G 0% /dev/shm 2025-08-26T20:20:09.0172990Z tmpfs 27G 791k 27G 1% /run 2025-08-26T20:20:09.0173780Z /dev/nvme0n1p1 215G 70G 146G 33% / 2025-08-26T20:20:09.0174074Z tmpfs 67G 13k 67G 1% /tmp 2025-08-26T20:20:09.0174453Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-08-26T20:20:09.0198551Z Prepare all required actions 2025-08-26T20:20:09.0199730Z Getting action download info 2025-08-26T20:20:09.1568382Z ##[group]Run ./.github/actions/download-td-artifacts 2025-08-26T20:20:09.1568655Z with: 2025-08-26T20:20:09.1568818Z env: 2025-08-26T20:20:09.1568987Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:09.1569299Z ##[endgroup] 2025-08-26T20:20:09.1660768Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-26T20:20:09.1661026Z with: 2025-08-26T20:20:09.1661188Z name: td_results 2025-08-26T20:20:09.1661373Z s3-bucket: gha-artifacts 2025-08-26T20:20:09.1661571Z region: us-east-1 2025-08-26T20:20:09.1661731Z env: 2025-08-26T20:20:09.1661892Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:09.1662088Z ##[endgroup] 2025-08-26T20:20:09.5187846Z (node:48632) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-26T20:20:09.5190472Z 2025-08-26T20:20:09.5190804Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-26T20:20:09.5191209Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-26T20:20:09.5191660Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-26T20:20:09.6121089Z Found 0 objects with prefix pytorch/pytorch/17248463670/td_results/ 2025-08-26T20:20:09.6126679Z Artifact download has finished successfully 2025-08-26T20:20:09.6535877Z ##[group]Run mkdir -p .additional_ci_files 2025-08-26T20:20:09.6536155Z mkdir -p .additional_ci_files 2025-08-26T20:20:09.6536442Z mv td_results.json .additional_ci_files/td_results.json || true 2025-08-26T20:20:09.6541973Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:09.6542240Z env: 2025-08-26T20:20:09.6542448Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:09.6542634Z ##[endgroup] 2025-08-26T20:20:09.6595008Z mv: cannot stat 'td_results.json': No such file or directory 2025-08-26T20:20:09.6866897Z ##[group]Run .github/scripts/parse_ref.py 2025-08-26T20:20:09.6867169Z .github/scripts/parse_ref.py 2025-08-26T20:20:09.6871810Z shell: /usr/bin/bash -e {0} 2025-08-26T20:20:09.6872001Z env: 2025-08-26T20:20:09.6872167Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:09.6872346Z ##[endgroup] 2025-08-26T20:20:09.7748134Z Setting output branch=main 2025-08-26T20:20:09.7844625Z Prepare all required actions 2025-08-26T20:20:09.7845017Z Getting action download info 2025-08-26T20:20:09.9338179Z ##[group]Run ./.github/actions/filter-test-configs 2025-08-26T20:20:09.9338434Z with: 2025-08-26T20:20:09.9338914Z github-token: *** 2025-08-26T20:20:09.9340517Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-08-26T20:20:09.9342325Z job-name: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:20:09.9342691Z env: 2025-08-26T20:20:09.9342847Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:09.9343022Z ##[endgroup] 2025-08-26T20:20:09.9487208Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-26T20:20:09.9487429Z with: 2025-08-26T20:20:09.9487580Z shell: bash 2025-08-26T20:20:09.9487732Z timeout_minutes: 10 2025-08-26T20:20:09.9487898Z max_attempts: 5 2025-08-26T20:20:09.9488061Z retry_wait_seconds: 30 2025-08-26T20:20:09.9488564Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-26T20:20:09.9489073Z polling_interval_seconds: 1 2025-08-26T20:20:09.9489292Z warning_on_retry: true 2025-08-26T20:20:09.9489603Z continue_on_error: false 2025-08-26T20:20:09.9489789Z env: 2025-08-26T20:20:09.9489944Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:09.9490394Z GITHUB_TOKEN: *** 2025-08-26T20:20:09.9490571Z ##[endgroup] 2025-08-26T20:20:10.1336595Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-26T20:20:10.3227688Z Defaulting to user installation because normal site-packages is not writeable 2025-08-26T20:20:11.2655914Z Collecting requests==2.27.1 2025-08-26T20:20:11.2807453Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-08-26T20:20:11.4992520Z Collecting pyyaml==6.0.2 2025-08-26T20:20:11.5028550Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-08-26T20:20:11.5671313Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-08-26T20:20:11.6496399Z Collecting certifi>=2017.4.17 2025-08-26T20:20:11.6541114Z Downloading certifi-2025.8.3-py3-none-any.whl (161 kB) 2025-08-26T20:20:11.9831292Z Collecting charset-normalizer~=2.0.0 2025-08-26T20:20:11.9866232Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-08-26T20:20:12.0396543Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-08-26T20:20:12.1046476Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-08-26T20:20:12.5752323Z Successfully installed certifi-2025.8.3 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-08-26T20:20:13.0147602Z Command completed after 1 attempt(s). 2025-08-26T20:20:13.0207555Z ##[group]Run set -x 2025-08-26T20:20:13.0207772Z set -x 2025-08-26T20:20:13.0207948Z  2025-08-26T20:20:13.0208224Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-26T20:20:13.0208557Z # in runner workspace 2025-08-26T20:20:13.0208998Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-08-26T20:20:13.0214315Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:13.0214566Z env: 2025-08-26T20:20:13.0214719Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:13.0214905Z ##[endgroup] 2025-08-26T20:20:13.0236595Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-08-26T20:20:13.0384886Z Setting output branch=main 2025-08-26T20:20:13.0445766Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-26T20:20:13.0446042Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-26T20:20:13.0446258Z echo "Job name: ${JOB_NAME}" 2025-08-26T20:20:13.0446445Z  2025-08-26T20:20:13.0446682Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-26T20:20:13.0446973Z # in runner workspace 2025-08-26T20:20:13.0447264Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-08-26T20:20:13.0447563Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-08-26T20:20:13.0447795Z  --job-name "${JOB_NAME}" \ 2025-08-26T20:20:13.0449390Z  --test-matrix "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" \ 2025-08-26T20:20:13.0451112Z  --selected-test-configs "" \ 2025-08-26T20:20:13.0451333Z  --pr-number "${PR_NUMBER}" \ 2025-08-26T20:20:13.0451533Z  --tag "${TAG}" \ 2025-08-26T20:20:13.0451729Z  --event-name "${EVENT_NAME}" \ 2025-08-26T20:20:13.0451939Z  --schedule "${SCHEDULE}" \ 2025-08-26T20:20:13.0452142Z  --branch "${HEAD_BRANCH}" 2025-08-26T20:20:13.0456792Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:13.0457053Z env: 2025-08-26T20:20:13.0457227Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:13.0457790Z GITHUB_TOKEN: *** 2025-08-26T20:20:13.0458188Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:20:13.0458571Z PR_NUMBER: 2025-08-26T20:20:13.0458749Z TAG: 2025-08-26T20:20:13.0458903Z EVENT_NAME: push 2025-08-26T20:20:13.0459069Z SCHEDULE: 2025-08-26T20:20:13.0459230Z HEAD_BRANCH: main 2025-08-26T20:20:13.0459404Z ##[endgroup] 2025-08-26T20:20:13.0480336Z Workflow: inductor 2025-08-26T20:20:13.0480815Z Job name: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:20:13.1994402Z Setting output keep-going=True 2025-08-26T20:20:13.1994820Z Setting output ci-verbose-test-logs=False 2025-08-26T20:20:13.1995097Z Setting output ci-test-showlocals=False 2025-08-26T20:20:13.1995340Z Setting output ci-no-test-timeout=False 2025-08-26T20:20:13.1995583Z Setting output ci-no-td=False 2025-08-26T20:20:13.1995821Z Setting output ci-td-distributed=False 2025-08-26T20:20:13.1996068Z Setting output is-unstable=False 2025-08-26T20:20:13.1996481Z Setting output reenabled-issues= 2025-08-26T20:20:13.1998788Z Setting output test-matrix={"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-08-26T20:20:13.2001110Z Setting output is-test-matrix-empty=False 2025-08-26T20:20:13.2132739Z ##[group]Run echo "Filtered matrix:" 2025-08-26T20:20:13.2132989Z echo "Filtered matrix:" 2025-08-26T20:20:13.2134575Z echo "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" 2025-08-26T20:20:13.2136079Z  2025-08-26T20:20:13.2136220Z echo 2025-08-26T20:20:13.2136413Z echo "Is the current job unstable? False" 2025-08-26T20:20:13.2136633Z  2025-08-26T20:20:13.2136777Z echo 2025-08-26T20:20:13.2136955Z echo "Is keep-going label set? True" 2025-08-26T20:20:13.2137266Z  2025-08-26T20:20:13.2137409Z echo 2025-08-26T20:20:13.2137572Z echo "Reenabled issues? " 2025-08-26T20:20:13.2142157Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:13.2142396Z env: 2025-08-26T20:20:13.2142551Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:13.2142734Z ##[endgroup] 2025-08-26T20:20:13.2168125Z Filtered matrix: 2025-08-26T20:20:13.2170134Z {include: [{config: cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: inductor_torchbench_cpu_smoketest_perf, shard: 1, num_shards: 1, runner: linux.24xl.spr-metal}]} 2025-08-26T20:20:13.2171736Z 2025-08-26T20:20:13.2171837Z Is the current job unstable? False 2025-08-26T20:20:13.2171983Z 2025-08-26T20:20:13.2172075Z Is keep-going label set? True 2025-08-26T20:20:13.2172209Z 2025-08-26T20:20:13.2172280Z Reenabled issues? 2025-08-26T20:20:13.2217447Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-26T20:20:13.2217781Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-26T20:20:13.2222111Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:13.2222354Z env: 2025-08-26T20:20:13.2222518Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:13.2222700Z JOB_TIMEOUT: 240 2025-08-26T20:20:13.2222867Z ##[endgroup] 2025-08-26T20:20:13.2273093Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-26T20:20:13.2273475Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-26T20:20:13.2273882Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-26T20:20:13.2278163Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:20:13.2278432Z env: 2025-08-26T20:20:13.2278606Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:13.2278850Z ##[endgroup] 2025-08-26T20:20:13.2390723Z ##[group]Run set -x 2025-08-26T20:20:13.2390976Z set -x 2025-08-26T20:20:13.2391127Z  2025-08-26T20:20:13.2391307Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-08-26T20:20:13.2391567Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-08-26T20:20:13.2391823Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-08-26T20:20:13.2392051Z  TEST_COMMAND=.ci/onnx/test.sh 2025-08-26T20:20:13.2392255Z else 2025-08-26T20:20:13.2392436Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-08-26T20:20:13.2392640Z fi 2025-08-26T20:20:13.2392794Z  2025-08-26T20:20:13.2392978Z # Leaving 1GB for the runner and other things 2025-08-26T20:20:13.2393367Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-08-26T20:20:13.2393943Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-08-26T20:20:13.2394396Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-08-26T20:20:13.2394748Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-08-26T20:20:13.2395024Z  2025-08-26T20:20:13.2395225Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-08-26T20:20:13.2395456Z  SHM_OPTS= 2025-08-26T20:20:13.2395639Z  JENKINS_USER= 2025-08-26T20:20:13.2395889Z  # ensure that docker container cleanly exits in 12 hours 2025-08-26T20:20:13.2396435Z  # if for some reason cleanup action doesn't stop container 2025-08-26T20:20:13.2396849Z  # when job is cancelled 2025-08-26T20:20:13.2397077Z  DOCKER_SHELL_CMD="sleep 12h" 2025-08-26T20:20:13.2397287Z else 2025-08-26T20:20:13.2397479Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-08-26T20:20:13.2397717Z  JENKINS_USER="--user jenkins" 2025-08-26T20:20:13.2397935Z  DOCKER_SHELL_CMD= 2025-08-26T20:20:13.2398129Z fi 2025-08-26T20:20:13.2398285Z  2025-08-26T20:20:13.2398520Z # detached container should get cleaned up by teardown_ec2_linux 2025-08-26T20:20:13.2398859Z # TODO: Stop building test binaries as part of the build phase 2025-08-26T20:20:13.2399335Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-08-26T20:20:13.2399695Z # shellcheck disable=SC2086,SC2090 2025-08-26T20:20:13.2399928Z container_name=$(docker run \ 2025-08-26T20:20:13.2400146Z  ${GPU_FLAG:-} \ 2025-08-26T20:20:13.2400365Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-08-26T20:20:13.2400607Z  -e BUILD_ENVIRONMENT \ 2025-08-26T20:20:13.2400811Z  -e PR_NUMBER \ 2025-08-26T20:20:13.2401002Z  -e GITHUB_ACTIONS \ 2025-08-26T20:20:13.2401196Z  -e GITHUB_REPOSITORY \ 2025-08-26T20:20:13.2401402Z  -e GITHUB_WORKFLOW \ 2025-08-26T20:20:13.2401597Z  -e GITHUB_JOB \ 2025-08-26T20:20:13.2401781Z  -e GITHUB_RUN_ID \ 2025-08-26T20:20:13.2401962Z  -e GITHUB_RUN_NUMBER \ 2025-08-26T20:20:13.2402160Z  -e GITHUB_RUN_ATTEMPT \ 2025-08-26T20:20:13.2402359Z  -e JOB_ID \ 2025-08-26T20:20:13.2402536Z  -e JOB_NAME \ 2025-08-26T20:20:13.2402706Z  -e BASE_SHA \ 2025-08-26T20:20:13.2402880Z  -e BRANCH \ 2025-08-26T20:20:13.2403048Z  -e SHA1 \ 2025-08-26T20:20:13.2403225Z  -e AWS_DEFAULT_REGION \ 2025-08-26T20:20:13.2403422Z  -e IN_WHEEL_TEST \ 2025-08-26T20:20:13.2403608Z  -e SHARD_NUMBER \ 2025-08-26T20:20:13.2403794Z  -e TEST_CONFIG \ 2025-08-26T20:20:13.2404018Z  -e NUM_TEST_SHARDS \ 2025-08-26T20:20:13.2404213Z  -e REENABLED_ISSUES \ 2025-08-26T20:20:13.2404420Z  -e CONTINUE_THROUGH_ERROR \ 2025-08-26T20:20:13.2404723Z  -e VERBOSE_TEST_LOGS \ 2025-08-26T20:20:13.2404924Z  -e TEST_SHOWLOCALS \ 2025-08-26T20:20:13.2405108Z  -e NO_TEST_TIMEOUT \ 2025-08-26T20:20:13.2405295Z  -e NO_TD \ 2025-08-26T20:20:13.2405480Z  -e TD_DISTRIBUTED \ 2025-08-26T20:20:13.2405676Z  -e PR_LABELS \ 2025-08-26T20:20:13.2405877Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-08-26T20:20:13.2406103Z  -e SCCACHE_BUCKET \ 2025-08-26T20:20:13.2406296Z  -e SCCACHE_REGION \ 2025-08-26T20:20:13.2406485Z  -e XLA_CUDA \ 2025-08-26T20:20:13.2406684Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-08-26T20:20:13.2406918Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-08-26T20:20:13.2407159Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-08-26T20:20:13.2407412Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-08-26T20:20:13.2407650Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-08-26T20:20:13.2407878Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-08-26T20:20:13.2408120Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-08-26T20:20:13.2408340Z  -e DASHBOARD_TAG \ 2025-08-26T20:20:13.2408546Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-08-26T20:20:13.2408792Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-08-26T20:20:13.2409079Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-08-26T20:20:13.2409362Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-08-26T20:20:13.2409635Z  --security-opt seccomp=unconfined \ 2025-08-26T20:20:13.2409865Z  --cap-add=SYS_PTRACE \ 2025-08-26T20:20:13.2410122Z  --ipc=host \ 2025-08-26T20:20:13.2410312Z  ${SHM_OPTS} \ 2025-08-26T20:20:13.2410493Z  --tty \ 2025-08-26T20:20:13.2410666Z  --detach \ 2025-08-26T20:20:13.2410855Z  --name="${container_name}" \ 2025-08-26T20:20:13.2411081Z  ${JENKINS_USER} \ 2025-08-26T20:20:13.2411318Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-08-26T20:20:13.2411576Z  -w /var/lib/jenkins/workspace \ 2025-08-26T20:20:13.2411779Z  "${DOCKER_IMAGE}" \ 2025-08-26T20:20:13.2411970Z  ${DOCKER_SHELL_CMD} 2025-08-26T20:20:13.2412149Z ) 2025-08-26T20:20:13.2412355Z # Propagate download.pytorch.org IP to container 2025-08-26T20:20:13.2412765Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2025-08-26T20:20:13.2413199Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-08-26T20:20:13.2413462Z  2025-08-26T20:20:13.2413651Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-08-26T20:20:13.2414013Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-08-26T20:20:13.2414325Z fi 2025-08-26T20:20:13.2414475Z  2025-08-26T20:20:13.2414797Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-08-26T20:20:13.2419411Z shell: /usr/bin/bash -e {0} 2025-08-26T20:20:13.2419610Z env: 2025-08-26T20:20:13.2419774Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:20:13.2420010Z BUILD_ENVIRONMENT: linux-jammy-py3.9-gcc11-build 2025-08-26T20:20:13.2420258Z PR_NUMBER: 2025-08-26T20:20:13.2420425Z GITHUB_REPOSITORY: pytorch/pytorch 2025-08-26T20:20:13.2420621Z GITHUB_WORKFLOW: inductor 2025-08-26T20:20:13.2420793Z GITHUB_JOB: test 2025-08-26T20:20:13.2420954Z GITHUB_RUN_ID: 17248463670 2025-08-26T20:20:13.2421137Z GITHUB_RUN_NUMBER: 149891 2025-08-26T20:20:13.2421305Z GITHUB_RUN_ATTEMPT: 1 2025-08-26T20:20:13.2421473Z JOB_ID: 48946862580 2025-08-26T20:20:13.2421818Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:20:13.2422180Z BRANCH: main 2025-08-26T20:20:13.2422425Z SHA1: 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:13.2422673Z BASE_SHA: 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:13.2422919Z TEST_CONFIG: dynamic_cpu_inductor_huggingface 2025-08-26T20:20:13.2423135Z SHARD_NUMBER: 1 2025-08-26T20:20:13.2423290Z NUM_TEST_SHARDS: 1 2025-08-26T20:20:13.2423459Z REENABLED_ISSUES: 2025-08-26T20:20:13.2423634Z CONTINUE_THROUGH_ERROR: True 2025-08-26T20:20:13.2423826Z VERBOSE_TEST_LOGS: False 2025-08-26T20:20:13.2424005Z TEST_SHOWLOCALS: False 2025-08-26T20:20:13.2424186Z NO_TEST_TIMEOUT: False 2025-08-26T20:20:13.2424362Z NO_TD: False 2025-08-26T20:20:13.2424544Z TD_DISTRIBUTED: False 2025-08-26T20:20:13.2424756Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-08-26T20:20:13.2424985Z SCCACHE_REGION: us-east-1 2025-08-26T20:20:13.2425164Z SHM_SIZE: 1g 2025-08-26T20:20:13.2425675Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:20:13.2426207Z XLA_CUDA: 2025-08-26T20:20:13.2426450Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-08-26T20:20:13.2426739Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-08-26T20:20:13.2426959Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-08-26T20:20:13.2427159Z DASHBOARD_TAG: 2025-08-26T20:20:13.2427511Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-08-26T20:20:13.2427789Z HUGGING_FACE_HUB_TOKEN: *** 2025-08-26T20:20:13.2428064Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-08-26T20:20:13.2428391Z ARTIFACTS_FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:20:13.2428786Z ##[endgroup] 2025-08-26T20:20:13.2452581Z + [[ dynamic_cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-08-26T20:20:13.2452958Z + [[ linux-jammy-py3.9-gcc11-build == *onnx* ]] 2025-08-26T20:20:13.2453230Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-08-26T20:20:13.2455834Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-08-26T20:20:13.2473898Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-08-26T20:20:13.2474171Z + TOTAL_MEMORY_WITH_SWAP=125 2025-08-26T20:20:13.2474745Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-08-26T20:20:13.2475045Z + SHM_OPTS=--shm-size=1g 2025-08-26T20:20:13.2475262Z + JENKINS_USER='--user jenkins' 2025-08-26T20:20:13.2475474Z + DOCKER_SHELL_CMD= 2025-08-26T20:20:13.2485664Z +++ nproc --ignore=2 2025-08-26T20:20:13.2509187Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_17248463670 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:20:24.0693393Z + container_name=0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:20:24.0700974Z + grep download.pytorch.org /etc/hosts 2025-08-26T20:20:24.0706896Z + docker exec -i 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce sudo bash -c '/bin/cat >> /etc/hosts' 2025-08-26T20:20:24.2393457Z + echo DOCKER_CONTAINER_ID=0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:20:24.2393957Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-08-26T20:20:24.2400167Z ++ echo dist/torch-2.9.0a0+git262640f-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:24.2404131Z + docker exec -t 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce sh -c 'python3 -m pip install dist/torch-2.9.0a0+git262640f-cp39-cp39-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-08-26T20:20:24.5887818Z Processing ./dist/torch-2.9.0a0+git262640f-cp39-cp39-linux_x86_64.whl (from torch==2.9.0a0+git262640f) 2025-08-26T20:20:24.8080493Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (3.19.1) 2025-08-26T20:20:24.8081483Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (4.15.0) 2025-08-26T20:20:24.8084429Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (1.13.3) 2025-08-26T20:20:24.8085221Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (2.8.8) 2025-08-26T20:20:24.8088395Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (3.1.6) 2025-08-26T20:20:24.8093226Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (2025.3.0) 2025-08-26T20:20:24.8102845Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (3.3.0) 2025-08-26T20:20:24.8409989Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from opt-einsum>=3.3->torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (1.22.4) 2025-08-26T20:20:24.8424840Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (1.3.0) 2025-08-26T20:20:24.8463685Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from jinja2->torch==2.9.0a0+git262640f->torch==2.9.0a0+git262640f) (3.0.2) 2025-08-26T20:20:25.6329070Z Installing collected packages: torch 2025-08-26T20:20:33.0592172Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-08-26T20:20:33.0592868Z dall-e 0.1 requires torchvision, which is not installed. 2025-08-26T20:20:33.0593204Z effdet 0.4.1 requires torchvision, which is not installed. 2025-08-26T20:20:33.0593573Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-08-26T20:20:33.0594041Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-08-26T20:20:33.0594558Z timm 1.0.14 requires torchvision, which is not installed. 2025-08-26T20:20:33.0594927Z Successfully installed torch-2.9.0a0+git262640f 2025-08-26T20:20:33.1579111Z + export TERM=vt100 2025-08-26T20:20:33.1585017Z + TERM=vt100 2025-08-26T20:20:33.1590161Z ++ dirname .ci/pytorch/test.sh 2025-08-26T20:20:33.1590589Z + source .ci/pytorch/common.sh 2025-08-26T20:20:33.1590833Z +++ dirname .ci/pytorch/common.sh 2025-08-26T20:20:33.1595610Z ++ source .ci/pytorch/common_utils.sh 2025-08-26T20:20:33.1595923Z +++ declare -f -t trap_add 2025-08-26T20:20:33.1596144Z ++ set -ex -o pipefail 2025-08-26T20:20:33.1596548Z ++ [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-26T20:20:33.1597226Z ++ BUILD_TEST_LIBTORCH=0 2025-08-26T20:20:33.1597666Z ++ dirname .ci/pytorch/test.sh 2025-08-26T20:20:33.1617932Z + source .ci/pytorch/common-build.sh 2025-08-26T20:20:33.1618390Z ++ [[ linux-jammy-py3.9-gcc11-build != *win-* ]] 2025-08-26T20:20:33.1618865Z ++++ dirname .ci/pytorch/common-build.sh 2025-08-26T20:20:33.1627457Z +++ cd .ci/pytorch 2025-08-26T20:20:33.1627878Z +++ pwd -P 2025-08-26T20:20:33.1628230Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-08-26T20:20:33.1628736Z ++ [[ linux-jammy-py3.9-gcc11-build == *-pch* ]] 2025-08-26T20:20:33.1629091Z ++ which sccache 2025-08-26T20:20:33.1646277Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-08-26T20:20:33.1646566Z ++ sccache --stop-server 2025-08-26T20:20:33.1675481Z ++ true 2025-08-26T20:20:33.1675717Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-08-26T20:20:33.1692664Z ++ trap_add sccache_epilogue EXIT 2025-08-26T20:20:33.1694909Z ++ trap_add_cmd=sccache_epilogue 2025-08-26T20:20:33.1695162Z ++ shift 2025-08-26T20:20:33.1695369Z ++ for trap_add_name in "$@" 2025-08-26T20:20:33.1695604Z ++++ trap -p EXIT 2025-08-26T20:20:33.1695787Z +++ eval 'extract_trap_cmd ' 2025-08-26T20:20:33.1695973Z ++++ extract_trap_cmd 2025-08-26T20:20:33.1696155Z ++++ printf '%s\n' '' 2025-08-26T20:20:33.1696543Z +++ printf '%s\n' sccache_epilogue 2025-08-26T20:20:33.1696760Z ++ trap -- ' 2025-08-26T20:20:33.1696961Z sccache_epilogue' EXIT 2025-08-26T20:20:33.1697199Z ++ [[ -n 1 ]] 2025-08-26T20:20:33.1697503Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-08-26T20:20:33.1697980Z Skipping sccache server initialization, setting environment variables 2025-08-26T20:20:33.1698591Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-08-26T20:20:33.1698806Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-08-26T20:20:33.1699055Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-26T20:20:33.1699353Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-26T20:20:33.1699673Z ++ export RUST_LOG=sccache::server=error 2025-08-26T20:20:33.1699912Z ++ RUST_LOG=sccache::server=error 2025-08-26T20:20:33.1700125Z ++ sccache --zero-stats 2025-08-26T20:20:33.3373802Z Statistics zeroed. 2025-08-26T20:20:33.3381669Z ++ which ccache 2025-08-26T20:20:33.3405396Z + [[ linux-jammy-py3.9-gcc11-build != *rocm* ]] 2025-08-26T20:20:33.3410437Z + [[ linux-jammy-py3.9-gcc11-build != *s390x* ]] 2025-08-26T20:20:33.3411774Z + [[ -d /var/lib/jenkins/workspace ]] 2025-08-26T20:20:33.3412406Z ++ stat -c %u /var/lib/jenkins/workspace 2025-08-26T20:20:33.3416586Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-08-26T20:20:33.3416857Z + trap_add cleanup_workspace EXIT 2025-08-26T20:20:33.3417129Z + trap_add_cmd=cleanup_workspace 2025-08-26T20:20:33.3417334Z + shift 2025-08-26T20:20:33.3417500Z + for trap_add_name in "$@" 2025-08-26T20:20:33.3423439Z +++ trap -p EXIT 2025-08-26T20:20:33.3424027Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-08-26T20:20:33.3424277Z sccache_epilogue'\'' EXIT' 2025-08-26T20:20:33.3424482Z +++ extract_trap_cmd trap -- ' 2025-08-26T20:20:33.3424714Z sccache_epilogue' EXIT 2025-08-26T20:20:33.3424922Z +++ printf '%s\n' ' 2025-08-26T20:20:33.3425100Z sccache_epilogue' 2025-08-26T20:20:33.3426999Z ++ printf '%s\n' cleanup_workspace 2025-08-26T20:20:33.3434260Z + trap -- ' 2025-08-26T20:20:33.3434505Z sccache_epilogue 2025-08-26T20:20:33.3434720Z cleanup_workspace' EXIT 2025-08-26T20:20:33.3434972Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-08-26T20:20:33.7955933Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-08-26T20:20:33.7973105Z + echo 'Environment variables:' 2025-08-26T20:20:33.7974309Z Environment variables: 2025-08-26T20:20:33.7974606Z + env 2025-08-26T20:20:33.7984072Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-26T20:20:33.7984521Z CONTINUE_THROUGH_ERROR=True 2025-08-26T20:20:33.7984793Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-08-26T20:20:33.7985337Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-08-26T20:20:33.7985564Z HOSTNAME=0dca33bcc852 2025-08-26T20:20:33.7986301Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.7986748Z GITHUB_ACTION=__run_2 2025-08-26T20:20:33.7986963Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-26T20:20:33.7987188Z GITHUB_RUN_NUMBER=149891 2025-08-26T20:20:33.7987420Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-08-26T20:20:33.7987674Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-26T20:20:33.7987924Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-08-26T20:20:33.7988153Z SCCACHE_IDLE_TIMEOUT=0 2025-08-26T20:20:33.7988450Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-08-26T20:20:33.7988698Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-08-26T20:20:33.7988932Z GITHUB_REF_TYPE=branch 2025-08-26T20:20:33.7989146Z BASE_SHA=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.7989386Z XLA_CUDA= 2025-08-26T20:20:33.7989570Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-08-26T20:20:33.7989863Z HUGGING_FACE_HUB_TOKEN=*** 2025-08-26T20:20:33.7990355Z *** 2025-08-26T20:20:33.7990534Z GITHUB_REPOSITORY_ID=65600975 2025-08-26T20:20:33.7990748Z GITHUB_ACTIONS=true 2025-08-26T20:20:33.7990976Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-26T20:20:33.7991253Z SHA1=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.7991508Z GITHUB_SHA=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.7991873Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-08-26T20:20:33.7992204Z UCC_HOME=/usr 2025-08-26T20:20:33.7992384Z VERBOSE_TEST_LOGS=False 2025-08-26T20:20:33.7992575Z GITHUB_REF=refs/heads/main 2025-08-26T20:20:33.7992895Z SHARD_NUMBER=1 2025-08-26T20:20:33.7993080Z GITHUB_REF_PROTECTED=true 2025-08-26T20:20:33.7993279Z HOME=/var/lib/jenkins 2025-08-26T20:20:33.7993485Z GITHUB_API_URL=https://api.github.com 2025-08-26T20:20:33.7993734Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-26T20:20:33.7993955Z UCX_COMMIT= 2025-08-26T20:20:33.7994122Z USE_SYSTEM_NCCL=1 2025-08-26T20:20:33.7994294Z NUM_TEST_SHARDS=1 2025-08-26T20:20:33.7994465Z UCX_HOME=/usr 2025-08-26T20:20:33.7994861Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.7995477Z JOB_NAME=linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:20:33.7996106Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.7996860Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-26T20:20:33.7997231Z GITHUB_EVENT_NAME=push 2025-08-26T20:20:33.7997421Z DASHBOARD_TAG= 2025-08-26T20:20:33.7997590Z GITHUB_RUN_ID=17248463670 2025-08-26T20:20:33.7997786Z INSTALLED_OPENBLAS= 2025-08-26T20:20:33.7998234Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.7998723Z GITHUB_ACTOR=pytorchmergebot 2025-08-26T20:20:33.7998930Z PR_NUMBER= 2025-08-26T20:20:33.7999089Z DESIRED_CUDA= 2025-08-26T20:20:33.7999555Z GITHUB_RUN_ATTEMPT=1 2025-08-26T20:20:33.7999810Z ANACONDA_PYTHON_VERSION=3.9 2025-08-26T20:20:33.8000050Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-26T20:20:33.8000294Z TERM=vt100 2025-08-26T20:20:33.8000458Z INSTALLED_VISION=yes 2025-08-26T20:20:33.8000634Z BRANCH=main 2025-08-26T20:20:33.8000796Z SCCACHE_REGION=us-east-1 2025-08-26T20:20:33.8000999Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-26T20:20:33.8001208Z CUDA_PATH=/usr/local/cuda 2025-08-26T20:20:33.8001622Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-08-26T20:20:33.8002028Z GITHUB_SERVER_URL=https://github.com 2025-08-26T20:20:33.8002247Z UCC_COMMIT= 2025-08-26T20:20:33.8002412Z REENABLED_ISSUES= 2025-08-26T20:20:33.8002581Z DOCS=yes 2025-08-26T20:20:33.8002731Z SHLVL=1 2025-08-26T20:20:33.8002890Z MAX_JOBS=30 2025-08-26T20:20:33.8003060Z GITHUB_ACTOR_ID=97764156 2025-08-26T20:20:33.8003421Z GITHUB_WORKFLOW_SHA=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.8003680Z GITHUB_REF_NAME=main 2025-08-26T20:20:33.8003962Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-08-26T20:20:33.8004263Z GITHUB_JOB=test 2025-08-26T20:20:33.8004439Z NO_TEST_TIMEOUT=False 2025-08-26T20:20:33.8004617Z TD_DISTRIBUTED=False 2025-08-26T20:20:33.8004820Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-26T20:20:33.8005043Z GITHUB_RETENTION_DAYS=90 2025-08-26T20:20:33.8005241Z OPENSSL_DIR=/opt/openssl 2025-08-26T20:20:33.8005433Z GITHUB_ACTION_REPOSITORY= 2025-08-26T20:20:33.8005951Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-26T20:20:33.8006444Z GITHUB_BASE_REF= 2025-08-26T20:20:33.8006617Z INSTALLED_ACL= 2025-08-26T20:20:33.8006931Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:20:33.8007287Z CI=true 2025-08-26T20:20:33.8007458Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-26T20:20:33.8007717Z RUST_LOG=sccache::server=error 2025-08-26T20:20:33.8007908Z JOB_ID=48946862580 2025-08-26T20:20:33.8008083Z GITHUB_HEAD_REF= 2025-08-26T20:20:33.8008256Z GITHUB_ACTION_REF= 2025-08-26T20:20:33.8008478Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-08-26T20:20:33.8008721Z TEST_SHOWLOCALS=False 2025-08-26T20:20:33.8008909Z GITHUB_WORKFLOW=inductor 2025-08-26T20:20:33.8009108Z DEBIAN_FRONTEND=noninteractive 2025-08-26T20:20:33.8009522Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.8010012Z NO_TD=False 2025-08-26T20:20:33.8010188Z SKIP_SCCACHE_INITIALIZATION=1 2025-08-26T20:20:33.8010412Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-08-26T20:20:33.8010636Z _=/usr/bin/env 2025-08-26T20:20:33.8010889Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-08-26T20:20:33.8253261Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch 2025-08-26T20:20:33.8253697Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2025-08-26T20:20:33.8254044Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib 2025-08-26T20:20:33.8254390Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/test 2025-08-26T20:20:33.8254670Z + BUILD_DIR=build 2025-08-26T20:20:33.8254846Z + BUILD_RENAMED_DIR=build_renamed 2025-08-26T20:20:33.8255059Z + BUILD_BIN_DIR=build/bin 2025-08-26T20:20:33.8255250Z + SHARD_NUMBER=1 2025-08-26T20:20:33.8255466Z + NUM_TEST_SHARDS=1 2025-08-26T20:20:33.8255666Z + export TORCH_SERIALIZATION_DEBUG=1 2025-08-26T20:20:33.8255879Z + TORCH_SERIALIZATION_DEBUG=1 2025-08-26T20:20:33.8256070Z + export VALGRIND=ON 2025-08-26T20:20:33.8256233Z + VALGRIND=ON 2025-08-26T20:20:33.8256431Z + [[ linux-jammy-py3.9-gcc11-build == *clang9* ]] 2025-08-26T20:20:33.8256688Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-26T20:20:33.8256903Z + detect_cuda_arch 2025-08-26T20:20:33.8257093Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-08-26T20:20:33.8257340Z + [[ linux-jammy-py3.9-gcc11-build == *s390x* ]] 2025-08-26T20:20:33.8257554Z + [[ 0 == \1 ]] 2025-08-26T20:20:33.8257710Z + [[ True == \1 ]] 2025-08-26T20:20:33.8257889Z + [[ linux-jammy-py3.9-gcc11-build != *bazel* ]] 2025-08-26T20:20:33.8262128Z ++ realpath build/custom_test_artifacts 2025-08-26T20:20:33.8263853Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-08-26T20:20:33.8264205Z + [[ -n '' ]] 2025-08-26T20:20:33.8264520Z + echo 'Environment variables' 2025-08-26T20:20:33.8264735Z Environment variables 2025-08-26T20:20:33.8264913Z + env 2025-08-26T20:20:33.8288781Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-26T20:20:33.8289127Z CONTINUE_THROUGH_ERROR=True 2025-08-26T20:20:33.8289433Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-08-26T20:20:33.8290303Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-08-26T20:20:33.8290518Z HOSTNAME=0dca33bcc852 2025-08-26T20:20:33.8290926Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.8291431Z GITHUB_ACTION=__run_2 2025-08-26T20:20:33.8291633Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-26T20:20:33.8291856Z GITHUB_RUN_NUMBER=149891 2025-08-26T20:20:33.8292095Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-08-26T20:20:33.8292339Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-26T20:20:33.8292575Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-08-26T20:20:33.8292804Z SCCACHE_IDLE_TIMEOUT=0 2025-08-26T20:20:33.8293098Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-08-26T20:20:33.8293323Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-08-26T20:20:33.8293554Z GITHUB_REF_TYPE=branch 2025-08-26T20:20:33.8293761Z BASE_SHA=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.8294009Z XLA_CUDA= 2025-08-26T20:20:33.8294181Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-08-26T20:20:33.8294611Z HUGGING_FACE_HUB_TOKEN=*** 2025-08-26T20:20:33.8294865Z *** 2025-08-26T20:20:33.8295024Z GITHUB_REPOSITORY_ID=65600975 2025-08-26T20:20:33.8295225Z GITHUB_ACTIONS=true 2025-08-26T20:20:33.8295443Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-26T20:20:33.8295705Z SHA1=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.8296016Z GITHUB_SHA=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.8296535Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-08-26T20:20:33.8296883Z UCC_HOME=/usr 2025-08-26T20:20:33.8297060Z TORCH_SERIALIZATION_DEBUG=1 2025-08-26T20:20:33.8297368Z VERBOSE_TEST_LOGS=False 2025-08-26T20:20:33.8297603Z GITHUB_REF=refs/heads/main 2025-08-26T20:20:33.8297795Z SHARD_NUMBER=1 2025-08-26T20:20:33.8297972Z GITHUB_REF_PROTECTED=true 2025-08-26T20:20:33.8298158Z HOME=/var/lib/jenkins 2025-08-26T20:20:33.8298378Z GITHUB_API_URL=https://api.github.com 2025-08-26T20:20:33.8298608Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-26T20:20:33.8298806Z UCX_COMMIT= 2025-08-26T20:20:33.8298950Z USE_SYSTEM_NCCL=1 2025-08-26T20:20:33.8299114Z NUM_TEST_SHARDS=1 2025-08-26T20:20:33.8299271Z UCX_HOME=/usr 2025-08-26T20:20:33.8299625Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.8300177Z JOB_NAME=linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:20:33.8300766Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.8301245Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-26T20:20:33.8301555Z GITHUB_EVENT_NAME=push 2025-08-26T20:20:33.8301725Z DASHBOARD_TAG= 2025-08-26T20:20:33.8301881Z GITHUB_RUN_ID=17248463670 2025-08-26T20:20:33.8302058Z INSTALLED_OPENBLAS= 2025-08-26T20:20:33.8302435Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.8302845Z GITHUB_ACTOR=pytorchmergebot 2025-08-26T20:20:33.8303023Z PR_NUMBER= 2025-08-26T20:20:33.8303172Z DESIRED_CUDA= 2025-08-26T20:20:33.8303330Z GITHUB_RUN_ATTEMPT=1 2025-08-26T20:20:33.8303545Z VALGRIND=ON 2025-08-26T20:20:33.8303706Z ANACONDA_PYTHON_VERSION=3.9 2025-08-26T20:20:33.8303921Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-26T20:20:33.8304145Z TERM=vt100 2025-08-26T20:20:33.8304294Z INSTALLED_VISION=yes 2025-08-26T20:20:33.8304458Z BRANCH=main 2025-08-26T20:20:33.8304608Z SCCACHE_REGION=us-east-1 2025-08-26T20:20:33.8304802Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-26T20:20:33.8304993Z CUDA_PATH=/usr/local/cuda 2025-08-26T20:20:33.8305317Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-08-26T20:20:33.8305666Z GITHUB_SERVER_URL=https://github.com 2025-08-26T20:20:33.8305864Z UCC_COMMIT= 2025-08-26T20:20:33.8306085Z REENABLED_ISSUES= 2025-08-26T20:20:33.8306245Z DOCS=yes 2025-08-26T20:20:33.8306383Z SHLVL=1 2025-08-26T20:20:33.8306522Z MAX_JOBS=30 2025-08-26T20:20:33.8306672Z GITHUB_ACTOR_ID=97764156 2025-08-26T20:20:33.8306889Z GITHUB_WORKFLOW_SHA=262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:20:33.8307131Z GITHUB_REF_NAME=main 2025-08-26T20:20:33.8307384Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-08-26T20:20:33.8307659Z GITHUB_JOB=test 2025-08-26T20:20:33.8307812Z NO_TEST_TIMEOUT=False 2025-08-26T20:20:33.8307984Z TD_DISTRIBUTED=False 2025-08-26T20:20:33.8308168Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-26T20:20:33.8308375Z GITHUB_RETENTION_DAYS=90 2025-08-26T20:20:33.8308547Z OPENSSL_DIR=/opt/openssl 2025-08-26T20:20:33.8308734Z GITHUB_ACTION_REPOSITORY= 2025-08-26T20:20:33.8309230Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-26T20:20:33.8309735Z GITHUB_BASE_REF= 2025-08-26T20:20:33.8309906Z INSTALLED_ACL= 2025-08-26T20:20:33.8310211Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:20:33.8310533Z CI=true 2025-08-26T20:20:33.8310691Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-26T20:20:33.8310936Z RUST_LOG=sccache::server=error 2025-08-26T20:20:33.8311114Z JOB_ID=48946862580 2025-08-26T20:20:33.8311272Z GITHUB_HEAD_REF= 2025-08-26T20:20:33.8311433Z GITHUB_ACTION_REF= 2025-08-26T20:20:33.8311625Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-08-26T20:20:33.8311856Z TEST_SHOWLOCALS=False 2025-08-26T20:20:33.8312113Z GITHUB_WORKFLOW=inductor 2025-08-26T20:20:33.8312295Z DEBIAN_FRONTEND=noninteractive 2025-08-26T20:20:33.8312678Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_d9853da8-3326-4f5a-91ad-e4af01fd8ca3 2025-08-26T20:20:33.8313050Z NO_TD=False 2025-08-26T20:20:33.8313216Z SKIP_SCCACHE_INITIALIZATION=1 2025-08-26T20:20:33.8313428Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-08-26T20:20:33.8313632Z _=/usr/bin/env 2025-08-26T20:20:33.8313787Z + echo 'Testing pytorch' 2025-08-26T20:20:33.8313959Z Testing pytorch 2025-08-26T20:20:33.8314138Z + export LANG=C.UTF-8 2025-08-26T20:20:33.8314303Z + LANG=C.UTF-8 2025-08-26T20:20:33.8314463Z + PR_NUMBER= 2025-08-26T20:20:33.8314676Z + [[ dynamic_cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-08-26T20:20:33.8314982Z + [[ dynamic_cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-08-26T20:20:33.8315277Z + [[ dynamic_cpu_inductor_huggingface == \s\l\o\w ]] 2025-08-26T20:20:33.8315566Z + [[ linux-jammy-py3.9-gcc11-build == *slow-gradcheck* ]] 2025-08-26T20:20:33.8315829Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-08-26T20:20:33.8316069Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-26T20:20:33.8316304Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-26T20:20:33.8316541Z + [[ dynamic_cpu_inductor_huggingface == *crossref* ]] 2025-08-26T20:20:33.8316798Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-26T20:20:33.8317044Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-26T20:20:33.8317297Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-08-26T20:20:33.8317533Z + pip_install ninja==1.10.2 2025-08-26T20:20:33.8317791Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-08-26T20:20:33.8318107Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-08-26T20:20:34.2199675Z Collecting ninja==1.10.2 2025-08-26T20:20:34.2304545Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-08-26T20:20:34.2432489Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-08-26T20:20:35.0100373Z Installing collected packages: ninja 2025-08-26T20:20:35.0100688Z Attempting uninstall: ninja 2025-08-26T20:20:35.0107906Z Found existing installation: ninja 1.11.1.3 2025-08-26T20:20:35.0128806Z Uninstalling ninja-1.11.1.3: 2025-08-26T20:20:35.0240969Z Successfully uninstalled ninja-1.11.1.3 2025-08-26T20:20:35.1332937Z Successfully installed ninja-1.10.2 2025-08-26T20:20:35.2357265Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-26T20:20:35.2358378Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-26T20:20:35.2359013Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-08-26T20:20:35.2359691Z + [[ linux-jammy-py3.9-gcc11-build == *asan* ]] 2025-08-26T20:20:35.2359983Z + [[ linux-jammy-py3.9-gcc11-build == *-debug* ]] 2025-08-26T20:20:35.2360259Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-08-26T20:20:35.2360643Z + echo 'We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass' 2025-08-26T20:20:35.2361105Z We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass 2025-08-26T20:20:35.2361430Z + cd test 2025-08-26T20:20:35.2361679Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-08-26T20:20:36.4906395Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-08-26T20:20:36.4906851Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-08-26T20:20:36.4907245Z + [[ dynamic_cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-08-26T20:20:36.4907605Z + DYNAMO_BENCHMARK_FLAGS=() 2025-08-26T20:20:36.4907893Z + [[ dynamic_cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-08-26T20:20:36.4908639Z + [[ dynamic_cpu_inductor_huggingface == *dynamo_eager* ]] 2025-08-26T20:20:36.4908908Z + [[ dynamic_cpu_inductor_huggingface == *aot_eager* ]] 2025-08-26T20:20:36.4909196Z + [[ dynamic_cpu_inductor_huggingface == *aot_inductor* ]] 2025-08-26T20:20:36.4909519Z + [[ dynamic_cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-08-26T20:20:36.4909819Z + [[ dynamic_cpu_inductor_huggingface == *inductor* ]] 2025-08-26T20:20:36.4910084Z + [[ dynamic_cpu_inductor_huggingface != *perf* ]] 2025-08-26T20:20:36.4910366Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-08-26T20:20:36.4910610Z + [[ dynamic_cpu_inductor_huggingface == *dynamic* ]] 2025-08-26T20:20:36.4910933Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2025-08-26T20:20:36.4911231Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-08-26T20:20:36.4911461Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-08-26T20:20:36.5157573Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-08-26T20:20:36.5157958Z + [[ linux-jammy-py3.9-gcc11-build == *-bazel-* ]] 2025-08-26T20:20:36.5158194Z + cd test 2025-08-26T20:20:36.5158502Z + python -c 'import torch; print(torch.__config__.show())' 2025-08-26T20:20:37.4655966Z PyTorch built with: 2025-08-26T20:20:37.4656399Z - GCC 11.4 2025-08-26T20:20:37.4656659Z - C++ Version: 201703 2025-08-26T20:20:37.4657177Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-26T20:20:37.4657680Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-26T20:20:37.4657996Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-26T20:20:37.4658250Z - LAPACK is enabled (usually provided by MKL) 2025-08-26T20:20:37.4658490Z - NNPACK is enabled 2025-08-26T20:20:37.4658696Z - CPU capability usage: AVX512 2025-08-26T20:20:37.4661951Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=262640fd220236042fbf4443cc163c8838c84c3d, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-08-26T20:20:37.4664687Z 2025-08-26T20:20:37.6655136Z + cd test 2025-08-26T20:20:37.6656040Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-08-26T20:20:38.6518916Z ATen/Parallel: 2025-08-26T20:20:38.6520126Z at::get_num_threads() : 16 2025-08-26T20:20:38.6520419Z at::get_num_interop_threads() : 16 2025-08-26T20:20:38.6524414Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-26T20:20:38.6524730Z omp_get_max_threads() : 16 2025-08-26T20:20:38.6528717Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-26T20:20:38.6529294Z mkl_get_max_threads() : 16 2025-08-26T20:20:38.6529821Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-26T20:20:38.6530171Z std::thread::hardware_concurrency() : 32 2025-08-26T20:20:38.6530411Z Environment variables: 2025-08-26T20:20:38.6530622Z OMP_NUM_THREADS : [not set] 2025-08-26T20:20:38.6530821Z MKL_NUM_THREADS : [not set] 2025-08-26T20:20:38.6531377Z ATen parallel backend: OpenMP 2025-08-26T20:20:38.6531507Z 2025-08-26T20:20:38.8623514Z + [[ dynamic_cpu_inductor_huggingface == *numpy_2* ]] 2025-08-26T20:20:38.8627994Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-08-26T20:20:38.8632543Z + [[ dynamic_cpu_inductor_huggingface == *backward* ]] 2025-08-26T20:20:38.8634406Z + [[ dynamic_cpu_inductor_huggingface == *xla* ]] 2025-08-26T20:20:38.8634680Z + [[ dynamic_cpu_inductor_huggingface == *vllm* ]] 2025-08-26T20:20:38.8634964Z + [[ dynamic_cpu_inductor_huggingface == *executorch* ]] 2025-08-26T20:20:38.8635255Z + [[ dynamic_cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-08-26T20:20:38.8635534Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-08-26T20:20:38.8635816Z + [[ dynamic_cpu_inductor_huggingface == distributed ]] 2025-08-26T20:20:38.8636119Z + [[ dynamic_cpu_inductor_huggingface == *operator_benchmark* ]] 2025-08-26T20:20:38.8636451Z + [[ dynamic_cpu_inductor_huggingface == *inductor_distributed* ]] 2025-08-26T20:20:38.8636801Z + [[ dynamic_cpu_inductor_huggingface == *inductor-halide* ]] 2025-08-26T20:20:38.8637113Z + [[ dynamic_cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-08-26T20:20:38.8637454Z + [[ dynamic_cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-08-26T20:20:38.8637774Z + [[ dynamic_cpu_inductor_huggingface == *huggingface* ]] 2025-08-26T20:20:38.8638022Z + install_torchvision 2025-08-26T20:20:38.8638202Z + local orig_preload 2025-08-26T20:20:38.8638386Z + local commit 2025-08-26T20:20:38.8638561Z ++ get_pinned_commit vision 2025-08-26T20:20:38.8638771Z ++ cat .github/ci_commit_pins/vision.txt 2025-08-26T20:20:38.8966475Z + commit=966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-08-26T20:20:38.8966814Z + orig_preload= 2025-08-26T20:20:38.8967020Z + '[' -n '' ']' 2025-08-26T20:20:38.8967259Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-08-26T20:20:38.8967806Z + pip_build_and_install git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 dist/vision 2025-08-26T20:20:38.8968476Z + local build_target=git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-08-26T20:20:38.8968898Z + local wheel_dir=dist/vision 2025-08-26T20:20:38.8969101Z + local found_whl=0 2025-08-26T20:20:38.8969294Z + for file in "${wheel_dir}"/*.whl 2025-08-26T20:20:38.8969914Z + [[ -f dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl ]] 2025-08-26T20:20:38.8970235Z + found_whl=1 2025-08-26T20:20:38.8970425Z + break 2025-08-26T20:20:38.8970584Z + '[' 1 == 0 ']' 2025-08-26T20:20:38.8970769Z + for file in "${wheel_dir}"/*.whl 2025-08-26T20:20:38.8971105Z + pip_install_whl dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:38.8971623Z + args=('dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl') 2025-08-26T20:20:38.8971949Z + local args 2025-08-26T20:20:38.8972218Z + [[ dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl == *\ * ]] 2025-08-26T20:20:38.8972658Z + for path in "${args[@]}" 2025-08-26T20:20:38.8972969Z + echo 'Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl' 2025-08-26T20:20:38.8973446Z Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:38.8973949Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:39.1871938Z Processing ./dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-26T20:20:39.1949734Z Installing collected packages: torchvision 2025-08-26T20:20:39.6397505Z Successfully installed torchvision-0.22.0a0+966da7e 2025-08-26T20:20:39.6829011Z + '[' -n '' ']' 2025-08-26T20:20:39.6834811Z + id=0 2025-08-26T20:20:39.6835238Z + test_dynamo_benchmark huggingface 0 2025-08-26T20:20:39.6835637Z ++ pwd 2025-08-26T20:20:39.6835945Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-08-26T20:20:39.6836260Z + local suite=huggingface 2025-08-26T20:20:39.6836893Z + shift 2025-08-26T20:20:39.6837044Z + local shard_id=0 2025-08-26T20:20:39.6837216Z + shift 2025-08-26T20:20:39.6837421Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-08-26T20:20:39.6837708Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-08-26T20:20:39.6837963Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-08-26T20:20:39.6838208Z + local dt=float32 2025-08-26T20:20:39.6838404Z + [[ dynamic_cpu_inductor_huggingface == *amp* ]] 2025-08-26T20:20:39.6838674Z + [[ dynamic_cpu_inductor_huggingface == *freezing* ]] 2025-08-26T20:20:39.6839010Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-08-26T20:20:39.6839607Z ++ pwd 2025-08-26T20:20:39.6839844Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-08-26T20:20:39.6840184Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-08-26T20:20:39.6860741Z + local name=inference 2025-08-26T20:20:39.6860983Z + shift 2025-08-26T20:20:39.6861167Z + local suite=huggingface 2025-08-26T20:20:39.6861371Z + shift 2025-08-26T20:20:39.6861519Z + local shard_id=0 2025-08-26T20:20:39.6861672Z + shift 2025-08-26T20:20:39.6861825Z + partition_flags=() 2025-08-26T20:20:39.6862014Z + local partition_flags 2025-08-26T20:20:39.6862183Z + [[ -n 1 ]] 2025-08-26T20:20:39.6862336Z + [[ -n 0 ]] 2025-08-26T20:20:39.6862617Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-08-26T20:20:39.6862989Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-08-26T20:20:39.6863249Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-08-26T20:20:39.6863499Z + [[ dynamic_cpu_inductor_huggingface == *_avx2* ]] 2025-08-26T20:20:39.6863756Z + [[ dynamic_cpu_inductor_huggingface == *_avx512* ]] 2025-08-26T20:20:39.6864609Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --dynamic-shapes --dynamic-batch-only --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-08-26T20:20:43.1320677Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:20:43.1321752Z from pkg_resources import resource_filename 2025-08-26T20:20:43.5421875Z 2025-08-26T20:20:43.5464630Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1740406Z 2025-08-26T20:22:56.1740518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1741047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1741554Z layer_outputs = layer_module( 2025-08-26T20:22:56.1741927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1742325Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1742782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1743265Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1743698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1744115Z self_outputs = self.self( 2025-08-26T20:22:56.1744523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1744992Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1745538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1746132Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1746401Z 2025-08-26T20:22:56.1746513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1747066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1747583Z layer_outputs = layer_module( 2025-08-26T20:22:56.1747964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1748322Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1748748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1749167Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1749586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1750000Z self_outputs = self.self( 2025-08-26T20:22:56.1750395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1750974Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1751540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1752271Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1752537Z 2025-08-26T20:22:56.1752658Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1753216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1753750Z layer_outputs = layer_module( 2025-08-26T20:22:56.1754122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1754524Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1754981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1755431Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1755892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1756351Z self_outputs = self.self( 2025-08-26T20:22:56.1756791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1757278Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1757820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1758457Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1758762Z 2025-08-26T20:22:56.1758852Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1759088Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1759366Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1759595Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1759866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1760425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1760972Z layer_outputs = layer_module( 2025-08-26T20:22:56.1761346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1761743Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1762200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1762659Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1763110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1763545Z self_outputs = self.self( 2025-08-26T20:22:56.1764490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.1764971Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1765512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1766095Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.1766662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.1767241Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.1767473Z 2025-08-26T20:22:56.1767563Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1767826Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1768522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1769067Z layer_outputs = layer_module( 2025-08-26T20:22:56.1769442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1769849Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1770305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1770752Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1771205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1771643Z self_outputs = self.self( 2025-08-26T20:22:56.1772053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.1772470Z attn_scores += diagonal_mask 2025-08-26T20:22:56.1772592Z 2025-08-26T20:22:56.1772701Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1773223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1773709Z layer_outputs = layer_module( 2025-08-26T20:22:56.1774055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1774471Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1774889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1775308Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1775726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1776143Z self_outputs = self.self( 2025-08-26T20:22:56.1776545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.1776963Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.1777107Z 2025-08-26T20:22:56.1777214Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1777730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1778226Z layer_outputs = layer_module( 2025-08-26T20:22:56.1778579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1778939Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1779365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1779783Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1780204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1780611Z self_outputs = self.self( 2025-08-26T20:22:56.1781019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1781481Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1782022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1782620Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.1783084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.1783451Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.1783612Z 2025-08-26T20:22:56.1783718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1784251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1784745Z layer_outputs = layer_module( 2025-08-26T20:22:56.1785087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1785460Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1785884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1786306Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1786730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1787143Z self_outputs = self.self( 2025-08-26T20:22:56.1787551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1788046Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1788610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1789224Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.1789743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.1790217Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.1790564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.1790924Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.1791072Z 2025-08-26T20:22:56.1791185Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1791681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1792164Z layer_outputs = layer_module( 2025-08-26T20:22:56.1792513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1792906Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1793354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1793797Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1794250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1794697Z self_outputs = self.self( 2025-08-26T20:22:56.1795122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1795603Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1796171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1796987Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.1797223Z 2025-08-26T20:22:56.1797339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1797977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1798514Z layer_outputs = layer_module( 2025-08-26T20:22:56.1798875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1799304Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1799791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1800251Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1800764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1801177Z self_outputs = self.self( 2025-08-26T20:22:56.1801574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1802028Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1802551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1803115Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.1803333Z 2025-08-26T20:22:56.1803438Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1803964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1804523Z layer_outputs = layer_module( 2025-08-26T20:22:56.1804875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1805238Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1805660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1806078Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1806494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1806911Z self_outputs = self.self( 2025-08-26T20:22:56.1807309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.1807847Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.1808103Z 2025-08-26T20:22:56.1808211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1808725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1809214Z layer_outputs = layer_module( 2025-08-26T20:22:56.1809555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1809922Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1810342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1810762Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1811179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.1811632Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.1812086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.1812519Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.1812663Z 2025-08-26T20:22:56.1812828Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1813349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1813840Z layer_outputs = layer_module( 2025-08-26T20:22:56.1814210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1814593Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1815040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.1815497Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.1815935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.1816365Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.1816817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.1817311Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.1817785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.1818244Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.1818403Z 2025-08-26T20:22:56.1818514Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1819118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1819643Z layer_outputs = layer_module( 2025-08-26T20:22:56.1820008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1820409Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1820869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.1821303Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.1821715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.1822114Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.1822540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.1823038Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.1823519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.1824008Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.1824411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.1824787Z return self.act(input) 2025-08-26T20:22:56.1824915Z 2025-08-26T20:22:56.1825028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1825576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1826097Z layer_outputs = layer_module( 2025-08-26T20:22:56.1826460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1826856Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1827317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.1827822Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.1828245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.1828667Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.1829084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.1829557Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.1830021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.1830448Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.1830595Z 2025-08-26T20:22:56.1830700Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1831212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1831697Z layer_outputs = layer_module( 2025-08-26T20:22:56.1832045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1832404Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1832820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1833238Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1833655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1834107Z self_outputs = self.self( 2025-08-26T20:22:56.1834515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.1834964Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.1835123Z 2025-08-26T20:22:56.1835235Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1835776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1836292Z layer_outputs = layer_module( 2025-08-26T20:22:56.1836653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1837038Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1837484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1837935Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1838370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1838814Z self_outputs = self.self( 2025-08-26T20:22:56.1839310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1839806Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1840344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1840961Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1841230Z 2025-08-26T20:22:56.1841346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1841896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1842416Z layer_outputs = layer_module( 2025-08-26T20:22:56.1842830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1843222Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1843675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1844128Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1844577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1845020Z self_outputs = self.self( 2025-08-26T20:22:56.1845445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.1845891Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.1846042Z 2025-08-26T20:22:56.1846152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1846704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1847219Z layer_outputs = layer_module( 2025-08-26T20:22:56.1847580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1847978Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1848424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1848877Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1849353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1849793Z self_outputs = self.self( 2025-08-26T20:22:56.1850234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1850715Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1851244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1851855Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1852120Z 2025-08-26T20:22:56.1852230Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1852774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1853260Z layer_outputs = layer_module( 2025-08-26T20:22:56.1853609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1853970Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1854394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1854815Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1855228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1855640Z self_outputs = self.self( 2025-08-26T20:22:56.1856036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1856488Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1856994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1857573Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1857843Z 2025-08-26T20:22:56.1857960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1858484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1858978Z layer_outputs = layer_module( 2025-08-26T20:22:56.1859334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1859755Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1860178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1860598Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1861022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1861469Z self_outputs = self.self( 2025-08-26T20:22:56.1861875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1862323Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1862840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1863450Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1863702Z 2025-08-26T20:22:56.1863843Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1864067Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1864287Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1864492Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1864751Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1865288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1865791Z layer_outputs = layer_module( 2025-08-26T20:22:56.1866163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1866569Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1867031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1867497Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1867978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1868430Z self_outputs = self.self( 2025-08-26T20:22:56.1868891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.1869401Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1869973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1870589Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.1871179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.1871788Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.1872030Z 2025-08-26T20:22:56.1872119Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1872385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1873009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1873536Z layer_outputs = layer_module( 2025-08-26T20:22:56.1873920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1874320Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1874783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1875250Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1891702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1892397Z self_outputs = self.self( 2025-08-26T20:22:56.1892844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.1893281Z attn_scores += diagonal_mask 2025-08-26T20:22:56.1893423Z 2025-08-26T20:22:56.1893542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1894086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1894589Z layer_outputs = layer_module( 2025-08-26T20:22:56.1894958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1895328Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1895762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1896563Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1897032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1897483Z self_outputs = self.self( 2025-08-26T20:22:56.1897915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.1898384Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.1898540Z 2025-08-26T20:22:56.1898667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1899199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1899695Z layer_outputs = layer_module( 2025-08-26T20:22:56.1900042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1900408Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1900838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1901264Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1901686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1902096Z self_outputs = self.self( 2025-08-26T20:22:56.1902530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.1902991Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.1903135Z 2025-08-26T20:22:56.1903252Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1903780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1904266Z layer_outputs = layer_module( 2025-08-26T20:22:56.1904736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1905108Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1905534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1905958Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1906395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1906839Z self_outputs = self.self( 2025-08-26T20:22:56.1907267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1907766Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1908329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1908949Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.1909385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.1909762Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.1909916Z 2025-08-26T20:22:56.1910030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1910538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1911067Z layer_outputs = layer_module( 2025-08-26T20:22:56.1911422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1911793Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1912230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1912673Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1913120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1913562Z self_outputs = self.self( 2025-08-26T20:22:56.1913991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1914480Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1915042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1915639Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.1916195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.1916715Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.1917094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.1917478Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.1917656Z 2025-08-26T20:22:56.1917771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1918347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1918895Z layer_outputs = layer_module( 2025-08-26T20:22:56.1919356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1919767Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1920271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1920722Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1921168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1921614Z self_outputs = self.self( 2025-08-26T20:22:56.1922038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1922529Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1923093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1923692Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.1923911Z 2025-08-26T20:22:56.1924031Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1924585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1925068Z layer_outputs = layer_module( 2025-08-26T20:22:56.1925406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1925764Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1926185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1926637Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1927065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1927488Z self_outputs = self.self( 2025-08-26T20:22:56.1927904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.1928381Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.1928950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.1929523Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.1929743Z 2025-08-26T20:22:56.1929852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1930391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1930878Z layer_outputs = layer_module( 2025-08-26T20:22:56.1931218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1931586Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1932005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1932422Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1932837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1933249Z self_outputs = self.self( 2025-08-26T20:22:56.1933651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.1934178Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.1934417Z 2025-08-26T20:22:56.1934530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1935075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1935565Z layer_outputs = layer_module( 2025-08-26T20:22:56.1935912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1936277Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1936696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1937108Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1937530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.1937992Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.1938450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.1938880Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.1939021Z 2025-08-26T20:22:56.1939125Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1939645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1940130Z layer_outputs = layer_module( 2025-08-26T20:22:56.1940490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1940878Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1941292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.1941721Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.1942130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.1942532Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.1942945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.1943400Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.1943851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.1944276Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.1944417Z 2025-08-26T20:22:56.1944529Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1945029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1945514Z layer_outputs = layer_module( 2025-08-26T20:22:56.1945864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1946228Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1946642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.1947058Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.1947478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.1947888Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.1948319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.1948786Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.1949269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.1949730Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.1950119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.1950470Z return self.act(input) 2025-08-26T20:22:56.1950585Z 2025-08-26T20:22:56.1950696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1951209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1951698Z layer_outputs = layer_module( 2025-08-26T20:22:56.1952047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1952415Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1952832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.1953267Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.1953674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.1954076Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.1954531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.1955045Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.1955547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.1955976Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.1956116Z 2025-08-26T20:22:56.1956232Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1956787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1957295Z layer_outputs = layer_module( 2025-08-26T20:22:56.1957684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1958070Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1958534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1958997Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1959525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1959982Z self_outputs = self.self( 2025-08-26T20:22:56.1960419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.1960872Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.1961014Z 2025-08-26T20:22:56.1961132Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1961642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1962136Z layer_outputs = layer_module( 2025-08-26T20:22:56.1962489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1962864Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1963284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1963702Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1964161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1964584Z self_outputs = self.self( 2025-08-26T20:22:56.1964995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1965438Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1965943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1966537Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1966796Z 2025-08-26T20:22:56.1966902Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1967424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1967913Z layer_outputs = layer_module( 2025-08-26T20:22:56.1968255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1968621Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1969044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1969467Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1969881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1970328Z self_outputs = self.self( 2025-08-26T20:22:56.1970727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.1971146Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.1971282Z 2025-08-26T20:22:56.1971396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1971896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1972380Z layer_outputs = layer_module( 2025-08-26T20:22:56.1972726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1973093Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1973512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1973926Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1974356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1974766Z self_outputs = self.self( 2025-08-26T20:22:56.1975157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1975593Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1976068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1976643Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1976896Z 2025-08-26T20:22:56.1977003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1977525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1978014Z layer_outputs = layer_module( 2025-08-26T20:22:56.1978425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1978812Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1979232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1979655Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1980064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1980483Z self_outputs = self.self( 2025-08-26T20:22:56.1980884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1981335Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1981843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1982465Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1982728Z 2025-08-26T20:22:56.1982840Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1983379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1983877Z layer_outputs = layer_module( 2025-08-26T20:22:56.1984227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1984624Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1985040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1985463Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1985887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1986301Z self_outputs = self.self( 2025-08-26T20:22:56.1986700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.1987145Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1987677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1988300Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.1988562Z 2025-08-26T20:22:56.1988660Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1988887Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1989113Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1989343Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1989583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1990103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1990583Z layer_outputs = layer_module( 2025-08-26T20:22:56.1990936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1991303Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1991724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1992150Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.1992564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.1993023Z self_outputs = self.self( 2025-08-26T20:22:56.1993483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.1993961Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.1994494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.1995071Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.1995633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.1996326Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.1996556Z 2025-08-26T20:22:56.1996651Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.1996905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.1997462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.1997980Z layer_outputs = layer_module( 2025-08-26T20:22:56.1998353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.1998750Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.1999189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.1999792Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2000252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2000709Z self_outputs = self.self( 2025-08-26T20:22:56.2001131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2001542Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2001674Z 2025-08-26T20:22:56.2001782Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2002299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2002790Z layer_outputs = layer_module( 2025-08-26T20:22:56.2003140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2003503Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2003923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2004339Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2004757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2005165Z self_outputs = self.self( 2025-08-26T20:22:56.2005568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2005991Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2006125Z 2025-08-26T20:22:56.2006238Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2006758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2007250Z layer_outputs = layer_module( 2025-08-26T20:22:56.2007585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2007929Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2008446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2008852Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2009244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2009640Z self_outputs = self.self( 2025-08-26T20:22:56.2010028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2010451Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2010592Z 2025-08-26T20:22:56.2010704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2011204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2011690Z layer_outputs = layer_module( 2025-08-26T20:22:56.2012034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2012403Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2012829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2013245Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2013664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2014115Z self_outputs = self.self( 2025-08-26T20:22:56.2014516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2014975Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2015516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2016108Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2016538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2016897Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2017051Z 2025-08-26T20:22:56.2017157Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2017681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2018170Z layer_outputs = layer_module( 2025-08-26T20:22:56.2018519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2018892Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2019310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2019750Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2020201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2020644Z self_outputs = self.self( 2025-08-26T20:22:56.2021055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2021511Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2022051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2022664Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2023212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2023712Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2024059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2024410Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2024567Z 2025-08-26T20:22:56.2024669Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2025210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2025740Z layer_outputs = layer_module( 2025-08-26T20:22:56.2026109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2026550Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2027056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2027520Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2027975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2028418Z self_outputs = self.self( 2025-08-26T20:22:56.2028858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2029414Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2029984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2030596Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2030815Z 2025-08-26T20:22:56.2030927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2031490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2032014Z layer_outputs = layer_module( 2025-08-26T20:22:56.2032386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2032778Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2033224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2033673Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2034132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2034578Z self_outputs = self.self( 2025-08-26T20:22:56.2035008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2035507Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2036062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2036658Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2036881Z 2025-08-26T20:22:56.2037001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2037543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2038065Z layer_outputs = layer_module( 2025-08-26T20:22:56.2038469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2038869Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2039402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2039865Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2040334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2040777Z self_outputs = self.self( 2025-08-26T20:22:56.2041199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2041732Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2041971Z 2025-08-26T20:22:56.2042082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2042603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2043092Z layer_outputs = layer_module( 2025-08-26T20:22:56.2043445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2043809Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2044220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2044689Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2045106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2045569Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2046024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2046448Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2046596Z 2025-08-26T20:22:56.2046700Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2047220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2047711Z layer_outputs = layer_module( 2025-08-26T20:22:56.2048070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2048431Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2048859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2049298Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2049711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2050111Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2050537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2050996Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2051463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2051892Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2052032Z 2025-08-26T20:22:56.2052136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2052687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2053182Z layer_outputs = layer_module( 2025-08-26T20:22:56.2053538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2053914Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2054335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2054774Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2055191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2055606Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2056045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2056507Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2056967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2057438Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2057823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2058162Z return self.act(input) 2025-08-26T20:22:56.2058282Z 2025-08-26T20:22:56.2058387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2058898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2059413Z layer_outputs = layer_module( 2025-08-26T20:22:56.2059752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2060106Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2060517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2060932Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2061336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2061730Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2062144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2062617Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2063084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2063513Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2063651Z 2025-08-26T20:22:56.2063761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2064256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2064731Z layer_outputs = layer_module( 2025-08-26T20:22:56.2065073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2065431Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2065850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2066273Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2066695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2067148Z self_outputs = self.self( 2025-08-26T20:22:56.2067562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2067984Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2068124Z 2025-08-26T20:22:56.2068227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2068748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2069238Z layer_outputs = layer_module( 2025-08-26T20:22:56.2069593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2069951Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2070377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2070803Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2071228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2071646Z self_outputs = self.self( 2025-08-26T20:22:56.2072043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2072498Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2073016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2073645Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2073893Z 2025-08-26T20:22:56.2074006Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2074539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2075054Z layer_outputs = layer_module( 2025-08-26T20:22:56.2075420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2075814Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2076259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2076697Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2077143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2077589Z self_outputs = self.self( 2025-08-26T20:22:56.2078017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2078471Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2078614Z 2025-08-26T20:22:56.2078723Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2079333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2079855Z layer_outputs = layer_module( 2025-08-26T20:22:56.2080223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2080611Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2081024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2081444Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2081903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2082335Z self_outputs = self.self( 2025-08-26T20:22:56.2082723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2083166Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2083666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2084254Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2084504Z 2025-08-26T20:22:56.2084618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2085132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2085617Z layer_outputs = layer_module( 2025-08-26T20:22:56.2085968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2086332Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2086753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2087169Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2087588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2088056Z self_outputs = self.self( 2025-08-26T20:22:56.2088453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2088897Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2089403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2089985Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2090236Z 2025-08-26T20:22:56.2090342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2090859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2091344Z layer_outputs = layer_module( 2025-08-26T20:22:56.2091703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2092063Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2092473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2092884Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2093293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2093695Z self_outputs = self.self( 2025-08-26T20:22:56.2094096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2094543Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2095041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2095629Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2095872Z 2025-08-26T20:22:56.2095956Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2096347Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2096637Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2096855Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2097088Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2097638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2098141Z layer_outputs = layer_module( 2025-08-26T20:22:56.2098494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2098869Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2099285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2099712Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2100134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2100551Z self_outputs = self.self( 2025-08-26T20:22:56.2100950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2101399Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2101913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2102457Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2103043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2103590Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2103799Z 2025-08-26T20:22:56.2103887Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2104140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2104670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2105158Z layer_outputs = layer_module( 2025-08-26T20:22:56.2105516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2105890Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2106322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2106749Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2107037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2107130Z self_outputs = self.self( 2025-08-26T20:22:56.2107417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2107505Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2107508Z 2025-08-26T20:22:56.2107618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2107972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2108057Z layer_outputs = layer_module( 2025-08-26T20:22:56.2108287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2108378Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2108711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2108790Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2109078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2109149Z self_outputs = self.self( 2025-08-26T20:22:56.2109435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2109514Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2109518Z 2025-08-26T20:22:56.2109629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2109986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2110058Z layer_outputs = layer_module( 2025-08-26T20:22:56.2110290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2110368Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2110659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2110732Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2111015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2111095Z self_outputs = self.self( 2025-08-26T20:22:56.2111376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2111501Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2111504Z 2025-08-26T20:22:56.2111609Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2111972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2112043Z layer_outputs = layer_module( 2025-08-26T20:22:56.2112265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2112351Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2112634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2112716Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2113001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2113081Z self_outputs = self.self( 2025-08-26T20:22:56.2113361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2113484Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2113845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2114020Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2114221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2114321Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2114325Z 2025-08-26T20:22:56.2114432Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2114792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2114864Z layer_outputs = layer_module( 2025-08-26T20:22:56.2115131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2115214Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2115501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2115576Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2115861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2115942Z self_outputs = self.self( 2025-08-26T20:22:56.2116239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2116372Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2116749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2116899Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2117245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2117343Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2117564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2117672Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2117711Z 2025-08-26T20:22:56.2117839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2118211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2118288Z layer_outputs = layer_module( 2025-08-26T20:22:56.2118531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2118617Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2118920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2118998Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2119368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2119448Z self_outputs = self.self( 2025-08-26T20:22:56.2119755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2119888Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2120259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2120433Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2120437Z 2025-08-26T20:22:56.2120547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2120932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2121009Z layer_outputs = layer_module( 2025-08-26T20:22:56.2121237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2121323Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2121590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2121669Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2121994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2122063Z self_outputs = self.self( 2025-08-26T20:22:56.2122345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2122457Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2122804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2122954Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2122958Z 2025-08-26T20:22:56.2123064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2123405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2123476Z layer_outputs = layer_module( 2025-08-26T20:22:56.2123701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2123778Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2124068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2124141Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2124412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2124509Z self_outputs = self.self( 2025-08-26T20:22:56.2124781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2124974Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2124977Z 2025-08-26T20:22:56.2125078Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2125423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2125494Z layer_outputs = layer_module( 2025-08-26T20:22:56.2125710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2125794Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2126071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2126157Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2126430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2126554Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2126835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2126921Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2126924Z 2025-08-26T20:22:56.2127035Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2127382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2127463Z layer_outputs = layer_module( 2025-08-26T20:22:56.2127685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2127769Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2128083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2128172Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2128447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2128521Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2128801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2128909Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2129178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2129267Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2129270Z 2025-08-26T20:22:56.2129368Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2129717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2129785Z layer_outputs = layer_module( 2025-08-26T20:22:56.2130002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2130077Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2130344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2130464Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2130715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2130795Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2131068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2131174Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2131444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2131551Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2131764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2131834Z return self.act(input) 2025-08-26T20:22:56.2131840Z 2025-08-26T20:22:56.2131943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2132272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2132340Z layer_outputs = layer_module( 2025-08-26T20:22:56.2132559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2132633Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2132910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2132988Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2133234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2133311Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2133584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2133710Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2134024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2134111Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2134115Z 2025-08-26T20:22:56.2134214Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2134555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2134632Z layer_outputs = layer_module( 2025-08-26T20:22:56.2134849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2134936Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2135210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2135290Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2135589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2135664Z self_outputs = self.self( 2025-08-26T20:22:56.2135965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2136051Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2136054Z 2025-08-26T20:22:56.2136168Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2136541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2136659Z layer_outputs = layer_module( 2025-08-26T20:22:56.2136909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2136987Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2137286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2137363Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2137657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2137728Z self_outputs = self.self( 2025-08-26T20:22:56.2138016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2138127Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2138483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2138684Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2138687Z 2025-08-26T20:22:56.2138791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2139146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2139216Z layer_outputs = layer_module( 2025-08-26T20:22:56.2139439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2139522Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2139801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2139888Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2140169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2140239Z self_outputs = self.self( 2025-08-26T20:22:56.2140571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2140652Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2140655Z 2025-08-26T20:22:56.2140760Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2141102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2141179Z layer_outputs = layer_module( 2025-08-26T20:22:56.2141395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2141471Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2141754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2141828Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2142104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2142172Z self_outputs = self.self( 2025-08-26T20:22:56.2142441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2142546Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2142876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2143102Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2143105Z 2025-08-26T20:22:56.2143205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2143560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2143630Z layer_outputs = layer_module( 2025-08-26T20:22:56.2143844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2143929Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2144201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2144279Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2144554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2144631Z self_outputs = self.self( 2025-08-26T20:22:56.2144910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2145014Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2145362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2145548Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2145551Z 2025-08-26T20:22:56.2145659Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2146010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2146090Z layer_outputs = layer_module( 2025-08-26T20:22:56.2146311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2146390Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2146733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2146812Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2147102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2147172Z self_outputs = self.self( 2025-08-26T20:22:56.2147452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2147562Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2147906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2148094Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2148098Z 2025-08-26T20:22:56.2148183Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2148270Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2148351Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2148431Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2148545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2148920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2149001Z layer_outputs = layer_module( 2025-08-26T20:22:56.2149275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2149360Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2149666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2149749Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2150054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2150128Z self_outputs = self.self( 2025-08-26T20:22:56.2150423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2150554Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2150890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2151045Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2151373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2151536Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2151540Z 2025-08-26T20:22:56.2151621Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2151724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2152082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2152154Z layer_outputs = layer_module( 2025-08-26T20:22:56.2152392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2152472Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2152753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2152826Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2153135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2153215Z self_outputs = self.self( 2025-08-26T20:22:56.2153498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2153583Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2153586Z 2025-08-26T20:22:56.2153696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2154069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2154156Z layer_outputs = layer_module( 2025-08-26T20:22:56.2154391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2154483Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2154788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2154876Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2155177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2155253Z self_outputs = self.self( 2025-08-26T20:22:56.2155559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2155681Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2155685Z 2025-08-26T20:22:56.2155802Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2156178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2156257Z layer_outputs = layer_module( 2025-08-26T20:22:56.2156499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2156583Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2156885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2156964Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2157269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2157346Z self_outputs = self.self( 2025-08-26T20:22:56.2157643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2157740Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2157744Z 2025-08-26T20:22:56.2157855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2158233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2158308Z layer_outputs = layer_module( 2025-08-26T20:22:56.2158540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2158630Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2158928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2159016Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2159396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2159483Z self_outputs = self.self( 2025-08-26T20:22:56.2159831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2159960Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2160347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2160535Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2160756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2160865Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2160871Z 2025-08-26T20:22:56.2160991Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2161371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2161448Z layer_outputs = layer_module( 2025-08-26T20:22:56.2161691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2161774Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2162084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2162164Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2162464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2162585Z self_outputs = self.self( 2025-08-26T20:22:56.2162881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2163012Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2163388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2163541Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2163879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2163978Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2164189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2164295Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2164301Z 2025-08-26T20:22:56.2164418Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2164793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2164876Z layer_outputs = layer_module( 2025-08-26T20:22:56.2165110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2165196Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2165508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2165586Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2165948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2166019Z self_outputs = self.self( 2025-08-26T20:22:56.2166291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2166447Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2166793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2166951Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2166955Z 2025-08-26T20:22:56.2167055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2167404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2167478Z layer_outputs = layer_module( 2025-08-26T20:22:56.2167696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2167779Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2168057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2168137Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2168413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2168482Z self_outputs = self.self( 2025-08-26T20:22:56.2168761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2168871Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2169276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2169423Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2169427Z 2025-08-26T20:22:56.2169537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2169876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2169946Z layer_outputs = layer_module( 2025-08-26T20:22:56.2170172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2170249Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2170531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2170606Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2170884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2170964Z self_outputs = self.self( 2025-08-26T20:22:56.2171230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2171415Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2171419Z 2025-08-26T20:22:56.2171516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2171853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2171921Z layer_outputs = layer_module( 2025-08-26T20:22:56.2172141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2172219Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2172485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2172599Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2172871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2172986Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2173254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2173337Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2173340Z 2025-08-26T20:22:56.2173445Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2173781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2173859Z layer_outputs = layer_module( 2025-08-26T20:22:56.2174070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2174152Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2174417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2174497Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2174755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2174828Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2175107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2175253Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2175524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2175614Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2175618Z 2025-08-26T20:22:56.2175718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2176068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2176138Z layer_outputs = layer_module( 2025-08-26T20:22:56.2176358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2176435Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2176711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2176799Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2177066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2177151Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2177423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2177564Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2177830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2177937Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2178149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2178221Z return self.act(input) 2025-08-26T20:22:56.2178226Z 2025-08-26T20:22:56.2178331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2178703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2178776Z layer_outputs = layer_module( 2025-08-26T20:22:56.2179002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2179078Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2179364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2179446Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2179711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2179787Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2180073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2180203Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2180483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2180571Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2180574Z 2025-08-26T20:22:56.2180674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2181019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2181098Z layer_outputs = layer_module( 2025-08-26T20:22:56.2181355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2181438Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2181706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2181786Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2182053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2182121Z self_outputs = self.self( 2025-08-26T20:22:56.2182391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2182468Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2182472Z 2025-08-26T20:22:56.2182575Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2182909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2182984Z layer_outputs = layer_module( 2025-08-26T20:22:56.2183195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2183269Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2183547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2183618Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2183889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2183956Z self_outputs = self.self( 2025-08-26T20:22:56.2184219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2184326Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2184647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2184877Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2184881Z 2025-08-26T20:22:56.2184981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2185332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2185403Z layer_outputs = layer_module( 2025-08-26T20:22:56.2185618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2185705Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2185985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2186064Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2186350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2186421Z self_outputs = self.self( 2025-08-26T20:22:56.2186709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2186788Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2186792Z 2025-08-26T20:22:56.2186898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2187252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2187363Z layer_outputs = layer_module( 2025-08-26T20:22:56.2187583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2187660Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2187952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2188026Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2188313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2188383Z self_outputs = self.self( 2025-08-26T20:22:56.2188658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2188766Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2189110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2189301Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2189304Z 2025-08-26T20:22:56.2189407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2189770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2189842Z layer_outputs = layer_module( 2025-08-26T20:22:56.2190069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2190156Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2190443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2190529Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2190819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2190906Z self_outputs = self.self( 2025-08-26T20:22:56.2191218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2191319Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2191659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2191836Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2191840Z 2025-08-26T20:22:56.2191947Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2192289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2192367Z layer_outputs = layer_module( 2025-08-26T20:22:56.2192585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2192661Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2192943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2193016Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2193301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2193370Z self_outputs = self.self( 2025-08-26T20:22:56.2193647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2194326Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2194667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2194857Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2194862Z 2025-08-26T20:22:56.2194946Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2195034Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2195115Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2195192Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2195303Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2195652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2195735Z layer_outputs = layer_module( 2025-08-26T20:22:56.2195958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2196039Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2196671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2196753Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2197046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2197118Z self_outputs = self.self( 2025-08-26T20:22:56.2197407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2197521Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2197875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2198030Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2198434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2198597Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2198602Z 2025-08-26T20:22:56.2198681Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2198784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2199144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2199263Z layer_outputs = layer_module( 2025-08-26T20:22:56.2199510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2199591Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2199903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2199985Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2200292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2200385Z self_outputs = self.self( 2025-08-26T20:22:56.2200664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2200744Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2200748Z 2025-08-26T20:22:56.2200852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2201271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2201345Z layer_outputs = layer_module( 2025-08-26T20:22:56.2201568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2201653Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2201938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2202020Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2202294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2202361Z self_outputs = self.self( 2025-08-26T20:22:56.2202635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2202718Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2202722Z 2025-08-26T20:22:56.2202828Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2203175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2203250Z layer_outputs = layer_module( 2025-08-26T20:22:56.2203465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2203542Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2203826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2203899Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2204183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2204252Z self_outputs = self.self( 2025-08-26T20:22:56.2204523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2204648Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2204653Z 2025-08-26T20:22:56.2204754Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2205102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2205172Z layer_outputs = layer_module( 2025-08-26T20:22:56.2205394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2205470Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2205744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2205825Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2206104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2206181Z self_outputs = self.self( 2025-08-26T20:22:56.2206452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2206570Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2206920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2207092Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2207325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2207423Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2207427Z 2025-08-26T20:22:56.2207536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2207881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2207951Z layer_outputs = layer_module( 2025-08-26T20:22:56.2208177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2208256Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2208536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2208614Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2208896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2208965Z self_outputs = self.self( 2025-08-26T20:22:56.2209244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2209365Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2209716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2209852Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2210157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2210247Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2210448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2210546Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2210550Z 2025-08-26T20:22:56.2210661Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2211061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2211140Z layer_outputs = layer_module( 2025-08-26T20:22:56.2211356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2211433Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2211725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2211804Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2212094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2212176Z self_outputs = self.self( 2025-08-26T20:22:56.2212452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2212574Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2212921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2213078Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2213082Z 2025-08-26T20:22:56.2213182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2213534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2213638Z layer_outputs = layer_module( 2025-08-26T20:22:56.2213858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2213943Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2214230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2214314Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2214599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2214678Z self_outputs = self.self( 2025-08-26T20:22:56.2214960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2215078Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2215442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2215596Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2215600Z 2025-08-26T20:22:56.2215711Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2216068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2216144Z layer_outputs = layer_module( 2025-08-26T20:22:56.2216367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2216444Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2216738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2216814Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2217113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2217226Z self_outputs = self.self( 2025-08-26T20:22:56.2217495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2217685Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2217689Z 2025-08-26T20:22:56.2217790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2218136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2218208Z layer_outputs = layer_module( 2025-08-26T20:22:56.2218435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2218514Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2218798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2218881Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2219161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2219281Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2219566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2219649Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2219700Z 2025-08-26T20:22:56.2219804Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2220158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2220236Z layer_outputs = layer_module( 2025-08-26T20:22:56.2220459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2220544Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2220826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2220910Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2221181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2221262Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2221560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2221668Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2221956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2222093Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2222103Z 2025-08-26T20:22:56.2222226Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2222588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2222659Z layer_outputs = layer_module( 2025-08-26T20:22:56.2222887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2222968Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2223249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2223338Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2223644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2223731Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2224016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2224134Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2224427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2224541Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2224759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2224830Z return self.act(input) 2025-08-26T20:22:56.2224834Z 2025-08-26T20:22:56.2224941Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2225285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2225358Z layer_outputs = layer_module( 2025-08-26T20:22:56.2225577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2225653Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2225939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2226060Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2226329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2226414Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2226691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2226820Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2227098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2227187Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2227190Z 2025-08-26T20:22:56.2227288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2227665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2227742Z layer_outputs = layer_module( 2025-08-26T20:22:56.2227964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2228051Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2228348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2228428Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2228704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2228773Z self_outputs = self.self( 2025-08-26T20:22:56.2229051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2229132Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2229138Z 2025-08-26T20:22:56.2229245Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2229586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2229697Z layer_outputs = layer_module( 2025-08-26T20:22:56.2229915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2229991Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2230270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2230344Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2230621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2230696Z self_outputs = self.self( 2025-08-26T20:22:56.2230973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2231084Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2231427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2231625Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2231629Z 2025-08-26T20:22:56.2231731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2232088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2232160Z layer_outputs = layer_module( 2025-08-26T20:22:56.2232417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2232504Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2232785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2232870Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2233151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2233230Z self_outputs = self.self( 2025-08-26T20:22:56.2233512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2233590Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2233594Z 2025-08-26T20:22:56.2233704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2234056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2234136Z layer_outputs = layer_module( 2025-08-26T20:22:56.2234356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2234440Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2234744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2234823Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2235125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2235199Z self_outputs = self.self( 2025-08-26T20:22:56.2235498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2235611Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2235972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2236214Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2236219Z 2025-08-26T20:22:56.2236329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2236706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2236779Z layer_outputs = layer_module( 2025-08-26T20:22:56.2237020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2237104Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2237409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2237496Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2237800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2237882Z self_outputs = self.self( 2025-08-26T20:22:56.2238180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2238288Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2238653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2238848Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2238892Z 2025-08-26T20:22:56.2239009Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2239661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2239758Z layer_outputs = layer_module( 2025-08-26T20:22:56.2240000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2240098Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2240413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2240494Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2240807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2240886Z self_outputs = self.self( 2025-08-26T20:22:56.2241165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2241276Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2241605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2241791Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2241795Z 2025-08-26T20:22:56.2241876Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2241963Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2242040Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2242118Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2242227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2242568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2242648Z layer_outputs = layer_module( 2025-08-26T20:22:56.2242919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2242999Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2243280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2243354Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2243637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2243707Z self_outputs = self.self( 2025-08-26T20:22:56.2243991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2244106Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2244445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2244599Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2244925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2245088Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2245092Z 2025-08-26T20:22:56.2245172Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2245282Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2245634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2245747Z layer_outputs = layer_module( 2025-08-26T20:22:56.2245983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2246063Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2246359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2246438Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2246738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2246823Z self_outputs = self.self( 2025-08-26T20:22:56.2247119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2247203Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2247206Z 2025-08-26T20:22:56.2247306Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2247658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2247733Z layer_outputs = layer_module( 2025-08-26T20:22:56.2247959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2248046Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2248334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2248417Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2248705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2248779Z self_outputs = self.self( 2025-08-26T20:22:56.2249067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2249147Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2249150Z 2025-08-26T20:22:56.2249304Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2249656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2249736Z layer_outputs = layer_module( 2025-08-26T20:22:56.2249954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2250033Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2250319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2250394Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2250678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2250749Z self_outputs = self.self( 2025-08-26T20:22:56.2251028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2251119Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2251123Z 2025-08-26T20:22:56.2251224Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2251576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2251647Z layer_outputs = layer_module( 2025-08-26T20:22:56.2251872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2251984Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2252265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2252345Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2252625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2252703Z self_outputs = self.self( 2025-08-26T20:22:56.2252981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2253098Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2253455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2253632Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2253832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2253929Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2253935Z 2025-08-26T20:22:56.2254046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2254396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2254469Z layer_outputs = layer_module( 2025-08-26T20:22:56.2254697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2254777Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2255075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2255157Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2255463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2255580Z self_outputs = self.self( 2025-08-26T20:22:56.2255877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2256009Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2256392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2256538Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2256861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2256964Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2257160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2257260Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2257264Z 2025-08-26T20:22:56.2257377Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2257728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2257808Z layer_outputs = layer_module( 2025-08-26T20:22:56.2258031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2258109Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2258397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2258511Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2258799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2258872Z self_outputs = self.self( 2025-08-26T20:22:56.2259165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2259282Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2259639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2259803Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2259807Z 2025-08-26T20:22:56.2259912Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2260276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2260348Z layer_outputs = layer_module( 2025-08-26T20:22:56.2260573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2260651Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2260938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2261018Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2261291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2261365Z self_outputs = self.self( 2025-08-26T20:22:56.2261640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2261752Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2262150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2262307Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2262311Z 2025-08-26T20:22:56.2262423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2262775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2262855Z layer_outputs = layer_module( 2025-08-26T20:22:56.2263077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2263159Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2263445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2263522Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2263819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2263888Z self_outputs = self.self( 2025-08-26T20:22:56.2264160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2264348Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2264352Z 2025-08-26T20:22:56.2264452Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2264862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2264934Z layer_outputs = layer_module( 2025-08-26T20:22:56.2265165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2265249Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2265541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2265623Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2265898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2266015Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2266296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2266391Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2266395Z 2025-08-26T20:22:56.2266496Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2266858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2266936Z layer_outputs = layer_module( 2025-08-26T20:22:56.2267153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2267235Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2267511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2267594Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2267858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2267935Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2268219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2268364Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2268647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2268728Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2268731Z 2025-08-26T20:22:56.2268831Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2269180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2269254Z layer_outputs = layer_module( 2025-08-26T20:22:56.2269478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2269554Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2269841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2269923Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2270178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2270260Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2270535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2270649Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2270923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2271072Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2271295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2271370Z return self.act(input) 2025-08-26T20:22:56.2271373Z 2025-08-26T20:22:56.2271483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2271836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2271917Z layer_outputs = layer_module( 2025-08-26T20:22:56.2272140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2272219Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2272508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2272595Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2272870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2272948Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2273223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2273350Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2273631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2273720Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2273724Z 2025-08-26T20:22:56.2273827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2274188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2274261Z layer_outputs = layer_module( 2025-08-26T20:22:56.2274518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2274609Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2274890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2274974Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2275261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2275330Z self_outputs = self.self( 2025-08-26T20:22:56.2275610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2275694Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2275698Z 2025-08-26T20:22:56.2275805Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2276158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2276236Z layer_outputs = layer_module( 2025-08-26T20:22:56.2276458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2276540Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2276848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2276928Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2277274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2277349Z self_outputs = self.self( 2025-08-26T20:22:56.2277658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2277774Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2278117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2278323Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2278327Z 2025-08-26T20:22:56.2278434Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2278812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2278890Z layer_outputs = layer_module( 2025-08-26T20:22:56.2279128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2279287Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2279602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2279690Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2279990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2280075Z self_outputs = self.self( 2025-08-26T20:22:56.2280370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2280454Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2280461Z 2025-08-26T20:22:56.2280580Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2280953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2281036Z layer_outputs = layer_module( 2025-08-26T20:22:56.2281316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2281409Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2281710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2281790Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2282095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2282175Z self_outputs = self.self( 2025-08-26T20:22:56.2282479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2282588Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2282955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2283161Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2283164Z 2025-08-26T20:22:56.2283273Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2283648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2283724Z layer_outputs = layer_module( 2025-08-26T20:22:56.2283964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2284083Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2284380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2284470Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2284766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2284846Z self_outputs = self.self( 2025-08-26T20:22:56.2285150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2285256Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2285627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2285824Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2285828Z 2025-08-26T20:22:56.2285942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2286394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2286473Z layer_outputs = layer_module( 2025-08-26T20:22:56.2286707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2286791Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2287096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2287175Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2287483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2287558Z self_outputs = self.self( 2025-08-26T20:22:56.2287870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2288011Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2288377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2288578Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2288582Z 2025-08-26T20:22:56.2288670Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2288764Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2288848Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2288933Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2289051Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2289426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2289509Z layer_outputs = layer_module( 2025-08-26T20:22:56.2289743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2289842Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2290121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2290194Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2290478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2290626Z self_outputs = self.self( 2025-08-26T20:22:56.2290914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2291027Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2291370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2291522Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2291849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2292009Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2292013Z 2025-08-26T20:22:56.2292093Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2292205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2292562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2292636Z layer_outputs = layer_module( 2025-08-26T20:22:56.2292867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2292945Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2293233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2293309Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2293587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2293668Z self_outputs = self.self( 2025-08-26T20:22:56.2293947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2294035Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2294039Z 2025-08-26T20:22:56.2294142Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2294538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2294614Z layer_outputs = layer_module( 2025-08-26T20:22:56.2294833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2294920Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2295201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2295284Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2295570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2295641Z self_outputs = self.self( 2025-08-26T20:22:56.2295932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2296016Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2296019Z 2025-08-26T20:22:56.2296129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2296622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2296713Z layer_outputs = layer_module( 2025-08-26T20:22:56.2296950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2297033Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2297425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2297506Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2297811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2297886Z self_outputs = self.self( 2025-08-26T20:22:56.2298187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2298283Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2298287Z 2025-08-26T20:22:56.2298389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2298749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2298825Z layer_outputs = layer_module( 2025-08-26T20:22:56.2299054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2299134Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2299419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2299510Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2299810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2299892Z self_outputs = self.self( 2025-08-26T20:22:56.2300193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2300320Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2300680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2300856Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2301116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2301220Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2301223Z 2025-08-26T20:22:56.2301332Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2301688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2301769Z layer_outputs = layer_module( 2025-08-26T20:22:56.2301993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2302074Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2302361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2302437Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2302727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2302799Z self_outputs = self.self( 2025-08-26T20:22:56.2303076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2303200Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2303558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2303702Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2304064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2304166Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2304365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2304466Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2304470Z 2025-08-26T20:22:56.2304587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2304954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2305039Z layer_outputs = layer_module( 2025-08-26T20:22:56.2305278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2305360Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2305651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2305730Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2306037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2306113Z self_outputs = self.self( 2025-08-26T20:22:56.2306416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2306537Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2306910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2307084Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2307088Z 2025-08-26T20:22:56.2307198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2307618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2307695Z layer_outputs = layer_module( 2025-08-26T20:22:56.2307938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2308021Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2308318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2308406Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2308701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2308786Z self_outputs = self.self( 2025-08-26T20:22:56.2309085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2309206Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2309588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2309746Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2309750Z 2025-08-26T20:22:56.2309867Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2310235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2310358Z layer_outputs = layer_module( 2025-08-26T20:22:56.2310595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2310677Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2310987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2311066Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2311373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2311450Z self_outputs = self.self( 2025-08-26T20:22:56.2311752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2311948Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2311955Z 2025-08-26T20:22:56.2312063Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2312447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2312524Z layer_outputs = layer_module( 2025-08-26T20:22:56.2312767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2312849Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2313146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2313232Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2313527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2313654Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2313953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2314051Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2314055Z 2025-08-26T20:22:56.2314215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2314587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2314669Z layer_outputs = layer_module( 2025-08-26T20:22:56.2314902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2314993Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2315292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2315391Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2315671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2315752Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2316066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2316185Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2316491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2316578Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2316581Z 2025-08-26T20:22:56.2316689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2317068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2317184Z layer_outputs = layer_module( 2025-08-26T20:22:56.2317426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2317509Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2317816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2317903Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2318184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2318275Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2318577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2318705Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2319004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2319124Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2319423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2319506Z return self.act(input) 2025-08-26T20:22:56.2319510Z 2025-08-26T20:22:56.2319625Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2319999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2320087Z layer_outputs = layer_module( 2025-08-26T20:22:56.2320324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2320409Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2320713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2320801Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2321127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2321210Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2321514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2321653Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2321951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2322050Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2322053Z 2025-08-26T20:22:56.2322162Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2322547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2322629Z layer_outputs = layer_module( 2025-08-26T20:22:56.2322864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2322955Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2323242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2323325Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2323596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2323708Z self_outputs = self.self( 2025-08-26T20:22:56.2323986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2324065Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2324069Z 2025-08-26T20:22:56.2324179Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2324527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2324605Z layer_outputs = layer_module( 2025-08-26T20:22:56.2324822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2324899Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2325181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2325261Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2325544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2325614Z self_outputs = self.self( 2025-08-26T20:22:56.2325908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2326009Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2326352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2326541Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2326544Z 2025-08-26T20:22:56.2326645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2327003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2327074Z layer_outputs = layer_module( 2025-08-26T20:22:56.2327340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2327419Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2327690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2327772Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2328042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2328120Z self_outputs = self.self( 2025-08-26T20:22:56.2328390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2328471Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2328475Z 2025-08-26T20:22:56.2328583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2328928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2329005Z layer_outputs = layer_module( 2025-08-26T20:22:56.2329220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2329302Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2329581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2329654Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2329934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2330041Z self_outputs = self.self( 2025-08-26T20:22:56.2330325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2330428Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2330766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2330955Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2330959Z 2025-08-26T20:22:56.2331057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2331416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2331489Z layer_outputs = layer_module( 2025-08-26T20:22:56.2331714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2331790Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2332070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2332152Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2332441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2332519Z self_outputs = self.self( 2025-08-26T20:22:56.2332814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2332920Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2333266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2333446Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2333449Z 2025-08-26T20:22:56.2333595Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2333941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2334019Z layer_outputs = layer_module( 2025-08-26T20:22:56.2334233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2334316Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2334593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2334668Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2334945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2335014Z self_outputs = self.self( 2025-08-26T20:22:56.2335291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2335390Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2335720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2335905Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2335908Z 2025-08-26T20:22:56.2335988Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2336114Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2336191Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2336265Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2336372Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2336720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2336797Z layer_outputs = layer_module( 2025-08-26T20:22:56.2337016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2337100Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2337377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2337450Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2337735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2337805Z self_outputs = self.self( 2025-08-26T20:22:56.2338076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2338184Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2338509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2338654Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2338963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2339115Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2339121Z 2025-08-26T20:22:56.2339196Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2339299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2339639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2339755Z layer_outputs = layer_module( 2025-08-26T20:22:56.2339977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2340052Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2340330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2340402Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2340678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2340748Z self_outputs = self.self( 2025-08-26T20:22:56.2341016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2341095Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2341098Z 2025-08-26T20:22:56.2341198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2341542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2341611Z layer_outputs = layer_module( 2025-08-26T20:22:56.2341823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2341905Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2342178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2342297Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2342567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2342643Z self_outputs = self.self( 2025-08-26T20:22:56.2342920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2343000Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2343004Z 2025-08-26T20:22:56.2343114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2343461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2343538Z layer_outputs = layer_module( 2025-08-26T20:22:56.2343756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2343837Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2344121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2344198Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2344488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2344558Z self_outputs = self.self( 2025-08-26T20:22:56.2344847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2344932Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2344936Z 2025-08-26T20:22:56.2345037Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2345402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2345481Z layer_outputs = layer_module( 2025-08-26T20:22:56.2345730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2345851Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2346150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2346237Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2346540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2346634Z self_outputs = self.self( 2025-08-26T20:22:56.2346916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2347044Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2347397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2347574Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2347778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2347884Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2347888Z 2025-08-26T20:22:56.2348004Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2348384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2348470Z layer_outputs = layer_module( 2025-08-26T20:22:56.2348753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2348836Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2349150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2349233Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2349551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2349628Z self_outputs = self.self( 2025-08-26T20:22:56.2349932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2350055Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2350396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2350542Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2350856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2350954Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2351157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2351262Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2351266Z 2025-08-26T20:22:56.2351384Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2351780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2351862Z layer_outputs = layer_module( 2025-08-26T20:22:56.2352101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2352191Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2352544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2352629Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2352954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2353031Z self_outputs = self.self( 2025-08-26T20:22:56.2353348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2353483Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2353852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2354027Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2354031Z 2025-08-26T20:22:56.2354140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2354524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2354601Z layer_outputs = layer_module( 2025-08-26T20:22:56.2354843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2354926Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2355240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2355367Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2355673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2355758Z self_outputs = self.self( 2025-08-26T20:22:56.2356075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2356207Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2356585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2356749Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2356753Z 2025-08-26T20:22:56.2356874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2357254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2357342Z layer_outputs = layer_module( 2025-08-26T20:22:56.2357583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2357669Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2357988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2358069Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2358388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2358461Z self_outputs = self.self( 2025-08-26T20:22:56.2358774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2358978Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2358982Z 2025-08-26T20:22:56.2359093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2359615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2359702Z layer_outputs = layer_module( 2025-08-26T20:22:56.2359952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2360039Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2360359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2360439Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2360755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2360892Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2361201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2361312Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2361317Z 2025-08-26T20:22:56.2361432Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2361814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2361900Z layer_outputs = layer_module( 2025-08-26T20:22:56.2362155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2362249Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2362610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2362711Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2363003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2363090Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2363416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2363536Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2363859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2363948Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2363952Z 2025-08-26T20:22:56.2364070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2364463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2364541Z layer_outputs = layer_module( 2025-08-26T20:22:56.2364795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2364879Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2365196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2365285Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2365571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2365663Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2365973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2366101Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2366414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2366582Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2366820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2366900Z return self.act(input) 2025-08-26T20:22:56.2366904Z 2025-08-26T20:22:56.2367023Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2367408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2367494Z layer_outputs = layer_module( 2025-08-26T20:22:56.2367737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2367821Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2368150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2368241Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2368534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2368615Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2368932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2369057Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2369338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2369464Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2369467Z 2025-08-26T20:22:56.2369574Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2369966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2370044Z layer_outputs = layer_module( 2025-08-26T20:22:56.2370287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2370372Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2370658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2370742Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2371027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2371106Z self_outputs = self.self( 2025-08-26T20:22:56.2371388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2371472Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2371476Z 2025-08-26T20:22:56.2371585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2371937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2372013Z layer_outputs = layer_module( 2025-08-26T20:22:56.2372233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2372317Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2372597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2372673Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2372994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2373069Z self_outputs = self.self( 2025-08-26T20:22:56.2373357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2373464Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2373828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2374033Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2374039Z 2025-08-26T20:22:56.2374148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2374526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2374603Z layer_outputs = layer_module( 2025-08-26T20:22:56.2374847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2374928Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2375227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2375314Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2375621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2375739Z self_outputs = self.self( 2025-08-26T20:22:56.2376033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2376116Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2376127Z 2025-08-26T20:22:56.2376236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2376607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2376691Z layer_outputs = layer_module( 2025-08-26T20:22:56.2376925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2377013Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2377311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2377392Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2377698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2377773Z self_outputs = self.self( 2025-08-26T20:22:56.2378073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2378183Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2378547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2378748Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2378752Z 2025-08-26T20:22:56.2378863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2379265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2379343Z layer_outputs = layer_module( 2025-08-26T20:22:56.2379584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2379702Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2380003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2380093Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2380389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2380472Z self_outputs = self.self( 2025-08-26T20:22:56.2380775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2380896Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2381278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2381475Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2381479Z 2025-08-26T20:22:56.2381596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2381967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2382051Z layer_outputs = layer_module( 2025-08-26T20:22:56.2382285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2382376Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2382716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2382795Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2383112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2383186Z self_outputs = self.self( 2025-08-26T20:22:56.2383491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2383599Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2383963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2384161Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2384168Z 2025-08-26T20:22:56.2384255Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2384349Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2384434Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2384523Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2384637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2385030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2385116Z layer_outputs = layer_module( 2025-08-26T20:22:56.2385361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2385453Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2385771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2385855Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2386166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2386241Z self_outputs = self.self( 2025-08-26T20:22:56.2386584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2386705Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2387079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2387234Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2387589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2387764Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2387768Z 2025-08-26T20:22:56.2387853Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2387972Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2388368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2388455Z layer_outputs = layer_module( 2025-08-26T20:22:56.2388698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2388782Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2389114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2389193Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2389544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2389619Z self_outputs = self.self( 2025-08-26T20:22:56.2389919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2390008Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2390012Z 2025-08-26T20:22:56.2390124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2390523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2390603Z layer_outputs = layer_module( 2025-08-26T20:22:56.2390857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2390945Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2391255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2391344Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2391661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2391747Z self_outputs = self.self( 2025-08-26T20:22:56.2392052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2392142Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2392146Z 2025-08-26T20:22:56.2392270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2392651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2392737Z layer_outputs = layer_module( 2025-08-26T20:22:56.2392981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2393072Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2393430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2393513Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2393826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2393902Z self_outputs = self.self( 2025-08-26T20:22:56.2394211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2394303Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2394307Z 2025-08-26T20:22:56.2394423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2394818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2394897Z layer_outputs = layer_module( 2025-08-26T20:22:56.2395149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2395232Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2395547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2395629Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2395937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2396019Z self_outputs = self.self( 2025-08-26T20:22:56.2396488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2396633Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2397035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2397228Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2397447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2397557Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2397561Z 2025-08-26T20:22:56.2397681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2398067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2398158Z layer_outputs = layer_module( 2025-08-26T20:22:56.2398402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2398489Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2398814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2398896Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2399208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2399345Z self_outputs = self.self( 2025-08-26T20:22:56.2399668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2399795Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2400185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2400339Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2400771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2400883Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2401097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2401205Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2401217Z 2025-08-26T20:22:56.2401341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2401725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2401816Z layer_outputs = layer_module( 2025-08-26T20:22:56.2402062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2402157Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2402469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2402553Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2402874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2402953Z self_outputs = self.self( 2025-08-26T20:22:56.2403266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2403453Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2403845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2404017Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2404023Z 2025-08-26T20:22:56.2404136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2404529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2404607Z layer_outputs = layer_module( 2025-08-26T20:22:56.2404855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2404941Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2405259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2405344Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2405652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2405739Z self_outputs = self.self( 2025-08-26T20:22:56.2406044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2406176Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2406561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2406724Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2406736Z 2025-08-26T20:22:56.2406852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2407233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2407320Z layer_outputs = layer_module( 2025-08-26T20:22:56.2407602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2407697Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2408004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2408087Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2408402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2408478Z self_outputs = self.self( 2025-08-26T20:22:56.2408796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2409000Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2409004Z 2025-08-26T20:22:56.2409121Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2409510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2409587Z layer_outputs = layer_module( 2025-08-26T20:22:56.2409839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2409919Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2410207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2410318Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2410602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2410720Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2411006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2411096Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2411100Z 2025-08-26T20:22:56.2411200Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2411561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2411633Z layer_outputs = layer_module( 2025-08-26T20:22:56.2411860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2411950Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2412238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2412328Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2412597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2412676Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2412981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2413098Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2413412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2413503Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2413507Z 2025-08-26T20:22:56.2413622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2413997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2414119Z layer_outputs = layer_module( 2025-08-26T20:22:56.2414367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2414449Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2414753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2414843Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2415131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2415217Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2415521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2415651Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2415936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2416056Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2416275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2416350Z return self.act(input) 2025-08-26T20:22:56.2416353Z 2025-08-26T20:22:56.2416467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2416825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2416943Z layer_outputs = layer_module( 2025-08-26T20:22:56.2417168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2417254Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2417538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2417621Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2417892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2417967Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2418259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2418387Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2418671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2418761Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2418764Z 2025-08-26T20:22:56.2418872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2419229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2419302Z layer_outputs = layer_module( 2025-08-26T20:22:56.2419531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2419610Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2419894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2419982Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2420336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2420416Z self_outputs = self.self( 2025-08-26T20:22:56.2420742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2420829Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2420840Z 2025-08-26T20:22:56.2420942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2421295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2421378Z layer_outputs = layer_module( 2025-08-26T20:22:56.2421600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2421687Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2421968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2422044Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2422335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2422407Z self_outputs = self.self( 2025-08-26T20:22:56.2422697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2422798Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2423147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2423373Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2423377Z 2025-08-26T20:22:56.2423480Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2423843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2423916Z layer_outputs = layer_module( 2025-08-26T20:22:56.2424148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2424227Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2424510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2424594Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2424877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2424959Z self_outputs = self.self( 2025-08-26T20:22:56.2425240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2425327Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2425331Z 2025-08-26T20:22:56.2425433Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2425785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2425864Z layer_outputs = layer_module( 2025-08-26T20:22:56.2426085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2426168Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2426455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2426537Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2426816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2426920Z self_outputs = self.self( 2025-08-26T20:22:56.2427207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2427309Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2427662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2427848Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2427854Z 2025-08-26T20:22:56.2427956Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2428322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2428394Z layer_outputs = layer_module( 2025-08-26T20:22:56.2428624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2428703Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2428992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2429068Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2429348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2429426Z self_outputs = self.self( 2025-08-26T20:22:56.2429745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2429855Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2430197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2430388Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2430392Z 2025-08-26T20:22:56.2430494Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2430846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2430928Z layer_outputs = layer_module( 2025-08-26T20:22:56.2431149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2431239Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2431527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2431601Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2431880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2431949Z self_outputs = self.self( 2025-08-26T20:22:56.2432224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2432323Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2432665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2432852Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2432855Z 2025-08-26T20:22:56.2432940Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2433028Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2433106Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2433229Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2433333Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2433683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2433762Z layer_outputs = layer_module( 2025-08-26T20:22:56.2433985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2434072Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2434359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2434444Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2434755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2434830Z self_outputs = self.self( 2025-08-26T20:22:56.2435133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2435249Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2435625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2435779Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2436165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2436333Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2436338Z 2025-08-26T20:22:56.2436421Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2436542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2436916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2437000Z layer_outputs = layer_module( 2025-08-26T20:22:56.2437239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2437322Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2437631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2437713Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2438018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2438092Z self_outputs = self.self( 2025-08-26T20:22:56.2438389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2438476Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2438479Z 2025-08-26T20:22:56.2438587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2438975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2439050Z layer_outputs = layer_module( 2025-08-26T20:22:56.2439372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2439464Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2439775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2439924Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2440243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2440329Z self_outputs = self.self( 2025-08-26T20:22:56.2440635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2440722Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2440735Z 2025-08-26T20:22:56.2440860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2441204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2441285Z layer_outputs = layer_module( 2025-08-26T20:22:56.2441546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2441632Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2441910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2441983Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2442268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2442335Z self_outputs = self.self( 2025-08-26T20:22:56.2442615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2442736Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2442743Z 2025-08-26T20:22:56.2442853Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2443210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2443282Z layer_outputs = layer_module( 2025-08-26T20:22:56.2443516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2443593Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2443884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2443959Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2444243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2444324Z self_outputs = self.self( 2025-08-26T20:22:56.2444604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2444727Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2445083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2445265Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2445457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2445557Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2445560Z 2025-08-26T20:22:56.2445671Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2446029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2446108Z layer_outputs = layer_module( 2025-08-26T20:22:56.2446380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2446468Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2446750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2446825Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2447114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2447185Z self_outputs = self.self( 2025-08-26T20:22:56.2447473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2447594Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2447950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2448099Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2448422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2448523Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2448728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2448840Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2448843Z 2025-08-26T20:22:56.2448953Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2449361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2449443Z layer_outputs = layer_module( 2025-08-26T20:22:56.2449671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2449759Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2450039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2450116Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2450406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2450475Z self_outputs = self.self( 2025-08-26T20:22:56.2450761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2450880Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2451239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2451394Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2451398Z 2025-08-26T20:22:56.2451499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2451860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2451932Z layer_outputs = layer_module( 2025-08-26T20:22:56.2452162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2452243Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2452535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2452610Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2452929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2453009Z self_outputs = self.self( 2025-08-26T20:22:56.2453291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2453419Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2453797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2453961Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2453972Z 2025-08-26T20:22:56.2454082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2454473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2454557Z layer_outputs = layer_module( 2025-08-26T20:22:56.2454794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2454885Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2455242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2455317Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2455612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2455718Z self_outputs = self.self( 2025-08-26T20:22:56.2456005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2456195Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2456199Z 2025-08-26T20:22:56.2456308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2456659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2456729Z layer_outputs = layer_module( 2025-08-26T20:22:56.2456957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2457034Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2457323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2457398Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2457675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2457798Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2458081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2458173Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2458176Z 2025-08-26T20:22:56.2458278Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2458632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2458705Z layer_outputs = layer_module( 2025-08-26T20:22:56.2458927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2459012Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2459327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2459421Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2459685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2459770Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2460056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2460166Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2460459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2460543Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2460546Z 2025-08-26T20:22:56.2460653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2461002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2461074Z layer_outputs = layer_module( 2025-08-26T20:22:56.2461302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2461380Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2461667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2461750Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2462069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2462145Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2462437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2462557Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2462842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2462964Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2463183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2463255Z return self.act(input) 2025-08-26T20:22:56.2463266Z 2025-08-26T20:22:56.2463370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2463746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2463831Z layer_outputs = layer_module( 2025-08-26T20:22:56.2464083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2464169Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2464456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2464538Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2464820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2464901Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2465214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2465349Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2465660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2465787Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2465792Z 2025-08-26T20:22:56.2465902Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2466286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2466364Z layer_outputs = layer_module( 2025-08-26T20:22:56.2466605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2466690Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2466994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2467082Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2467385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2467467Z self_outputs = self.self( 2025-08-26T20:22:56.2467764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-26T20:22:56.2467858Z query_vectors = self.query(hidden_states) 2025-08-26T20:22:56.2467861Z 2025-08-26T20:22:56.2467969Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2468338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2468460Z layer_outputs = layer_module( 2025-08-26T20:22:56.2468700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2468796Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2469104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2469185Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2469496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2469571Z self_outputs = self.self( 2025-08-26T20:22:56.2469878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2469986Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2470367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2470566Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2470569Z 2025-08-26T20:22:56.2470683Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2471072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2471147Z layer_outputs = layer_module( 2025-08-26T20:22:56.2471393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2471477Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2471790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2471874Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2472181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2472265Z self_outputs = self.self( 2025-08-26T20:22:56.2472611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-26T20:22:56.2472705Z key_vectors = self.key(hidden_states) 2025-08-26T20:22:56.2472709Z 2025-08-26T20:22:56.2472819Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2473192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2473277Z layer_outputs = layer_module( 2025-08-26T20:22:56.2473514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2473609Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2473909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2473996Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2474294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2474368Z self_outputs = self.self( 2025-08-26T20:22:56.2474673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2474780Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2475149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2475378Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2475382Z 2025-08-26T20:22:56.2475499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2475873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2475950Z layer_outputs = layer_module( 2025-08-26T20:22:56.2476191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2476274Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2476577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2476656Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2476953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2477038Z self_outputs = self.self( 2025-08-26T20:22:56.2477335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2477450Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2477808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2478007Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2478011Z 2025-08-26T20:22:56.2478118Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2478489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2478575Z layer_outputs = layer_module( 2025-08-26T20:22:56.2478815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2478904Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2479329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2479433Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2479736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2479811Z self_outputs = self.self( 2025-08-26T20:22:56.2480117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-26T20:22:56.2480226Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2480598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2480794Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-26T20:22:56.2480798Z 2025-08-26T20:22:56.2480882Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2480977Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2481057Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2481142Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2481249Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2481606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2481687Z layer_outputs = layer_module( 2025-08-26T20:22:56.2481910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2482038Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2482324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2482405Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2482692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2482764Z self_outputs = self.self( 2025-08-26T20:22:56.2483052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-26T20:22:56.2483163Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-26T20:22:56.2483509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-26T20:22:56.2483655Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-26T20:22:56.2483988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-26T20:22:56.2484140Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-26T20:22:56.2484144Z 2025-08-26T20:22:56.2484223Z cudagraph partition due to non gpu ops 2025-08-26T20:22:56.2484335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2484688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2484766Z layer_outputs = layer_module( 2025-08-26T20:22:56.2484992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2485072Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2485361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2485436Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2485761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2485834Z self_outputs = self.self( 2025-08-26T20:22:56.2486122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-26T20:22:56.2486197Z attn_scores += diagonal_mask 2025-08-26T20:22:56.2486200Z 2025-08-26T20:22:56.2486304Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2486662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2486737Z layer_outputs = layer_module( 2025-08-26T20:22:56.2486966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2487044Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2487324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2487404Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2487682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2487760Z self_outputs = self.self( 2025-08-26T20:22:56.2488039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-26T20:22:56.2488125Z attn_probs = nn.functional.softmax( 2025-08-26T20:22:56.2488172Z 2025-08-26T20:22:56.2488278Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2488629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2488709Z layer_outputs = layer_module( 2025-08-26T20:22:56.2488933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2489019Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2489300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2489375Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2489663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2489732Z self_outputs = self.self( 2025-08-26T20:22:56.2490022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-26T20:22:56.2490108Z value_vectors = self.value(hidden_states) 2025-08-26T20:22:56.2490111Z 2025-08-26T20:22:56.2490222Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2490577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2490648Z layer_outputs = layer_module( 2025-08-26T20:22:56.2490880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2490960Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2491246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2491323Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2491605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2491681Z self_outputs = self.self( 2025-08-26T20:22:56.2491997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2492125Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2492478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2492658Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-26T20:22:56.2492856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2492957Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2492971Z 2025-08-26T20:22:56.2493076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2493435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2493518Z layer_outputs = layer_module( 2025-08-26T20:22:56.2493739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2493825Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2494109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2494184Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2494474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2494580Z self_outputs = self.self( 2025-08-26T20:22:56.2494867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2494983Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2495339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2495482Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-26T20:22:56.2495806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-26T20:22:56.2495905Z chunked_hidden_states = nn.functional.pad( 2025-08-26T20:22:56.2496098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-26T20:22:56.2496316Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-26T20:22:56.2496322Z 2025-08-26T20:22:56.2496431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2496783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2496870Z layer_outputs = layer_module( 2025-08-26T20:22:56.2497094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2497185Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2497465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2497549Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2497830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2497908Z self_outputs = self.self( 2025-08-26T20:22:56.2498212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2498334Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2498780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2498946Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2498950Z 2025-08-26T20:22:56.2499068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2499440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2499512Z layer_outputs = layer_module( 2025-08-26T20:22:56.2499745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2499824Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2500116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2500192Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2500465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2500542Z self_outputs = self.self( 2025-08-26T20:22:56.2500815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-26T20:22:56.2500936Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-26T20:22:56.2501277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-26T20:22:56.2501483Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-26T20:22:56.2501486Z 2025-08-26T20:22:56.2501587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2501932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2502011Z layer_outputs = layer_module( 2025-08-26T20:22:56.2502230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2502315Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2502598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2502673Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2502967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-26T20:22:56.2503038Z self_outputs = self.self( 2025-08-26T20:22:56.2503330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-26T20:22:56.2503518Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-26T20:22:56.2503522Z 2025-08-26T20:22:56.2503633Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2503981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2504052Z layer_outputs = layer_module( 2025-08-26T20:22:56.2504283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2504363Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2504650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-26T20:22:56.2504734Z self_attn_outputs = self.attention( 2025-08-26T20:22:56.2505048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-26T20:22:56.2505160Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:22:56.2505435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-26T20:22:56.2505526Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2505530Z 2025-08-26T20:22:56.2505631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2506008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2506084Z layer_outputs = layer_module( 2025-08-26T20:22:56.2506306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2506393Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2506676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2506768Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2507034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2507118Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2507405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2507556Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2507870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-26T20:22:56.2507963Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2507967Z 2025-08-26T20:22:56.2508090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2508479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2508560Z layer_outputs = layer_module( 2025-08-26T20:22:56.2508796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2508877Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2509184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2509274Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2509560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2509641Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2509934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-26T20:22:56.2510047Z intermediate_output = self.intermediate(attn_output) 2025-08-26T20:22:56.2510322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-26T20:22:56.2510442Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:22:56.2510651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:22:56.2510731Z return self.act(input) 2025-08-26T20:22:56.2510735Z 2025-08-26T20:22:56.2510834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:22:56.2511184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-26T20:22:56.2511303Z layer_outputs = layer_module( 2025-08-26T20:22:56.2511524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:22:56.2511609Z return super().__call__(*args, **kwargs) 2025-08-26T20:22:56.2511891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-26T20:22:56.2511974Z layer_output = apply_chunking_to_forward( 2025-08-26T20:22:56.2512247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:22:56.2512324Z return forward_fn(*input_tensors) 2025-08-26T20:22:56.2512608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-26T20:22:56.2512729Z layer_output = self.output(intermediate_output, attn_output) 2025-08-26T20:22:56.2513015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-26T20:22:56.2513095Z hidden_states = self.dense(hidden_states) 2025-08-26T20:22:56.2513099Z 2025-08-26T20:24:07.7404601Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:07.7409502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-08-26T20:24:07.7410115Z prediction_scores = self.lm_head(sequence_output) 2025-08-26T20:24:07.7410602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward 2025-08-26T20:24:07.7411550Z x = self.dense(features) 2025-08-26T20:24:07.7411674Z 2025-08-26T20:24:07.7411795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:07.7412387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-08-26T20:24:07.7412912Z prediction_scores = self.lm_head(sequence_output) 2025-08-26T20:24:07.7413370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward 2025-08-26T20:24:07.7413843Z x = self.decoder(x) 2025-08-26T20:24:07.7416823Z 2025-08-26T20:24:07.7417332Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:07.7418000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1723, in torch_dynamo_resume_in_forward_at_1703 2025-08-26T20:24:07.7418749Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:24:07.7419026Z 2025-08-26T20:24:09.2781026Z Compilation time (from dynamo_timed): 103.551812115 2025-08-26T20:24:09.2996031Z pass 2025-08-26T20:24:09.2996695Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:24:09.2997580Z TIMING: gc:0.00587 entire_frame_compile:103.55181 _recursive_pre_grad_passes:0.02032 _recursive_joint_graph_passes:0.9966 _recursive_post_grad_passes:1.83049 async_compile.wait:2.92503 code_gen:80.8356 inductor_compile:88.27323 backend_compile:98.10175 total_wall_time:103.55181 2025-08-26T20:24:09.2998677Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:57385 | FakeTensor.__torch_dispatch__:16284 | ProxyTorchDispatchMode.__torch_dispatch__:17446 2025-08-26T20:24:09.2999350Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-08-26T20:24:15.8344097Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:24:15.8345170Z from pkg_resources import resource_filename 2025-08-26T20:24:16.4501208Z 2025-08-26T20:24:19.4031987Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:24:19.4032338Z loading model: 0it [00:02, ?it/s] 2025-08-26T20:24:19.4053355Z cpu eval BartForCausalLM 2025-08-26T20:24:21.2304209Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:24:21.9301114Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:24:22.6290422Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:24:30.4961219Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4961774Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4962646Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4962933Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4963215Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4963447Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4965818Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4966077Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4966296Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4966519Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4966741Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4966965Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4967221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.4967638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.4969184Z return mod(**inputs) 2025-08-26T20:24:30.4969620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.4970070Z outputs = self.model.decoder( 2025-08-26T20:24:30.4970502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.4971019Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.4971407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.4971814Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.4972242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.4972698Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.4973163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.4973682Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.4973927Z 2025-08-26T20:24:30.4974054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.4974467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.4974821Z return mod(**inputs) 2025-08-26T20:24:30.4975211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.4975650Z outputs = self.model.decoder( 2025-08-26T20:24:30.4976063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.4976474Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.4976853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.4977247Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.4977665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.4978211Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.4978656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.4979088Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.4979244Z 2025-08-26T20:24:30.4979360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.4979749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.4980104Z return mod(**inputs) 2025-08-26T20:24:30.4980504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.4980935Z outputs = self.model.decoder( 2025-08-26T20:24:30.4981357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.4981785Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.4982164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.4982572Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.4983000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.4983462Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.4983927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.4984354Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.4984553Z 2025-08-26T20:24:30.4984644Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4984880Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4985116Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4985374Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.4985649Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.4986069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.4986480Z return mod(**inputs) 2025-08-26T20:24:30.4986863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.4987428Z outputs = self.model.decoder( 2025-08-26T20:24:30.4988008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.4988453Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.4988851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.4989321Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.4989774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.4990241Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.4990684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.4991141Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.4991624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.4992161Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.4992375Z 2025-08-26T20:24:30.4992493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.4992905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.4993283Z return mod(**inputs) 2025-08-26T20:24:30.4993708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.4994198Z outputs = self.model.decoder( 2025-08-26T20:24:30.4994617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.4995058Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.4995438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.4995840Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.4996458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.4996940Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.4997390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.4997838Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.4998338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.4998851Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.4999031Z 2025-08-26T20:24:30.4999156Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.4999636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.4999998Z return mod(**inputs) 2025-08-26T20:24:30.5000399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5000906Z outputs = self.model.decoder( 2025-08-26T20:24:30.5001321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5001746Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5002130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5002539Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5002969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5003435Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5003879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5004314Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5004475Z 2025-08-26T20:24:30.5004590Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5004995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5005356Z return mod(**inputs) 2025-08-26T20:24:30.5005744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5006177Z outputs = self.model.decoder( 2025-08-26T20:24:30.5006593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5007026Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5007412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5007807Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5008233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5008720Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5008908Z 2025-08-26T20:24:30.5009026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5009407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5009837Z return mod(**inputs) 2025-08-26T20:24:30.5010230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5010655Z outputs = self.model.decoder( 2025-08-26T20:24:30.5011056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5011473Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5011847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5012246Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5012664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5013137Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5013553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5013926Z return self.act(input) 2025-08-26T20:24:30.5014047Z 2025-08-26T20:24:30.5014166Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5014553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5014902Z return mod(**inputs) 2025-08-26T20:24:30.5015299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5015723Z outputs = self.model.decoder( 2025-08-26T20:24:30.5016129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5016579Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5016958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5017370Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5017789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5018207Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5018359Z 2025-08-26T20:24:30.5018474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5018860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5019246Z return mod(**inputs) 2025-08-26T20:24:30.5019637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5020046Z outputs = self.model.decoder( 2025-08-26T20:24:30.5020441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5020855Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5021233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5021623Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5022037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5022466Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5022903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5023402Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5023632Z 2025-08-26T20:24:30.5023752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5024145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5024514Z return mod(**inputs) 2025-08-26T20:24:30.5024963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5025388Z outputs = self.model.decoder( 2025-08-26T20:24:30.5025803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5026223Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5026615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5027053Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5027493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5027943Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5028384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5028827Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5028985Z 2025-08-26T20:24:30.5029101Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5029497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5029851Z return mod(**inputs) 2025-08-26T20:24:30.5030248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5030675Z outputs = self.model.decoder( 2025-08-26T20:24:30.5031089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5031553Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5031928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5032333Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5032759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5033207Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5033648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5034082Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5034247Z 2025-08-26T20:24:30.5034336Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5034573Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5034803Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5035029Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5035286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5035678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5036039Z return mod(**inputs) 2025-08-26T20:24:30.5036430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5036851Z outputs = self.model.decoder( 2025-08-26T20:24:30.5037262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5037734Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5038126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5038528Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5038957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5039752Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5040209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5040730Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5041225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5041764Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5041981Z 2025-08-26T20:24:30.5042095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5042503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5042866Z return mod(**inputs) 2025-08-26T20:24:30.5043254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5043706Z outputs = self.model.decoder( 2025-08-26T20:24:30.5044124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5044562Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5044932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5045329Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5045742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5046180Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5046624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5047122Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5047623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5048115Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5048287Z 2025-08-26T20:24:30.5048411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5048808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5049149Z return mod(**inputs) 2025-08-26T20:24:30.5049536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5049960Z outputs = self.model.decoder( 2025-08-26T20:24:30.5050363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5050773Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5051151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5051541Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5051957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5052406Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5052839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5053340Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5053494Z 2025-08-26T20:24:30.5053605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5053987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5054336Z return mod(**inputs) 2025-08-26T20:24:30.5054719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5055139Z outputs = self.model.decoder( 2025-08-26T20:24:30.5055542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5056004Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5056372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5056767Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5057186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5057656Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5057842Z 2025-08-26T20:24:30.5057961Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5058338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5058691Z return mod(**inputs) 2025-08-26T20:24:30.5059074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5059498Z outputs = self.model.decoder( 2025-08-26T20:24:30.5059897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5060311Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5060683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5061073Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5061488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5061938Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5062409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5062774Z return self.act(input) 2025-08-26T20:24:30.5062892Z 2025-08-26T20:24:30.5063012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5063401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5063740Z return mod(**inputs) 2025-08-26T20:24:30.5064124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5064535Z outputs = self.model.decoder( 2025-08-26T20:24:30.5064937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5065348Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5065724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5066114Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5066525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5066944Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5067091Z 2025-08-26T20:24:30.5067207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5067588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5067934Z return mod(**inputs) 2025-08-26T20:24:30.5068320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5068724Z outputs = self.model.decoder( 2025-08-26T20:24:30.5069127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5069541Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5069912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5070299Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5070741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5071178Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5071612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5072100Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5072316Z 2025-08-26T20:24:30.5072432Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5072818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5073176Z return mod(**inputs) 2025-08-26T20:24:30.5073571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5074004Z outputs = self.model.decoder( 2025-08-26T20:24:30.5074410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5074830Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5075211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5075615Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5076039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5076489Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5076936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5077419Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5077568Z 2025-08-26T20:24:30.5077695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5078091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5078450Z return mod(**inputs) 2025-08-26T20:24:30.5078848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5079401Z outputs = self.model.decoder( 2025-08-26T20:24:30.5079857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5080280Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5080668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5081075Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5081500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5081952Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5082391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5082829Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5082990Z 2025-08-26T20:24:30.5083078Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5083317Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5083553Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5083778Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5084037Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5084434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5084798Z return mod(**inputs) 2025-08-26T20:24:30.5085188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5085611Z outputs = self.model.decoder( 2025-08-26T20:24:30.5086072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5086494Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5086873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5087270Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5087693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5088143Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5088586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5089032Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5089523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5090074Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5090276Z 2025-08-26T20:24:30.5090399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5090809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5091167Z return mod(**inputs) 2025-08-26T20:24:30.5091563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5091991Z outputs = self.model.decoder( 2025-08-26T20:24:30.5092398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5092872Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5093254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5093655Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5094080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5094522Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5094974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5095423Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5095922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5096639Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5096830Z 2025-08-26T20:24:30.5096946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5097343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5097702Z return mod(**inputs) 2025-08-26T20:24:30.5098106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5098529Z outputs = self.model.decoder( 2025-08-26T20:24:30.5098944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5099370Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5099757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5100157Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5100586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5101039Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5101446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5101939Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5102081Z 2025-08-26T20:24:30.5102194Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5102553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5102885Z return mod(**inputs) 2025-08-26T20:24:30.5103251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5103644Z outputs = self.model.decoder( 2025-08-26T20:24:30.5104041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5104459Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5104836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5105221Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5105641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5106069Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5106252Z 2025-08-26T20:24:30.5106359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5106726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5107057Z return mod(**inputs) 2025-08-26T20:24:30.5107422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5107861Z outputs = self.model.decoder( 2025-08-26T20:24:30.5108253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5108648Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5109008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5109380Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5109781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5110221Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5110628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5110987Z return self.act(input) 2025-08-26T20:24:30.5111103Z 2025-08-26T20:24:30.5111215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5111593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5111929Z return mod(**inputs) 2025-08-26T20:24:30.5112302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5112703Z outputs = self.model.decoder( 2025-08-26T20:24:30.5113084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5113546Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5113907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5114278Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5114668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5115099Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5115255Z 2025-08-26T20:24:30.5115370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5115759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5116116Z return mod(**inputs) 2025-08-26T20:24:30.5116548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5116966Z outputs = self.model.decoder( 2025-08-26T20:24:30.5117365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5117827Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5118192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5118586Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5119003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5119547Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5120005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5120512Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5120739Z 2025-08-26T20:24:30.5120851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5121238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5121591Z return mod(**inputs) 2025-08-26T20:24:30.5121976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5122398Z outputs = self.model.decoder( 2025-08-26T20:24:30.5122854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5123274Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5123647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5124029Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5124442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5124877Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5125312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5125734Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5125879Z 2025-08-26T20:24:30.5125990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5126377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5126723Z return mod(**inputs) 2025-08-26T20:24:30.5127111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5127523Z outputs = self.model.decoder( 2025-08-26T20:24:30.5127917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5128303Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5128653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5129019Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5129400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5129816Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5130225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5130624Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5130767Z 2025-08-26T20:24:30.5130862Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5131120Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5131349Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5131578Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5131837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5132225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5132582Z return mod(**inputs) 2025-08-26T20:24:30.5132972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5133373Z outputs = self.model.decoder( 2025-08-26T20:24:30.5133765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5134160Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5134519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5134893Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5135288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5135744Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5136197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5136641Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5137134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5137686Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5137876Z 2025-08-26T20:24:30.5137986Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5138363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5138701Z return mod(**inputs) 2025-08-26T20:24:30.5139071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5139471Z outputs = self.model.decoder( 2025-08-26T20:24:30.5139858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5140255Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5140616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5140995Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5141390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5141811Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5142233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5142653Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5143127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5143618Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5143794Z 2025-08-26T20:24:30.5143904Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5144272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5144616Z return mod(**inputs) 2025-08-26T20:24:30.5144990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5145387Z outputs = self.model.decoder( 2025-08-26T20:24:30.5145836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5146254Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5146638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5146999Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5147407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5147849Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5148286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5148711Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5148859Z 2025-08-26T20:24:30.5148979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5149369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5149722Z return mod(**inputs) 2025-08-26T20:24:30.5150110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5150527Z outputs = self.model.decoder( 2025-08-26T20:24:30.5150928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5151339Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5151712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5152138Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5152542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5153005Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5153196Z 2025-08-26T20:24:30.5153312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5153695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5154040Z return mod(**inputs) 2025-08-26T20:24:30.5154417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5154825Z outputs = self.model.decoder( 2025-08-26T20:24:30.5155224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5155635Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5156006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5156390Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5156816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5157286Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5157713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5158080Z return self.act(input) 2025-08-26T20:24:30.5158209Z 2025-08-26T20:24:30.5158325Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5158721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5159083Z return mod(**inputs) 2025-08-26T20:24:30.5159660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5160098Z outputs = self.model.decoder( 2025-08-26T20:24:30.5160513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5160982Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5161337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5161698Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5162088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5162483Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5162621Z 2025-08-26T20:24:30.5162752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5163117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5163444Z return mod(**inputs) 2025-08-26T20:24:30.5163809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5164196Z outputs = self.model.decoder( 2025-08-26T20:24:30.5164580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5164966Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5165315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5165680Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5166067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5166481Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5166924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5167412Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5167637Z 2025-08-26T20:24:30.5167748Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5168134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5168488Z return mod(**inputs) 2025-08-26T20:24:30.5168865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5169277Z outputs = self.model.decoder( 2025-08-26T20:24:30.5169658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5170043Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5170393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5170758Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5171148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5171572Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5172006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5172422Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5172573Z 2025-08-26T20:24:30.5172685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5173065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5173416Z return mod(**inputs) 2025-08-26T20:24:30.5173802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5174217Z outputs = self.model.decoder( 2025-08-26T20:24:30.5174620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5175023Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5175414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5175788Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5176199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5176647Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5177087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5177526Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5177680Z 2025-08-26T20:24:30.5177771Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5178016Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5178250Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5178487Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5178741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5179126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5179483Z return mod(**inputs) 2025-08-26T20:24:30.5179878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5180309Z outputs = self.model.decoder( 2025-08-26T20:24:30.5180711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5181139Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5181554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5181955Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5182372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5182814Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5183250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5183700Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5184172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5184689Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5184889Z 2025-08-26T20:24:30.5185004Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5185402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5185766Z return mod(**inputs) 2025-08-26T20:24:30.5186161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5186587Z outputs = self.model.decoder( 2025-08-26T20:24:30.5187009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5187438Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5187813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5188204Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5188604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5189045Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5189475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5189904Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5190431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5190923Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5191104Z 2025-08-26T20:24:30.5191216Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5191604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5191953Z return mod(**inputs) 2025-08-26T20:24:30.5192335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5192758Z outputs = self.model.decoder( 2025-08-26T20:24:30.5193162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5193581Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5193970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5194370Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5194803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5195255Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5195706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5196145Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5196439Z 2025-08-26T20:24:30.5196562Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5197059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5197423Z return mod(**inputs) 2025-08-26T20:24:30.5197819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5198241Z outputs = self.model.decoder( 2025-08-26T20:24:30.5198655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5199089Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5199650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5200099Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5200523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5201007Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5201210Z 2025-08-26T20:24:30.5201329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5201733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5202088Z return mod(**inputs) 2025-08-26T20:24:30.5202494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5202929Z outputs = self.model.decoder( 2025-08-26T20:24:30.5203351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5203786Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5204164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5204575Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5205007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5205482Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5205910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5206379Z return self.act(input) 2025-08-26T20:24:30.5206512Z 2025-08-26T20:24:30.5206627Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5207025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5207404Z return mod(**inputs) 2025-08-26T20:24:30.5207790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5208186Z outputs = self.model.decoder( 2025-08-26T20:24:30.5208578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5209044Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5209403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5209772Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5210175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5210581Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5210723Z 2025-08-26T20:24:30.5210839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5211214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5211547Z return mod(**inputs) 2025-08-26T20:24:30.5211963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5212445Z outputs = self.model.decoder( 2025-08-26T20:24:30.5212859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5213273Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5213647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5214037Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5214431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5214880Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5215315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5215818Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5216046Z 2025-08-26T20:24:30.5216157Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5216545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5216894Z return mod(**inputs) 2025-08-26T20:24:30.5217273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5217683Z outputs = self.model.decoder( 2025-08-26T20:24:30.5218074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5218462Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5218807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5219177Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5219563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5219978Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5220385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5220777Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5220961Z 2025-08-26T20:24:30.5221070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5221441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5221782Z return mod(**inputs) 2025-08-26T20:24:30.5222147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5222582Z outputs = self.model.decoder( 2025-08-26T20:24:30.5222964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5223360Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5223723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5224091Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5224490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5224911Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5225325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5225788Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5225928Z 2025-08-26T20:24:30.5226012Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5226230Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5226442Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5226683Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5226907Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5227271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5227607Z return mod(**inputs) 2025-08-26T20:24:30.5227975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5228367Z outputs = self.model.decoder( 2025-08-26T20:24:30.5228759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5229137Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5229485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5229842Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5230215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5230620Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5231020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5231433Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5231889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5232371Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5232577Z 2025-08-26T20:24:30.5232692Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5233080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5233431Z return mod(**inputs) 2025-08-26T20:24:30.5233816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5234248Z outputs = self.model.decoder( 2025-08-26T20:24:30.5234652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5235061Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5235471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5235850Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5236260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5236696Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5237138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5237594Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5238075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5238572Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5238756Z 2025-08-26T20:24:30.5238875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5239342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5239719Z return mod(**inputs) 2025-08-26T20:24:30.5240113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5240553Z outputs = self.model.decoder( 2025-08-26T20:24:30.5240960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5241382Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5241797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5242198Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5242613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5243061Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5243491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5243911Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5257715Z 2025-08-26T20:24:30.5257863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5258281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5258635Z return mod(**inputs) 2025-08-26T20:24:30.5259039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5259477Z outputs = self.model.decoder( 2025-08-26T20:24:30.5259880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5260277Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5260648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5261020Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5261431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5261860Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5262035Z 2025-08-26T20:24:30.5262154Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5262527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5262865Z return mod(**inputs) 2025-08-26T20:24:30.5263241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5263643Z outputs = self.model.decoder( 2025-08-26T20:24:30.5264152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5264548Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5264909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5265284Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5265678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5266118Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5266509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5266861Z return self.act(input) 2025-08-26T20:24:30.5266986Z 2025-08-26T20:24:30.5267095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5267470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5267822Z return mod(**inputs) 2025-08-26T20:24:30.5268204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5268620Z outputs = self.model.decoder( 2025-08-26T20:24:30.5269038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5269425Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5269770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5270194Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5270588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5270986Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5271125Z 2025-08-26T20:24:30.5271246Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5271607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5271938Z return mod(**inputs) 2025-08-26T20:24:30.5272300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5272693Z outputs = self.model.decoder( 2025-08-26T20:24:30.5273073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5273458Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5273811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5274176Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5274568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5274982Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5275412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5275905Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5276123Z 2025-08-26T20:24:30.5276241Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5276624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5276972Z return mod(**inputs) 2025-08-26T20:24:30.5277360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5277768Z outputs = self.model.decoder( 2025-08-26T20:24:30.5278205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5278614Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5278978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5279475Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5279902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5280358Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5280783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5281222Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5281382Z 2025-08-26T20:24:30.5281496Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5281895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5282256Z return mod(**inputs) 2025-08-26T20:24:30.5282646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5283073Z outputs = self.model.decoder( 2025-08-26T20:24:30.5283488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5283908Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5284290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5284678Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5285137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5285588Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5286033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5286459Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5286625Z 2025-08-26T20:24:30.5286716Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5286956Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5287187Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5287413Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5287662Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5288055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5288417Z return mod(**inputs) 2025-08-26T20:24:30.5288815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5289226Z outputs = self.model.decoder( 2025-08-26T20:24:30.5289638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5290054Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5290441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5290796Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5291164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5291564Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5291955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5292358Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5292792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5293313Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5293509Z 2025-08-26T20:24:30.5293612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5293969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5294293Z return mod(**inputs) 2025-08-26T20:24:30.5294643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5295029Z outputs = self.model.decoder( 2025-08-26T20:24:30.5295400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5295786Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5296125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5296611Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5297002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5297414Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5297819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5298222Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5298656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5299114Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5299383Z 2025-08-26T20:24:30.5299491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5299857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5300186Z return mod(**inputs) 2025-08-26T20:24:30.5300563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5300968Z outputs = self.model.decoder( 2025-08-26T20:24:30.5301345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5301730Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5302074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5302437Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5302826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5303235Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5303632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5304020Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5304161Z 2025-08-26T20:24:30.5304264Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5304621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5304943Z return mod(**inputs) 2025-08-26T20:24:30.5305290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5305672Z outputs = self.model.decoder( 2025-08-26T20:24:30.5306047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5306437Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5306793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5307152Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5307590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5308037Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5308206Z 2025-08-26T20:24:30.5308316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5308671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5308985Z return mod(**inputs) 2025-08-26T20:24:30.5309340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5309734Z outputs = self.model.decoder( 2025-08-26T20:24:30.5310114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5310491Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5310846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5311215Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5311605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5312044Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5312430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5312778Z return self.act(input) 2025-08-26T20:24:30.5312898Z 2025-08-26T20:24:30.5313005Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5313423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5313778Z return mod(**inputs) 2025-08-26T20:24:30.5314177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5314595Z outputs = self.model.decoder( 2025-08-26T20:24:30.5315016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5315421Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5315788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5316179Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5316588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5317017Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5317162Z 2025-08-26T20:24:30.5317280Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5317655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5318000Z return mod(**inputs) 2025-08-26T20:24:30.5318395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5318805Z outputs = self.model.decoder( 2025-08-26T20:24:30.5319215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5319728Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5320114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5320522Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5320917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5321330Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5321746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5322255Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5322464Z 2025-08-26T20:24:30.5322581Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5322943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5323269Z return mod(**inputs) 2025-08-26T20:24:30.5323629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5324017Z outputs = self.model.decoder( 2025-08-26T20:24:30.5324397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5324772Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5325120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5325505Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5325910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5326340Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5326762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5327163Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5327301Z 2025-08-26T20:24:30.5327402Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5327790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5328118Z return mod(**inputs) 2025-08-26T20:24:30.5328476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5328864Z outputs = self.model.decoder( 2025-08-26T20:24:30.5329247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5329634Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5329980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5330345Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5330737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5331141Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5331543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5331933Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5332075Z 2025-08-26T20:24:30.5332154Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5332365Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5332568Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5332896Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5333126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5333475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5333790Z return mod(**inputs) 2025-08-26T20:24:30.5334137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5334505Z outputs = self.model.decoder( 2025-08-26T20:24:30.5334876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5335249Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5335596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5335982Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5336362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5336774Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5337185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5337591Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5338022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5338501Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5338692Z 2025-08-26T20:24:30.5338797Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5339150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5339482Z return mod(**inputs) 2025-08-26T20:24:30.5339821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5340199Z outputs = self.model.decoder( 2025-08-26T20:24:30.5340571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5340948Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5341288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5341697Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5342101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5342513Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5342922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5343324Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5343775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5344230Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5344400Z 2025-08-26T20:24:30.5344508Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5344871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5345206Z return mod(**inputs) 2025-08-26T20:24:30.5345587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5345980Z outputs = self.model.decoder( 2025-08-26T20:24:30.5346359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5346750Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5347123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5347516Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5347910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5348330Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5348736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5349142Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5349297Z 2025-08-26T20:24:30.5349404Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5349796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5350120Z return mod(**inputs) 2025-08-26T20:24:30.5350479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5350880Z outputs = self.model.decoder( 2025-08-26T20:24:30.5351269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5351667Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5352024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5352397Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5352790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5353231Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5353412Z 2025-08-26T20:24:30.5353536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5353923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5354288Z return mod(**inputs) 2025-08-26T20:24:30.5354693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5355117Z outputs = self.model.decoder( 2025-08-26T20:24:30.5355536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5355986Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5356358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5356748Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5357160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5357626Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5358048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5358415Z return self.act(input) 2025-08-26T20:24:30.5358541Z 2025-08-26T20:24:30.5358654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5359043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5359493Z return mod(**inputs) 2025-08-26T20:24:30.5359909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5360343Z outputs = self.model.decoder( 2025-08-26T20:24:30.5360758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5361200Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5361579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5361989Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5362415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5362852Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5363005Z 2025-08-26T20:24:30.5363126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5363514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5363872Z return mod(**inputs) 2025-08-26T20:24:30.5364268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5364705Z outputs = self.model.decoder( 2025-08-26T20:24:30.5365153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5365576Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5365959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5366357Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5366781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5367220Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5367670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5368189Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5368414Z 2025-08-26T20:24:30.5368537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5368939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5369302Z return mod(**inputs) 2025-08-26T20:24:30.5369684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5370161Z outputs = self.model.decoder( 2025-08-26T20:24:30.5370558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5370967Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5371340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5371763Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5372175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5372622Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5373049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5373475Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5373623Z 2025-08-26T20:24:30.5373735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5374116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5374455Z return mod(**inputs) 2025-08-26T20:24:30.5374837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5375256Z outputs = self.model.decoder( 2025-08-26T20:24:30.5375660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5376072Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5376439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5376829Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5377238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5377675Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5378110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5378525Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5378688Z 2025-08-26T20:24:30.5378775Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5379010Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5379237Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5379458Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5379699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5380119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5380484Z return mod(**inputs) 2025-08-26T20:24:30.5380844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5381237Z outputs = self.model.decoder( 2025-08-26T20:24:30.5381618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5382005Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5382356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5382720Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5383109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5383529Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5383942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5384354Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5384802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5385295Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5385492Z 2025-08-26T20:24:30.5385598Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5386000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5386329Z return mod(**inputs) 2025-08-26T20:24:30.5386688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5387085Z outputs = self.model.decoder( 2025-08-26T20:24:30.5387466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5387853Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5388194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5388561Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5388945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5389360Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5389768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5390172Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5390624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5391090Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5391252Z 2025-08-26T20:24:30.5391363Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5391724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5392045Z return mod(**inputs) 2025-08-26T20:24:30.5392408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5392802Z outputs = self.model.decoder( 2025-08-26T20:24:30.5393207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5393609Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5393983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5394404Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5394821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5395265Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5395689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5396110Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5396399Z 2025-08-26T20:24:30.5396518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5396921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5397290Z return mod(**inputs) 2025-08-26T20:24:30.5397694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5398137Z outputs = self.model.decoder( 2025-08-26T20:24:30.5398551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5398977Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5399417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5399838Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5400264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5400833Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5401018Z 2025-08-26T20:24:30.5401136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5401516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5401870Z return mod(**inputs) 2025-08-26T20:24:30.5402268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5402689Z outputs = self.model.decoder( 2025-08-26T20:24:30.5403082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5403509Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5403881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5404268Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5404677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5405124Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5405537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5405916Z return self.act(input) 2025-08-26T20:24:30.5406032Z 2025-08-26T20:24:30.5406149Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5406527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5406872Z return mod(**inputs) 2025-08-26T20:24:30.5407254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5407657Z outputs = self.model.decoder( 2025-08-26T20:24:30.5408033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5408419Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5408769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5409130Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5409565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5409974Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5410126Z 2025-08-26T20:24:30.5410239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5410645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5410992Z return mod(**inputs) 2025-08-26T20:24:30.5411380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5411787Z outputs = self.model.decoder( 2025-08-26T20:24:30.5412186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5412574Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5412930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5413299Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5413700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5414137Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5414570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5415070Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5415347Z 2025-08-26T20:24:30.5415460Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5415819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5416150Z return mod(**inputs) 2025-08-26T20:24:30.5416514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5416905Z outputs = self.model.decoder( 2025-08-26T20:24:30.5417287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5417694Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5418060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5418443Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5418852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5419281Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5419689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5420081Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5420216Z 2025-08-26T20:24:30.5420329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5420689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5421008Z return mod(**inputs) 2025-08-26T20:24:30.5421368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5421755Z outputs = self.model.decoder( 2025-08-26T20:24:30.5422133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5422518Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5422864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5423228Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5423666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5424105Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5424532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5424933Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5425077Z 2025-08-26T20:24:30.5425159Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5425377Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5425588Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5425811Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5426064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5426449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5426798Z return mod(**inputs) 2025-08-26T20:24:30.5427180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5427592Z outputs = self.model.decoder( 2025-08-26T20:24:30.5427998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5428405Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5428769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5429152Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5429564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5430041Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5430471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5430896Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5431376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5431895Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5432092Z 2025-08-26T20:24:30.5432213Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5432597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5432934Z return mod(**inputs) 2025-08-26T20:24:30.5433315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5433729Z outputs = self.model.decoder( 2025-08-26T20:24:30.5434141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5434556Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5434942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5435338Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5435760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5436207Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5436479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5436587Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5436914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5437036Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5437040Z 2025-08-26T20:24:30.5437199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5437421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5437502Z return mod(**inputs) 2025-08-26T20:24:30.5437778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5437861Z outputs = self.model.decoder( 2025-08-26T20:24:30.5438146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5438226Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5438482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5438570Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5438836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5438957Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5439298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5439407Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5439411Z 2025-08-26T20:24:30.5439527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5439767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5439843Z return mod(**inputs) 2025-08-26T20:24:30.5440115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5440252Z outputs = self.model.decoder( 2025-08-26T20:24:30.5440534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5440619Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5440859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5440944Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5441218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5441349Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5441353Z 2025-08-26T20:24:30.5441473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5441685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5441760Z return mod(**inputs) 2025-08-26T20:24:30.5442034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5442112Z outputs = self.model.decoder( 2025-08-26T20:24:30.5442386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5442463Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5442707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5442790Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5443058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5443182Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5443395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5443473Z return self.act(input) 2025-08-26T20:24:30.5443476Z 2025-08-26T20:24:30.5443588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5443823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5443897Z return mod(**inputs) 2025-08-26T20:24:30.5444135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5444216Z outputs = self.model.decoder( 2025-08-26T20:24:30.5444461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5444530Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5444754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5444836Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5445091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5445176Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5445179Z 2025-08-26T20:24:30.5445293Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5445495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5445563Z return mod(**inputs) 2025-08-26T20:24:30.5445819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5445894Z outputs = self.model.decoder( 2025-08-26T20:24:30.5446152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5446224Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5446479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5446567Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5446827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5446935Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5447179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5447338Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5447342Z 2025-08-26T20:24:30.5447443Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5447638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5447715Z return mod(**inputs) 2025-08-26T20:24:30.5447963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5448043Z outputs = self.model.decoder( 2025-08-26T20:24:30.5448288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5448362Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5448585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5448666Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5448938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5449046Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5449310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5449408Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5449412Z 2025-08-26T20:24:30.5449522Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5449741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5449869Z return mod(**inputs) 2025-08-26T20:24:30.5450146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5450227Z outputs = self.model.decoder( 2025-08-26T20:24:30.5450501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5450582Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5450807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5450893Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5451154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5451249Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5451495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5451579Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5451582Z 2025-08-26T20:24:30.5451669Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5451745Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5451821Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5451902Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5451999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5452192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5452290Z return mod(**inputs) 2025-08-26T20:24:30.5452534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5452618Z outputs = self.model.decoder( 2025-08-26T20:24:30.5452865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5452949Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5453172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5453259Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5453506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5453606Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5453858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5453960Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5454265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5454410Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5454414Z 2025-08-26T20:24:30.5454528Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5454738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5454816Z return mod(**inputs) 2025-08-26T20:24:30.5455074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5455148Z outputs = self.model.decoder( 2025-08-26T20:24:30.5455404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5455479Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5455699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5455783Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5456060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5456167Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5456415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5456513Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5456818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5456935Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5456942Z 2025-08-26T20:24:30.5457058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5457271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5457348Z return mod(**inputs) 2025-08-26T20:24:30.5457618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5457699Z outputs = self.model.decoder( 2025-08-26T20:24:30.5457974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5458050Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5458294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5458377Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5458640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5458800Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5459049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5459140Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5459144Z 2025-08-26T20:24:30.5459248Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5459456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5459523Z return mod(**inputs) 2025-08-26T20:24:30.5459773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5459855Z outputs = self.model.decoder( 2025-08-26T20:24:30.5460117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5460204Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5460440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5460523Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5460799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5460925Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5460929Z 2025-08-26T20:24:30.5461046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5461256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5461328Z return mod(**inputs) 2025-08-26T20:24:30.5461599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5461681Z outputs = self.model.decoder( 2025-08-26T20:24:30.5461952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5462027Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5462296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5462383Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5462645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5462778Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5463012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5463095Z return self.act(input) 2025-08-26T20:24:30.5463099Z 2025-08-26T20:24:30.5463208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5463420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5463501Z return mod(**inputs) 2025-08-26T20:24:30.5463766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5463855Z outputs = self.model.decoder( 2025-08-26T20:24:30.5464122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5464197Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5464436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5464522Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5464789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5464912Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5464916Z 2025-08-26T20:24:30.5465034Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5465245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5465314Z return mod(**inputs) 2025-08-26T20:24:30.5465588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5465667Z outputs = self.model.decoder( 2025-08-26T20:24:30.5465945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5466022Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5466257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5466351Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5466619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5466732Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5466997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:24:30.5467167Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:24:30.5467171Z 2025-08-26T20:24:30.5467281Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5467492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5467573Z return mod(**inputs) 2025-08-26T20:24:30.5467840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5467925Z outputs = self.model.decoder( 2025-08-26T20:24:30.5468193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5468270Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5468512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5468636Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5468905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5469012Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5469274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:24:30.5469367Z key_states = self.k_proj(current_states) 2025-08-26T20:24:30.5469371Z 2025-08-26T20:24:30.5469479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5469694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5469766Z return mod(**inputs) 2025-08-26T20:24:30.5470035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5470113Z outputs = self.model.decoder( 2025-08-26T20:24:30.5470376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5470461Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5470695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5470785Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5471044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5471146Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5471446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:24:30.5471539Z value_states = self.v_proj(current_states) 2025-08-26T20:24:30.5471543Z 2025-08-26T20:24:30.5471634Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5471721Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5471809Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5471889Z cudagraph partition due to non gpu ops 2025-08-26T20:24:30.5472000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5472217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5472296Z return mod(**inputs) 2025-08-26T20:24:30.5472556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5472631Z outputs = self.model.decoder( 2025-08-26T20:24:30.5472884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5472970Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5473207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5473301Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5473565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5473671Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5473941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5474047Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5474364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:24:30.5474508Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:24:30.5474512Z 2025-08-26T20:24:30.5474629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5474840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5474945Z return mod(**inputs) 2025-08-26T20:24:30.5475219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5475298Z outputs = self.model.decoder( 2025-08-26T20:24:30.5475568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5475645Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5475877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5475971Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5476232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5476345Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5476610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:24:30.5476715Z attn_output, attn_weights = attention_interface( 2025-08-26T20:24:30.5477030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:24:30.5477146Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:24:30.5477150Z 2025-08-26T20:24:30.5477266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5477474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5477586Z return mod(**inputs) 2025-08-26T20:24:30.5477851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5477930Z outputs = self.model.decoder( 2025-08-26T20:24:30.5478206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5478283Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5478526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5478610Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5478872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:24:30.5478985Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:24:30.5479330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:24:30.5479441Z attn_output = self.out_proj(attn_output) 2025-08-26T20:24:30.5479445Z 2025-08-26T20:24:30.5479557Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5479781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5479857Z return mod(**inputs) 2025-08-26T20:24:30.5480138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5480230Z outputs = self.model.decoder( 2025-08-26T20:24:30.5480501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5480588Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5480838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5480924Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5481194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5481321Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5481325Z 2025-08-26T20:24:30.5481478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5481692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5481765Z return mod(**inputs) 2025-08-26T20:24:30.5482037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5482114Z outputs = self.model.decoder( 2025-08-26T20:24:30.5482387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5482464Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5482708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5482793Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5483063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:24:30.5483202Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:24:30.5483433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:24:30.5483514Z return self.act(input) 2025-08-26T20:24:30.5483518Z 2025-08-26T20:24:30.5483631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5483847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5483928Z return mod(**inputs) 2025-08-26T20:24:30.5484202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-26T20:24:30.5484329Z outputs = self.model.decoder( 2025-08-26T20:24:30.5484600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:24:30.5484678Z layer_outputs = decoder_layer( 2025-08-26T20:24:30.5484927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:24:30.5485015Z return super().__call__(*args, **kwargs) 2025-08-26T20:24:30.5485299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:24:30.5485386Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:24:30.5485390Z 2025-08-26T20:24:30.5485502Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5485713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5485784Z return mod(**inputs) 2025-08-26T20:24:30.5486056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1917, in forward 2025-08-26T20:24:30.5486139Z logits = self.lm_head(outputs[0]) 2025-08-26T20:24:30.5486143Z 2025-08-26T20:24:30.5486260Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:24:30.5486470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:24:30.5486539Z return mod(**inputs) 2025-08-26T20:24:30.5486808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1923, in forward 2025-08-26T20:24:30.5486966Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:24:30.5486970Z 2025-08-26T20:24:40.3654920Z Compilation time (from dynamo_timed): 15.825380118 2025-08-26T20:24:40.3921115Z pass 2025-08-26T20:24:40.3921656Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:24:40.3922955Z TIMING: _recursive_pre_grad_passes:0.00819 _recursive_joint_graph_passes:0.66922 _recursive_post_grad_passes:0.08041 async_compile.wait:0.74342 code_gen:8.35243 inductor_compile:9.69751 backend_compile:13.09938 gc:0.00166 entire_frame_compile:15.82538 total_wall_time:15.82538 2025-08-26T20:24:40.3924162Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:13192 | FakeTensor.__torch_dispatch__:4538 | ProxyTorchDispatchMode.__torch_dispatch__:4813 2025-08-26T20:24:40.3924736Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2025-08-26T20:24:45.7374333Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:24:45.7375373Z from pkg_resources import resource_filename 2025-08-26T20:24:46.3360003Z 2025-08-26T20:24:51.6384784Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:24:51.6385067Z loading model: 0it [00:05, ?it/s] 2025-08-26T20:24:51.6426257Z cpu eval BartForConditionalGeneration 2025-08-26T20:24:55.2793285Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:24:56.5663416Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:24:57.8400087Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:25:15.2500214Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2503187Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2503503Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2503734Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2504352Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2504579Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2504806Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2505112Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2505348Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2505579Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2505828Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2506049Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2506319Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2506738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2507145Z return mod(**inputs) 2025-08-26T20:25:15.2507581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2508004Z outputs = self.model( 2025-08-26T20:25:15.2508414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2508839Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2509260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2509675Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2510062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2510479Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2510910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2511349Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2511779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2512283Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2512515Z 2025-08-26T20:25:15.2512633Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2513042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2513397Z return mod(**inputs) 2025-08-26T20:25:15.2513891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2514331Z outputs = self.model( 2025-08-26T20:25:15.2514859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2515295Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2515741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2516165Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2516558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2516972Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2517423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2517876Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2518318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2518743Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2518904Z 2025-08-26T20:25:15.2519028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2519503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2519886Z return mod(**inputs) 2025-08-26T20:25:15.2520282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2520821Z outputs = self.model( 2025-08-26T20:25:15.2521223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2521654Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2522080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2522503Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2522893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2523312Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2523738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2524185Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2524626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2525073Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2525239Z 2025-08-26T20:25:15.2525331Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2525571Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2525799Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2526029Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2526292Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2526699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2527070Z return mod(**inputs) 2025-08-26T20:25:15.2527478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2527911Z outputs = self.model( 2025-08-26T20:25:15.2528322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2528752Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2529165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2529633Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2530028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2530435Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2530860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2531296Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2531742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2532201Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2532708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2533252Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2533460Z 2025-08-26T20:25:15.2533578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2533974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2534335Z return mod(**inputs) 2025-08-26T20:25:15.2534742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2535158Z outputs = self.model( 2025-08-26T20:25:15.2535559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2536029Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2536450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2536884Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2537277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2537708Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2538134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2538576Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2539008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2539465Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2539961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2540487Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2540677Z 2025-08-26T20:25:15.2540817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2541231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2541611Z return mod(**inputs) 2025-08-26T20:25:15.2542007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2542428Z outputs = self.model( 2025-08-26T20:25:15.2542828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2543241Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2543658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2544080Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2544468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2544874Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2545328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2545768Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2546215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2546665Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2546814Z 2025-08-26T20:25:15.2546930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2547318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2547676Z return mod(**inputs) 2025-08-26T20:25:15.2548072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2548492Z outputs = self.model( 2025-08-26T20:25:15.2548879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2549294Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2549707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2550131Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2550517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2550927Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2551357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2551880Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2552070Z 2025-08-26T20:25:15.2552190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2552575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2552941Z return mod(**inputs) 2025-08-26T20:25:15.2553345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2553768Z outputs = self.model( 2025-08-26T20:25:15.2554172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2554610Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2555029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2555456Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2555841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2556237Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2556667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2557144Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2557582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2557968Z return self.act(input) 2025-08-26T20:25:15.2558091Z 2025-08-26T20:25:15.2558207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2558621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2559021Z return mod(**inputs) 2025-08-26T20:25:15.2559675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2560118Z outputs = self.model( 2025-08-26T20:25:15.2560518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2561001Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2561431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2561862Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2562253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2562660Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2563097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2563553Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2563709Z 2025-08-26T20:25:15.2563837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2564230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2564597Z return mod(**inputs) 2025-08-26T20:25:15.2565004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2565484Z outputs = self.model( 2025-08-26T20:25:15.2565888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2566335Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2566758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2567194Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2567591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2568836Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2569267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2569713Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2570158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2570672Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2570903Z 2025-08-26T20:25:15.2571021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2571424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2571786Z return mod(**inputs) 2025-08-26T20:25:15.2572184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2572607Z outputs = self.model( 2025-08-26T20:25:15.2573005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2573441Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2573863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2574282Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2574669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2575075Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2575502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2575949Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2576377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2576790Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2576946Z 2025-08-26T20:25:15.2577059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2577479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2577844Z return mod(**inputs) 2025-08-26T20:25:15.2578262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2578702Z outputs = self.model( 2025-08-26T20:25:15.2579119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2579565Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2579994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2580406Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2580842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2581240Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2581659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2582093Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2582542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2582985Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2583152Z 2025-08-26T20:25:15.2583244Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2583485Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2583780Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2584021Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2584297Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2584689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2585046Z return mod(**inputs) 2025-08-26T20:25:15.2585441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2585860Z outputs = self.model( 2025-08-26T20:25:15.2586259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2586745Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2587146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2587556Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2587931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2588320Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2588741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2589168Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2589608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2590059Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2590532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2591049Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2591247Z 2025-08-26T20:25:15.2591363Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2591752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2592102Z return mod(**inputs) 2025-08-26T20:25:15.2592504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2592949Z outputs = self.model( 2025-08-26T20:25:15.2593347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2593784Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2594198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2594613Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2594984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2595390Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2595822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2596474Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2596922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2597380Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2597993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2598503Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2598684Z 2025-08-26T20:25:15.2598816Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2599217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2599813Z return mod(**inputs) 2025-08-26T20:25:15.2600224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2600652Z outputs = self.model( 2025-08-26T20:25:15.2601058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2601477Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2601896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2602323Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2602711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2603114Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2603530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2603982Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2604417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2604848Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2604998Z 2025-08-26T20:25:15.2605115Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2605513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2605873Z return mod(**inputs) 2025-08-26T20:25:15.2606267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2606687Z outputs = self.model( 2025-08-26T20:25:15.2607088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2607515Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2607927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2608347Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2608782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2609183Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2609605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2610089Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2610287Z 2025-08-26T20:25:15.2610412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2610799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2611163Z return mod(**inputs) 2025-08-26T20:25:15.2611568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2611992Z outputs = self.model( 2025-08-26T20:25:15.2612393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2612827Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2613243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2613666Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2614053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2614444Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2614869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2615377Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2615813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2616199Z return self.act(input) 2025-08-26T20:25:15.2616317Z 2025-08-26T20:25:15.2616428Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2616816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2617179Z return mod(**inputs) 2025-08-26T20:25:15.2617580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2617988Z outputs = self.model( 2025-08-26T20:25:15.2618368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2618781Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2619196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2619626Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2620005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2620406Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2620836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2621270Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2621421Z 2025-08-26T20:25:15.2621539Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2621920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2622273Z return mod(**inputs) 2025-08-26T20:25:15.2622661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2623069Z outputs = self.model( 2025-08-26T20:25:15.2623464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2623874Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2624316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2624726Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2625102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2625485Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2625897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2626326Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2626760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2627254Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2627481Z 2025-08-26T20:25:15.2627597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2627999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2628374Z return mod(**inputs) 2025-08-26T20:25:15.2628784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2629191Z outputs = self.model( 2025-08-26T20:25:15.2629573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2629986Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2630389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2630837Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2631209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2631599Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2632016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2632448Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2632900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2633325Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2633485Z 2025-08-26T20:25:15.2633603Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2634000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2634372Z return mod(**inputs) 2025-08-26T20:25:15.2634764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2635189Z outputs = self.model( 2025-08-26T20:25:15.2635599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2636025Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2636445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2636864Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2637250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2637652Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2638079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2638524Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2638956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2639483Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2639713Z 2025-08-26T20:25:15.2639810Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2640059Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2640289Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2640530Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2640792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2641193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2641547Z return mod(**inputs) 2025-08-26T20:25:15.2641951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2642376Z outputs = self.model( 2025-08-26T20:25:15.2642780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2643208Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2643625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2644050Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2644440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2644845Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2645271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2645718Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2646202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2646654Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2647239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2647823Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2648041Z 2025-08-26T20:25:15.2648162Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2648562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2648929Z return mod(**inputs) 2025-08-26T20:25:15.2649342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2649845Z outputs = self.model( 2025-08-26T20:25:15.2650351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2650796Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2651204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2651616Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2651982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2652372Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2652787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2653219Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2653637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2654078Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2654551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2655037Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2655211Z 2025-08-26T20:25:15.2655413Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2655795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2656145Z return mod(**inputs) 2025-08-26T20:25:15.2656533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2656941Z outputs = self.model( 2025-08-26T20:25:15.2657340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2657768Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2658181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2658589Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2658965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2659354Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2659767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2660193Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2660617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2661041Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2661189Z 2025-08-26T20:25:15.2661302Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2661737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2662090Z return mod(**inputs) 2025-08-26T20:25:15.2662479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2662893Z outputs = self.model( 2025-08-26T20:25:15.2663282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2663697Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2664104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2664514Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2664881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2665270Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2665688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2666151Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2666338Z 2025-08-26T20:25:15.2666457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2666835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2667187Z return mod(**inputs) 2025-08-26T20:25:15.2667571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2667980Z outputs = self.model( 2025-08-26T20:25:15.2668359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2668775Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2669180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2669597Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2669968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2670390Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2670807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2671268Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2671689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2672071Z return self.act(input) 2025-08-26T20:25:15.2672195Z 2025-08-26T20:25:15.2672312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2672707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2673066Z return mod(**inputs) 2025-08-26T20:25:15.2673455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2673860Z outputs = self.model( 2025-08-26T20:25:15.2674256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2674679Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2675089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2675501Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2675874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2676273Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2676702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2677172Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2677325Z 2025-08-26T20:25:15.2677449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2677842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2678204Z return mod(**inputs) 2025-08-26T20:25:15.2678600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2679018Z outputs = self.model( 2025-08-26T20:25:15.2679509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2679955Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2680377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2680807Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2681194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2681597Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2682083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2682532Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2682971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2683470Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2683705Z 2025-08-26T20:25:15.2683820Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2684215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2684590Z return mod(**inputs) 2025-08-26T20:25:15.2684997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2685420Z outputs = self.model( 2025-08-26T20:25:15.2685927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2686357Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2686772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2687225Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2687602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2688005Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2688430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2688877Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2689310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2689738Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2689899Z 2025-08-26T20:25:15.2690017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2690415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2690777Z return mod(**inputs) 2025-08-26T20:25:15.2691166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2691585Z outputs = self.model( 2025-08-26T20:25:15.2691948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2692399Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2692804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2693206Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2693577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2693966Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2694375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2694798Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2695223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2695643Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2695795Z 2025-08-26T20:25:15.2695889Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2696122Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2696547Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2696775Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2697030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2697429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2697775Z return mod(**inputs) 2025-08-26T20:25:15.2698169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2698581Z outputs = self.model( 2025-08-26T20:25:15.2698973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2699388Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2699788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2700205Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2700581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2700974Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2701491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2701929Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2702359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2702805Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2703283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2703771Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2703973Z 2025-08-26T20:25:15.2704083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2704455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2704793Z return mod(**inputs) 2025-08-26T20:25:15.2705168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2705554Z outputs = self.model( 2025-08-26T20:25:15.2705926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2706342Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2706749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2707156Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2707535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2707988Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2708405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2708842Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2709262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2709699Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2710175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2710664Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2710838Z 2025-08-26T20:25:15.2710956Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2711336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2711687Z return mod(**inputs) 2025-08-26T20:25:15.2712181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2712600Z outputs = self.model( 2025-08-26T20:25:15.2712994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2713426Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2713845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2714268Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2714655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2715048Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2715479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2715921Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2716360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2716836Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2716991Z 2025-08-26T20:25:15.2717109Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2717510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2717871Z return mod(**inputs) 2025-08-26T20:25:15.2718272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2718699Z outputs = self.model( 2025-08-26T20:25:15.2719113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2719619Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2720039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2720460Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2720840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2721240Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2721672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2722152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2722345Z 2025-08-26T20:25:15.2722480Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2722867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2723280Z return mod(**inputs) 2025-08-26T20:25:15.2723685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2724099Z outputs = self.model( 2025-08-26T20:25:15.2724480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2724896Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2725291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2725678Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2726030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2726399Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2726789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2727228Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2727621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2727969Z return self.act(input) 2025-08-26T20:25:15.2728091Z 2025-08-26T20:25:15.2728197Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2728566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2728895Z return mod(**inputs) 2025-08-26T20:25:15.2729259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2729633Z outputs = self.model( 2025-08-26T20:25:15.2729999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2730391Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2730773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2731155Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2731541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2731920Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2732324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2732737Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2732885Z 2025-08-26T20:25:15.2732997Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2733376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2733728Z return mod(**inputs) 2025-08-26T20:25:15.2734097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2734478Z outputs = self.model( 2025-08-26T20:25:15.2734833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2735222Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2735600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2735984Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2736325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2736691Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2737075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2737538Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2737967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2738451Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2738681Z 2025-08-26T20:25:15.2738798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2739185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2739533Z return mod(**inputs) 2025-08-26T20:25:15.2739901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2740282Z outputs = self.model( 2025-08-26T20:25:15.2740668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2741161Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2741563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2741961Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2742342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2742737Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2743150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2743595Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2744014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2744437Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2744590Z 2025-08-26T20:25:15.2744702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2745090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2745445Z return mod(**inputs) 2025-08-26T20:25:15.2745824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2746266Z outputs = self.model( 2025-08-26T20:25:15.2746662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2747075Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2747469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2747878Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2748253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2748645Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2749059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2749480Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2749910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2750336Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2750487Z 2025-08-26T20:25:15.2750585Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2750809Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2751039Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2751266Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2751521Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2751910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2752309Z return mod(**inputs) 2025-08-26T20:25:15.2752707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2753130Z outputs = self.model( 2025-08-26T20:25:15.2753540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2753961Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2754389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2754800Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2755182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2755573Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2755989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2756439Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2756885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2757330Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2757807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2758331Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2758542Z 2025-08-26T20:25:15.2758660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2759056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2759501Z return mod(**inputs) 2025-08-26T20:25:15.2759900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2760351Z outputs = self.model( 2025-08-26T20:25:15.2760766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2761188Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2761647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2762063Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2762450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2762857Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2763277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2763712Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2764151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2764611Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2765103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2765611Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2765789Z 2025-08-26T20:25:15.2765906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2766304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2766677Z return mod(**inputs) 2025-08-26T20:25:15.2767071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2767484Z outputs = self.model( 2025-08-26T20:25:15.2767876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2768368Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2768781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2769203Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2769593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2769991Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2770398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2770804Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2771207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2771596Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2771747Z 2025-08-26T20:25:15.2771854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2772240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2772593Z return mod(**inputs) 2025-08-26T20:25:15.2772982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2773384Z outputs = self.model( 2025-08-26T20:25:15.2773769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2774183Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2774587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2774965Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2775331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2775722Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2776131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2776627Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2776816Z 2025-08-26T20:25:15.2777681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2778066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2778417Z return mod(**inputs) 2025-08-26T20:25:15.2778803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2779213Z outputs = self.model( 2025-08-26T20:25:15.2779593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2780016Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2780423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2780834Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2781204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2781593Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2782008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2782467Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2782880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2783242Z return self.act(input) 2025-08-26T20:25:15.2783369Z 2025-08-26T20:25:15.2783523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2783911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2784260Z return mod(**inputs) 2025-08-26T20:25:15.2784646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2785049Z outputs = self.model( 2025-08-26T20:25:15.2785437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2785847Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2786255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2786668Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2787040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2787437Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2787846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2788268Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2788415Z 2025-08-26T20:25:15.2788529Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2788916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2789270Z return mod(**inputs) 2025-08-26T20:25:15.2789659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2790060Z outputs = self.model( 2025-08-26T20:25:15.2790447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2790913Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2791321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2791757Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2792133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2792604Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2793045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2793500Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2793922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2794419Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2794649Z 2025-08-26T20:25:15.2794764Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2795179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2795553Z return mod(**inputs) 2025-08-26T20:25:15.2795957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2796542Z outputs = self.model( 2025-08-26T20:25:15.2796956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2797396Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2797817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2798252Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2798647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2799054Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2799658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2800117Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2800560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2801006Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2801163Z 2025-08-26T20:25:15.2801285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2801672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2802019Z return mod(**inputs) 2025-08-26T20:25:15.2802406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2802814Z outputs = self.model( 2025-08-26T20:25:15.2803208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2803636Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2804042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2804471Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2804843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2805229Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2805630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2806058Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2806482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2806911Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2807060Z 2025-08-26T20:25:15.2807155Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2807384Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2807601Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2807815Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2808124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2808490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2808827Z return mod(**inputs) 2025-08-26T20:25:15.2809193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2809590Z outputs = self.model( 2025-08-26T20:25:15.2809963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2810349Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2810755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2811171Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2811545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2811933Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2812352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2812781Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2813209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2813646Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2814114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2814686Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2814890Z 2025-08-26T20:25:15.2815003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2815393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2815743Z return mod(**inputs) 2025-08-26T20:25:15.2816128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2816537Z outputs = self.model( 2025-08-26T20:25:15.2816918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2817309Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2817683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2818087Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2818461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2818852Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2819259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2819679Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2820103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2820539Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2821017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2821511Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2821689Z 2025-08-26T20:25:15.2821801Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2822189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2822540Z return mod(**inputs) 2025-08-26T20:25:15.2822965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2823380Z outputs = self.model( 2025-08-26T20:25:15.2823768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2824185Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2824587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2824998Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2825368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2825761Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2826178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2826607Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2827041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2827461Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2827618Z 2025-08-26T20:25:15.2827731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2828118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2828470Z return mod(**inputs) 2025-08-26T20:25:15.2828852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2829310Z outputs = self.model( 2025-08-26T20:25:15.2829694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2830108Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2830513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2830917Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2831290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2831678Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2832094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2832547Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2832738Z 2025-08-26T20:25:15.2832854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2833240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2833591Z return mod(**inputs) 2025-08-26T20:25:15.2833981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2834386Z outputs = self.model( 2025-08-26T20:25:15.2834782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2835219Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2835633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2836056Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2836433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2836834Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2837254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2837725Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2838188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2838569Z return self.act(input) 2025-08-26T20:25:15.2838701Z 2025-08-26T20:25:15.2838815Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2839208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2839661Z return mod(**inputs) 2025-08-26T20:25:15.2840058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2840543Z outputs = self.model( 2025-08-26T20:25:15.2840948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2841389Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2841796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2842220Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2842608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2843012Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2843436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2843860Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2844022Z 2025-08-26T20:25:15.2844139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2844533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2844940Z return mod(**inputs) 2025-08-26T20:25:15.2845336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2845748Z outputs = self.model( 2025-08-26T20:25:15.2846147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2846577Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2846992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2847406Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2847789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2848189Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2848612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2849057Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2849489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2850007Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2850231Z 2025-08-26T20:25:15.2850344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2850728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2851077Z return mod(**inputs) 2025-08-26T20:25:15.2851455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2851863Z outputs = self.model( 2025-08-26T20:25:15.2852246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2852663Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2853058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2853468Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2853873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2854265Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2854676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2855098Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2855521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2855939Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2856084Z 2025-08-26T20:25:15.2856202Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2856581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2856923Z return mod(**inputs) 2025-08-26T20:25:15.2857310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2857710Z outputs = self.model( 2025-08-26T20:25:15.2858098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2858482Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2858860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2859246Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2859598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2860000Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2860383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2860792Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2861196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2861599Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2861741Z 2025-08-26T20:25:15.2861833Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2862060Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2862289Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2862515Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2862771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2863154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2863506Z return mod(**inputs) 2025-08-26T20:25:15.2863897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2864326Z outputs = self.model( 2025-08-26T20:25:15.2864720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2865159Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2865561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2865973Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2866346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2866734Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2867146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2867591Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2868037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2868445Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2868899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2869387Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2869581Z 2025-08-26T20:25:15.2869687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2870049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2870376Z return mod(**inputs) 2025-08-26T20:25:15.2870746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2871133Z outputs = self.model( 2025-08-26T20:25:15.2871519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2871944Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2872339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2872756Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2873136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2873543Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2873962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2874444Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2874877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2875322Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2875811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2876307Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2876494Z 2025-08-26T20:25:15.2876610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2877005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2877370Z return mod(**inputs) 2025-08-26T20:25:15.2877768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2878192Z outputs = self.model( 2025-08-26T20:25:15.2878594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2879037Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2879531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2879957Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2880345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2880753Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2881182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2881628Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2882065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2882509Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2882671Z 2025-08-26T20:25:15.2882786Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2883186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2883592Z return mod(**inputs) 2025-08-26T20:25:15.2883984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2884402Z outputs = self.model( 2025-08-26T20:25:15.2884798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2885239Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2885642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2886063Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2886412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2886792Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2887206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2887660Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2887849Z 2025-08-26T20:25:15.2887962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2888346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2888692Z return mod(**inputs) 2025-08-26T20:25:15.2889056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2889432Z outputs = self.model( 2025-08-26T20:25:15.2889834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2890224Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2890607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2890990Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2891344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2891711Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2892099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2892528Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2892909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2893259Z return self.act(input) 2025-08-26T20:25:15.2893377Z 2025-08-26T20:25:15.2893483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2893847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2894182Z return mod(**inputs) 2025-08-26T20:25:15.2894570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2894978Z outputs = self.model( 2025-08-26T20:25:15.2895363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2895776Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2896282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2896688Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2897048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2897416Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2897807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2898360Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2898520Z 2025-08-26T20:25:15.2898635Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2899022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2899374Z return mod(**inputs) 2025-08-26T20:25:15.2899752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2900157Z outputs = self.model( 2025-08-26T20:25:15.2900538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2900932Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2901316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2901704Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2902082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2902469Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2902884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2903311Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2903736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2904227Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2904505Z 2025-08-26T20:25:15.2904614Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2904987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2905329Z return mod(**inputs) 2025-08-26T20:25:15.2905718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2906124Z outputs = self.model( 2025-08-26T20:25:15.2906512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2906945Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2907338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2907747Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2908121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2908524Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2908926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2909356Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2909781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2910196Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2910338Z 2025-08-26T20:25:15.2910456Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2910831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2911182Z return mod(**inputs) 2025-08-26T20:25:15.2911573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2911987Z outputs = self.model( 2025-08-26T20:25:15.2912373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2912778Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2913219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2913635Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2914008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2914095Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2914360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2914469Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2914732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2914837Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2914841Z 2025-08-26T20:25:15.2914931Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2915027Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2915113Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2915196Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2915321Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2915537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2915615Z return mod(**inputs) 2025-08-26T20:25:15.2915889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2915965Z outputs = self.model( 2025-08-26T20:25:15.2916245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2916362Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2916633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2916713Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2916952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2917049Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2917313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2917426Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2917697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2917817Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2918146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2918292Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2918296Z 2025-08-26T20:25:15.2918423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2918643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2918722Z return mod(**inputs) 2025-08-26T20:25:15.2918997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2919074Z outputs = self.model( 2025-08-26T20:25:15.2919433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2919522Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2919808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2919889Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2920132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2920271Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2920548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2920665Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2920915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2921023Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2921320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2921436Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2921440Z 2025-08-26T20:25:15.2921560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2921764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2921843Z return mod(**inputs) 2025-08-26T20:25:15.2922108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2922184Z outputs = self.model( 2025-08-26T20:25:15.2922459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2922541Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2922808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2922916Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2923145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2923227Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2923481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2923583Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2923829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2923921Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2923925Z 2025-08-26T20:25:15.2924030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2924231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2924308Z return mod(**inputs) 2025-08-26T20:25:15.2924563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2924640Z outputs = self.model( 2025-08-26T20:25:15.2924891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2924969Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2925223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2925298Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2925528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2925607Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2925862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2925989Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2925992Z 2025-08-26T20:25:15.2926098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2926305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2926371Z return mod(**inputs) 2025-08-26T20:25:15.2926663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2926735Z outputs = self.model( 2025-08-26T20:25:15.2926986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2927071Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2927321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2927402Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2927626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2927705Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2927958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2928085Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2928308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2928380Z return self.act(input) 2025-08-26T20:25:15.2928383Z 2025-08-26T20:25:15.2928495Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2928695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2928762Z return mod(**inputs) 2025-08-26T20:25:15.2929023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2929131Z outputs = self.model( 2025-08-26T20:25:15.2929390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2929466Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2929720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2929800Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2930026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2930115Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2930362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2930454Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2930461Z 2025-08-26T20:25:15.2930566Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2930765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2930840Z return mod(**inputs) 2025-08-26T20:25:15.2931096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2931175Z outputs = self.model( 2025-08-26T20:25:15.2931426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2931503Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2931759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2931833Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2932074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2932161Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2932422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2932530Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2932846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2933021Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2933026Z 2025-08-26T20:25:15.2933137Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2933356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2933426Z return mod(**inputs) 2025-08-26T20:25:15.2933703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2933797Z outputs = self.model( 2025-08-26T20:25:15.2934052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2934135Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2934392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2934470Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2934715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2934800Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2935071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2935170Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2935439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2935560Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2935564Z 2025-08-26T20:25:15.2935674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2935896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2935966Z return mod(**inputs) 2025-08-26T20:25:15.2936260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2936336Z outputs = self.model( 2025-08-26T20:25:15.2936616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2936704Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2936967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2937057Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2937297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2937380Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2937657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2937755Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2938029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2938123Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2938127Z 2025-08-26T20:25:15.2938222Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2938308Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2938391Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2938483Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2938593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2938815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2938886Z return mod(**inputs) 2025-08-26T20:25:15.2939189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2939272Z outputs = self.model( 2025-08-26T20:25:15.2939539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2939623Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2939889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2939966Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2940206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2940293Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2940559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2940657Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2940920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2941032Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2941343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2941491Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2941495Z 2025-08-26T20:25:15.2941604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2941869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2941939Z return mod(**inputs) 2025-08-26T20:25:15.2942207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2942290Z outputs = self.model( 2025-08-26T20:25:15.2942557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2942644Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2942907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2942984Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2943227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2943311Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2943583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2943681Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2943948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2944055Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2944366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2944494Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2944498Z 2025-08-26T20:25:15.2944608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2944826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2944900Z return mod(**inputs) 2025-08-26T20:25:15.2945168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2945249Z outputs = self.model( 2025-08-26T20:25:15.2945511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2945634Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2945901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2945979Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2946222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2946306Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2946574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2946678Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2946954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2947038Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2947042Z 2025-08-26T20:25:15.2947147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2947358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2947427Z return mod(**inputs) 2025-08-26T20:25:15.2947686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2947757Z outputs = self.model( 2025-08-26T20:25:15.2948011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2948094Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2948377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2948459Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2948680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2948763Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2949016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2949136Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2949139Z 2025-08-26T20:25:15.2949251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2949451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2949524Z return mod(**inputs) 2025-08-26T20:25:15.2949777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2949852Z outputs = self.model( 2025-08-26T20:25:15.2950112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2950188Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2950445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2950519Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2950742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2950829Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2951075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2951201Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2951419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2951496Z return self.act(input) 2025-08-26T20:25:15.2951499Z 2025-08-26T20:25:15.2951605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2951836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2951915Z return mod(**inputs) 2025-08-26T20:25:15.2952171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2952250Z outputs = self.model( 2025-08-26T20:25:15.2952500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2952576Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2952834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2952913Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2953141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2953222Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2953474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2953566Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2953570Z 2025-08-26T20:25:15.2953674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2953882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2953949Z return mod(**inputs) 2025-08-26T20:25:15.2954209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2954315Z outputs = self.model( 2025-08-26T20:25:15.2954568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2954651Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2954901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2954980Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2955203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2955281Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2955544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2955642Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2955913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2956076Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2956080Z 2025-08-26T20:25:15.2956197Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2956411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2956480Z return mod(**inputs) 2025-08-26T20:25:15.2956754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2956828Z outputs = self.model( 2025-08-26T20:25:15.2957100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2957178Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2957444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2957532Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2957767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2957857Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2958156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2958259Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2958528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2958615Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2958618Z 2025-08-26T20:25:15.2958736Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2958950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2959028Z return mod(**inputs) 2025-08-26T20:25:15.2959374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2959455Z outputs = self.model( 2025-08-26T20:25:15.2959732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2959816Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2960087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2960165Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2960402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2960495Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2960759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2960905Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2961171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2961274Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2961286Z 2025-08-26T20:25:15.2961372Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2961453Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2961539Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2961618Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2961724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2961933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2962000Z return mod(**inputs) 2025-08-26T20:25:15.2962262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2962335Z outputs = self.model( 2025-08-26T20:25:15.2962596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2962672Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2962920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2963003Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2963235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2963326Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2963590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2963695Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2963948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2964051Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2964353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2964515Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2964519Z 2025-08-26T20:25:15.2964631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2964828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2964896Z return mod(**inputs) 2025-08-26T20:25:15.2965171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2965243Z outputs = self.model( 2025-08-26T20:25:15.2965519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2965599Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2965861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2965947Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2966184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2966279Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2966541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2966641Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2966910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2967023Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2967365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2967479Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2967483Z 2025-08-26T20:25:15.2967593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2967798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2967864Z return mod(**inputs) 2025-08-26T20:25:15.2968126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2968199Z outputs = self.model( 2025-08-26T20:25:15.2968470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2968549Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2968810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2968898Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2969130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2969221Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2969490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2969595Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2969865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2969954Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2969958Z 2025-08-26T20:25:15.2970074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2970285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2970364Z return mod(**inputs) 2025-08-26T20:25:15.2970639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2970712Z outputs = self.model( 2025-08-26T20:25:15.2971029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2971110Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2971386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2971465Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2971704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2971796Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2972066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2972202Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2972206Z 2025-08-26T20:25:15.2972316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2972536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2972608Z return mod(**inputs) 2025-08-26T20:25:15.2972885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2972966Z outputs = self.model( 2025-08-26T20:25:15.2973243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2973330Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2973605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2973717Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2973965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2974049Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2974326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2974452Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2974684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2974768Z return self.act(input) 2025-08-26T20:25:15.2974771Z 2025-08-26T20:25:15.2974884Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2975104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2975179Z return mod(**inputs) 2025-08-26T20:25:15.2975462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2975538Z outputs = self.model( 2025-08-26T20:25:15.2975813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2975903Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2976173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2976257Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2976498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2976583Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2976862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.2976954Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.2976958Z 2025-08-26T20:25:15.2977076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2977288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2977406Z return mod(**inputs) 2025-08-26T20:25:15.2977679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2977754Z outputs = self.model( 2025-08-26T20:25:15.2978031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2978108Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2978383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2978459Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2978701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2978793Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2979058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2979168Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2979432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.2979594Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.2979605Z 2025-08-26T20:25:15.2979716Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2979931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2980009Z return mod(**inputs) 2025-08-26T20:25:15.2980319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2980401Z outputs = self.model( 2025-08-26T20:25:15.2980669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2980752Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2981026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2981104Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2981347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2981434Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2981700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2981809Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2982073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.2982164Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.2982167Z 2025-08-26T20:25:15.2982280Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2982501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2982572Z return mod(**inputs) 2025-08-26T20:25:15.2982844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2982927Z outputs = self.model( 2025-08-26T20:25:15.2983195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2983282Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2983551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2983627Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2983875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2983999Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2984271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2984370Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2984632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.2984734Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.2984738Z 2025-08-26T20:25:15.2984823Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2984919Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2985001Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2985082Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.2985199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2985410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2985489Z return mod(**inputs) 2025-08-26T20:25:15.2985758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2985830Z outputs = self.model( 2025-08-26T20:25:15.2986100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2986179Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2986446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2986559Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2986801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2986886Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2987153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2987259Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2987522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2987634Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2987944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.2988081Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.2988094Z 2025-08-26T20:25:15.2988197Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2988405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2988481Z return mod(**inputs) 2025-08-26T20:25:15.2988740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2988820Z outputs = self.model( 2025-08-26T20:25:15.2989087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2989167Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2989437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2989513Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2989754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2989843Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2990106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2990214Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2990522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.2990638Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.2990952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.2991079Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.2991082Z 2025-08-26T20:25:15.2991194Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2991406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2991489Z return mod(**inputs) 2025-08-26T20:25:15.2991764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2991846Z outputs = self.model( 2025-08-26T20:25:15.2992117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2992197Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2992475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2992554Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2992804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2992890Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2993156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.2993298Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.2993558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.2993656Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.2993660Z 2025-08-26T20:25:15.2993769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2993988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2994059Z return mod(**inputs) 2025-08-26T20:25:15.2994331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2994413Z outputs = self.model( 2025-08-26T20:25:15.2994677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2994766Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2995028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2995105Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2995349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2995434Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2995701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2995828Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2995832Z 2025-08-26T20:25:15.2995949Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2996275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2996363Z return mod(**inputs) 2025-08-26T20:25:15.2996644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2996718Z outputs = self.model( 2025-08-26T20:25:15.2997077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2997159Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.2997424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.2997511Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.2997745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.2997837Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.2998099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.2998232Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.2998477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.2998555Z return self.act(input) 2025-08-26T20:25:15.2998559Z 2025-08-26T20:25:15.2998682Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.2998904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.2998986Z return mod(**inputs) 2025-08-26T20:25:15.2999306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.2999392Z outputs = self.model( 2025-08-26T20:25:15.2999679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.2999761Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3000105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3000186Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3000435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3000533Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3000804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.3000903Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3000907Z 2025-08-26T20:25:15.3001032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3001242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3001321Z return mod(**inputs) 2025-08-26T20:25:15.3001589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3001670Z outputs = self.model( 2025-08-26T20:25:15.3001934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3002019Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3002282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3002360Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3002603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3002687Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3002957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.3003055Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.3003319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3003488Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3003492Z 2025-08-26T20:25:15.3003633Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3003854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3003925Z return mod(**inputs) 2025-08-26T20:25:15.3004196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3004272Z outputs = self.model( 2025-08-26T20:25:15.3004538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3004625Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3004887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3004972Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3005205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3005291Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3005562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.3005659Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.3005928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3006013Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3006017Z 2025-08-26T20:25:15.3006128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3006417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3006488Z return mod(**inputs) 2025-08-26T20:25:15.3006764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3006840Z outputs = self.model( 2025-08-26T20:25:15.3007117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3007197Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3007463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3007550Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3007787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3007878Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3008144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.3008243Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.3008511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3008608Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3008612Z 2025-08-26T20:25:15.3008706Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3008791Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3008877Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3008966Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3009076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3009298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3009370Z return mod(**inputs) 2025-08-26T20:25:15.3009643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3009725Z outputs = self.model( 2025-08-26T20:25:15.3009993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3010113Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3010377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3010467Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3010704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3010788Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3011057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.3011159Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.3011500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3011600Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3011900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3012044Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3012047Z 2025-08-26T20:25:15.3012151Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3012360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3012427Z return mod(**inputs) 2025-08-26T20:25:15.3012689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3012793Z outputs = self.model( 2025-08-26T20:25:15.3013045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3013127Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3013376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3013456Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3013677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3013756Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3014014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.3014111Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.3014380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3014487Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3014794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3014920Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3014927Z 2025-08-26T20:25:15.3015037Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3015251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3015322Z return mod(**inputs) 2025-08-26T20:25:15.3015594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3015669Z outputs = self.model( 2025-08-26T20:25:15.3015938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3016029Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3016332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3016418Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3017336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3017432Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3017712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-26T20:25:15.3017812Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:25:15.3018084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3018172Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3018176Z 2025-08-26T20:25:15.3018293Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3018507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3018577Z return mod(**inputs) 2025-08-26T20:25:15.3018854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3018928Z outputs = self.model( 2025-08-26T20:25:15.3019199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3019279Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3019549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3019633Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3019867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3019994Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3020299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.3020425Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3020436Z 2025-08-26T20:25:15.3020550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3020758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3020836Z return mod(**inputs) 2025-08-26T20:25:15.3021102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3021182Z outputs = self.model( 2025-08-26T20:25:15.3021446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3021525Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3021801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3021878Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3022123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3022210Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3022472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-26T20:25:15.3022606Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3022834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3022913Z return self.act(input) 2025-08-26T20:25:15.3022917Z 2025-08-26T20:25:15.3023021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3023220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3023295Z return mod(**inputs) 2025-08-26T20:25:15.3023546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3023626Z outputs = self.model( 2025-08-26T20:25:15.3023915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-26T20:25:15.3024001Z encoder_outputs = self.encoder( 2025-08-26T20:25:15.3024266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-26T20:25:15.3024343Z layer_outputs = encoder_layer( 2025-08-26T20:25:15.3024584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3024668Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3024944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-26T20:25:15.3025032Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3025036Z 2025-08-26T20:25:15.3025145Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3025365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3025436Z return mod(**inputs) 2025-08-26T20:25:15.3025711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3025785Z outputs = self.model( 2025-08-26T20:25:15.3026056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3026138Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3026389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3026504Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3026726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3026813Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3027064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3027168Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3027423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3027578Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3027582Z 2025-08-26T20:25:15.3027693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3027905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3027979Z return mod(**inputs) 2025-08-26T20:25:15.3028251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3028325Z outputs = self.model( 2025-08-26T20:25:15.3028596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3028676Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3028951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3029028Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3029262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3029347Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3029600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3029712Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3029959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3030076Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3030080Z 2025-08-26T20:25:15.3030195Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3030398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3030474Z return mod(**inputs) 2025-08-26T20:25:15.3030724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3030795Z outputs = self.model( 2025-08-26T20:25:15.3031053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3031130Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3031385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3031458Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3031690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3031770Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3032025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3032137Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3032404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3032505Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3032543Z 2025-08-26T20:25:15.3032632Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3032717Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3032809Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3032892Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3033012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3033222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3033292Z return mod(**inputs) 2025-08-26T20:25:15.3033568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3033643Z outputs = self.model( 2025-08-26T20:25:15.3033915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3033995Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3034262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3034347Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3034583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3034678Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3034939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3035055Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3035318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3035424Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3035748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3035893Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3035896Z 2025-08-26T20:25:15.3036017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3036232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3036345Z return mod(**inputs) 2025-08-26T20:25:15.3036624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3036698Z outputs = self.model( 2025-08-26T20:25:15.3036975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3037055Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3037328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3037408Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3037646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3037741Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3038014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3038133Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3038408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3038515Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3038843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3038964Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3039013Z 2025-08-26T20:25:15.3039134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3039430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3039518Z return mod(**inputs) 2025-08-26T20:25:15.3039803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3039882Z outputs = self.model( 2025-08-26T20:25:15.3040173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3040257Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3040549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3040630Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3040871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3040969Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3041241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3041361Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3041633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3041723Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3041737Z 2025-08-26T20:25:15.3041851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3042066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3042148Z return mod(**inputs) 2025-08-26T20:25:15.3042424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3042510Z outputs = self.model( 2025-08-26T20:25:15.3042783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3042864Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3043193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3043274Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3043517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3043601Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3043864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3043989Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3044256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3044427Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3044431Z 2025-08-26T20:25:15.3044544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3044767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3044837Z return mod(**inputs) 2025-08-26T20:25:15.3045104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3045187Z outputs = self.model( 2025-08-26T20:25:15.3045458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3045547Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3045823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3045950Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3046192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3046279Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3046549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3046668Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3046931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3047026Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3047030Z 2025-08-26T20:25:15.3047141Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3047361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3047435Z return mod(**inputs) 2025-08-26T20:25:15.3047713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3047788Z outputs = self.model( 2025-08-26T20:25:15.3048054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3048141Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3048409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3048492Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3048727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3048808Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3049078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3049197Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3049465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3049560Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3049564Z 2025-08-26T20:25:15.3049707Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3049791Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3049871Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3049956Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3050062Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3050265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3050338Z return mod(**inputs) 2025-08-26T20:25:15.3050593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3050675Z outputs = self.model( 2025-08-26T20:25:15.3050928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3051008Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3051265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3051341Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3051569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3051650Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3051908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3052017Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3052307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3052421Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3052734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3052884Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3052888Z 2025-08-26T20:25:15.3052999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3053220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3053291Z return mod(**inputs) 2025-08-26T20:25:15.3053560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3053643Z outputs = self.model( 2025-08-26T20:25:15.3053911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3054002Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3054277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3054355Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3054588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3054671Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3054946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3055061Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3055339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3055459Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3055778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3055907Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3055911Z 2025-08-26T20:25:15.3056071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3056290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3056361Z return mod(**inputs) 2025-08-26T20:25:15.3056636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3056720Z outputs = self.model( 2025-08-26T20:25:15.3056988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3057073Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3057344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3057423Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3057666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3057753Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3058039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3058154Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3058437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3058524Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3058528Z 2025-08-26T20:25:15.3058641Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3058906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3058977Z return mod(**inputs) 2025-08-26T20:25:15.3059250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3059326Z outputs = self.model( 2025-08-26T20:25:15.3059594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3059676Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3059925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3060006Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3060242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3060324Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3060597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3060725Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3060729Z 2025-08-26T20:25:15.3060848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3061063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3061140Z return mod(**inputs) 2025-08-26T20:25:15.3061406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3061480Z outputs = self.model( 2025-08-26T20:25:15.3061754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3061834Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3062107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3062184Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3062418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3062548Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3062816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3062950Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3063182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3063262Z return self.act(input) 2025-08-26T20:25:15.3063266Z 2025-08-26T20:25:15.3063376Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3063587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3063668Z return mod(**inputs) 2025-08-26T20:25:15.3063934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3064014Z outputs = self.model( 2025-08-26T20:25:15.3064283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3064360Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3064633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3064710Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3064955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3065040Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3065303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3065446Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3065450Z 2025-08-26T20:25:15.3065560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3065781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3065853Z return mod(**inputs) 2025-08-26T20:25:15.3066127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3066201Z outputs = self.model( 2025-08-26T20:25:15.3066467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3066554Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3066820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3066907Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3067141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3067224Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3067497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3067604Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3067874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3068033Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3068037Z 2025-08-26T20:25:15.3068155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3068368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3068442Z return mod(**inputs) 2025-08-26T20:25:15.3068716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3068790Z outputs = self.model( 2025-08-26T20:25:15.3069108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3069191Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3069465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3069562Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3069800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3069892Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3070153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3070264Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3070533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3070617Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3070623Z 2025-08-26T20:25:15.3070741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3070950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3071027Z return mod(**inputs) 2025-08-26T20:25:15.3071292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3071365Z outputs = self.model( 2025-08-26T20:25:15.3071640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3071754Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3072027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3072103Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3072341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3072432Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3072695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3072807Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3073070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3073163Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3073174Z 2025-08-26T20:25:15.3073264Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3073350Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3073441Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3073523Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3073634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3073859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3073929Z return mod(**inputs) 2025-08-26T20:25:15.3074202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3074277Z outputs = self.model( 2025-08-26T20:25:15.3074551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3074631Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3074897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3074985Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3075227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3075322Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3075628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3075742Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3076024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3076134Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3076464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3076614Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3076618Z 2025-08-26T20:25:15.3076740Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3076963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3077035Z return mod(**inputs) 2025-08-26T20:25:15.3077323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3077399Z outputs = self.model( 2025-08-26T20:25:15.3077679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3077762Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3078034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3078122Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3078399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3078492Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3078760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3078874Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3079152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3079333Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3079677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3079799Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3079804Z 2025-08-26T20:25:15.3079924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3080149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3080224Z return mod(**inputs) 2025-08-26T20:25:15.3080509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3080588Z outputs = self.model( 2025-08-26T20:25:15.3080874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3080967Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3081237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3081324Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3081561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3081657Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3081922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3082034Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3082340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3082429Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3082433Z 2025-08-26T20:25:15.3082550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3082764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3082841Z return mod(**inputs) 2025-08-26T20:25:15.3083111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3083184Z outputs = self.model( 2025-08-26T20:25:15.3083458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3083536Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3083805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3083886Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3084127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3084214Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3084461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3084580Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3084827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3085025Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3085029Z 2025-08-26T20:25:15.3085135Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3085337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3085415Z return mod(**inputs) 2025-08-26T20:25:15.3085665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3085743Z outputs = self.model( 2025-08-26T20:25:15.3085996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3086069Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3086328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3086404Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3086635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3086712Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3086963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3087081Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3087345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3087438Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3087442Z 2025-08-26T20:25:15.3087551Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3087776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3087843Z return mod(**inputs) 2025-08-26T20:25:15.3088098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3088176Z outputs = self.model( 2025-08-26T20:25:15.3088425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3088552Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3088808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3088883Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3089112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3089193Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3089449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3089560Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3089817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3089907Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3089910Z 2025-08-26T20:25:15.3089996Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3090089Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3090174Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3090262Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3090373Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3090585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3090663Z return mod(**inputs) 2025-08-26T20:25:15.3090927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3091074Z outputs = self.model( 2025-08-26T20:25:15.3091343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3091423Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3091697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3091776Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3092029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3092109Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3092360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3092476Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3092723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3092830Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3093123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3093265Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3093269Z 2025-08-26T20:25:15.3093376Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3093574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3093649Z return mod(**inputs) 2025-08-26T20:25:15.3093899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3093976Z outputs = self.model( 2025-08-26T20:25:15.3094228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3094307Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3094571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3094643Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3094905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3094989Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3095245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3095353Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3095603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3095707Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3096003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3096121Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3096125Z 2025-08-26T20:25:15.3096358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3096583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3096666Z return mod(**inputs) 2025-08-26T20:25:15.3096937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3097019Z outputs = self.model( 2025-08-26T20:25:15.3097288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3097368Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3097643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3097805Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3098051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3098142Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3098411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3098528Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3098789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3098885Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3098889Z 2025-08-26T20:25:15.3098999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3099218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3099293Z return mod(**inputs) 2025-08-26T20:25:15.3099561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3099643Z outputs = self.model( 2025-08-26T20:25:15.3099915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3100003Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3100272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3100358Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3100598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3100682Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3100958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3101086Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3101090Z 2025-08-26T20:25:15.3101208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3101472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3101546Z return mod(**inputs) 2025-08-26T20:25:15.3101821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3101896Z outputs = self.model( 2025-08-26T20:25:15.3102165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3102245Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3102512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3102598Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3102832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3102922Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3103188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3103323Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3103551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3103628Z return self.act(input) 2025-08-26T20:25:15.3103631Z 2025-08-26T20:25:15.3103750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3103959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3104079Z return mod(**inputs) 2025-08-26T20:25:15.3104352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3104425Z outputs = self.model( 2025-08-26T20:25:15.3104710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3104791Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3105070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3105149Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3105387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3105481Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3105747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3105849Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3105853Z 2025-08-26T20:25:15.3105964Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3106184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3106261Z return mod(**inputs) 2025-08-26T20:25:15.3106529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3106612Z outputs = self.model( 2025-08-26T20:25:15.3106881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3106969Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3107235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3107317Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3107561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3107645Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3107955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3108065Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3108336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3108497Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3108501Z 2025-08-26T20:25:15.3108610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3108826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3108900Z return mod(**inputs) 2025-08-26T20:25:15.3109173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3109246Z outputs = self.model( 2025-08-26T20:25:15.3109513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3109602Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3109867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3109954Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3110190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3110272Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3110542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3110715Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3110986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3111070Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3111074Z 2025-08-26T20:25:15.3111191Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3111404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3111474Z return mod(**inputs) 2025-08-26T20:25:15.3111745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3111818Z outputs = self.model( 2025-08-26T20:25:15.3112094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3112172Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3112441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3112527Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3112763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3112857Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3113123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3113236Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3113500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3113593Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3113597Z 2025-08-26T20:25:15.3113694Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3113783Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3113875Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3113958Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3114070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3114336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3114410Z return mod(**inputs) 2025-08-26T20:25:15.3114687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3114762Z outputs = self.model( 2025-08-26T20:25:15.3115027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3115115Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3115385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3115478Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3115714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3115799Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3116071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3120835Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3121149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3121260Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3121595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3121751Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3121817Z 2025-08-26T20:25:15.3121939Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3122169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3122245Z return mod(**inputs) 2025-08-26T20:25:15.3122532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3122653Z outputs = self.model( 2025-08-26T20:25:15.3122942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3123028Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3123308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3123399Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3123651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3123751Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3124026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3124148Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3124424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3124534Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3124868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3124991Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3124995Z 2025-08-26T20:25:15.3125118Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3125344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3125419Z return mod(**inputs) 2025-08-26T20:25:15.3125706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3125783Z outputs = self.model( 2025-08-26T20:25:15.3126121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3126209Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3126492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3126573Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3126819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3126917Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3127192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3127308Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3127580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3127676Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3127680Z 2025-08-26T20:25:15.3127874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3128092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3128176Z return mod(**inputs) 2025-08-26T20:25:15.3128453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3128533Z outputs = self.model( 2025-08-26T20:25:15.3128818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3128923Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3129205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3129288Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3129542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3129630Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3129905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3130029Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3130291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3130458Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3130464Z 2025-08-26T20:25:15.3130574Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3130784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3149376Z return mod(**inputs) 2025-08-26T20:25:15.3149921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3150012Z outputs = self.model( 2025-08-26T20:25:15.3150314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3150404Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3150681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3150774Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3151029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3151132Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3151408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3151648Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3151936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3152030Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3152036Z 2025-08-26T20:25:15.3152167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3152395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3152478Z return mod(**inputs) 2025-08-26T20:25:15.3152753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3152834Z outputs = self.model( 2025-08-26T20:25:15.3153110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3153194Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3153476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3153637Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3153883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3153980Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3154250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3154383Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3154693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3154801Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3154805Z 2025-08-26T20:25:15.3154899Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3154991Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3155086Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3155171Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3155301Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3155529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3155605Z return mod(**inputs) 2025-08-26T20:25:15.3155892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3155971Z outputs = self.model( 2025-08-26T20:25:15.3156251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3156334Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3156605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3156696Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3156939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3157040Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3157311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3157431Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3157707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3157821Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3158154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3158303Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3158340Z 2025-08-26T20:25:15.3158466Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3158698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3158776Z return mod(**inputs) 2025-08-26T20:25:15.3159065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3159144Z outputs = self.model( 2025-08-26T20:25:15.3159521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3159614Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3159894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3159982Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3160230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3160328Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3160627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3160752Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3161023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3161135Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3161465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3161607Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3161611Z 2025-08-26T20:25:15.3161733Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3161958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3162032Z return mod(**inputs) 2025-08-26T20:25:15.3162313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3162389Z outputs = self.model( 2025-08-26T20:25:15.3162670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3162752Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3163030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3163110Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3163348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3163443Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3163711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3163836Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3164104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3164194Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3164198Z 2025-08-26T20:25:15.3164318Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3164537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3164615Z return mod(**inputs) 2025-08-26T20:25:15.3164889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3164970Z outputs = self.model( 2025-08-26T20:25:15.3165274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3165357Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3165643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3165722Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3165971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3166056Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3166329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3166471Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3166476Z 2025-08-26T20:25:15.3166590Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3166819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3166889Z return mod(**inputs) 2025-08-26T20:25:15.3167174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3167273Z outputs = self.model( 2025-08-26T20:25:15.3167545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3167629Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3167899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3168000Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3168235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3168320Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3168592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3168721Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3168957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3169032Z return self.act(input) 2025-08-26T20:25:15.3169035Z 2025-08-26T20:25:15.3169147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3169367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3169437Z return mod(**inputs) 2025-08-26T20:25:15.3169711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3169784Z outputs = self.model( 2025-08-26T20:25:15.3170047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3170135Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3170403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3170490Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3170727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3170818Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3171080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3171170Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3171174Z 2025-08-26T20:25:15.3171291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3171501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3171578Z return mod(**inputs) 2025-08-26T20:25:15.3171874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3171950Z outputs = self.model( 2025-08-26T20:25:15.3172226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3172303Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3172571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3172647Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3172883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3172975Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3173237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3173356Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3173622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3173813Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3173818Z 2025-08-26T20:25:15.3173927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3174138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3174215Z return mod(**inputs) 2025-08-26T20:25:15.3174498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3174577Z outputs = self.model( 2025-08-26T20:25:15.3174842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3174921Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3175192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3175269Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3175513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3175597Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3175867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3175979Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3176257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3176352Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3176356Z 2025-08-26T20:25:15.3176466Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3176685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3176757Z return mod(**inputs) 2025-08-26T20:25:15.3177032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3177112Z outputs = self.model( 2025-08-26T20:25:15.3177384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3177468Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3177754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3177832Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3178073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3178203Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3178479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3178585Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3178853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3178944Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3178948Z 2025-08-26T20:25:15.3179035Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3179126Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3179210Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3179297Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3179406Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3179618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3179697Z return mod(**inputs) 2025-08-26T20:25:15.3179963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3180063Z outputs = self.model( 2025-08-26T20:25:15.3180332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3180410Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3180687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3180786Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3181028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3181111Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3181378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3181488Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3181751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3181862Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3182175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3182325Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3182331Z 2025-08-26T20:25:15.3182439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3182650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3182728Z return mod(**inputs) 2025-08-26T20:25:15.3182998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3183079Z outputs = self.model( 2025-08-26T20:25:15.3183349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3183426Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3183699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3183775Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3184014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3184100Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3184369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3184472Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3184766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3184881Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3185194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3185317Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3185321Z 2025-08-26T20:25:15.3185430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3185643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3185722Z return mod(**inputs) 2025-08-26T20:25:15.3185988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3186068Z outputs = self.model( 2025-08-26T20:25:15.3186338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3186422Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3186709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3186785Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3187030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3187116Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3187387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3187511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3187772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3187871Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3187875Z 2025-08-26T20:25:15.3187984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3188206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3188275Z return mod(**inputs) 2025-08-26T20:25:15.3188541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3188618Z outputs = self.model( 2025-08-26T20:25:15.3188884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3188972Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3189238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3189320Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3189560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3189642Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3189918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3190033Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3190305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3190466Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3190472Z 2025-08-26T20:25:15.3190582Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3190802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3190871Z return mod(**inputs) 2025-08-26T20:25:15.3191175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3191250Z outputs = self.model( 2025-08-26T20:25:15.3191528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3191605Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3191868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3191952Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3192190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3192285Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3192544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3192657Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3192929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3193035Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3193039Z 2025-08-26T20:25:15.3193156Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3193369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3193439Z return mod(**inputs) 2025-08-26T20:25:15.3193711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3193803Z outputs = self.model( 2025-08-26T20:25:15.3194072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3194150Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3194423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3194502Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3194736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3194826Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3195086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3195206Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3195468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3195559Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3195563Z 2025-08-26T20:25:15.3195655Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3195740Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3195834Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3195916Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3196028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3196553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3196630Z return mod(**inputs) 2025-08-26T20:25:15.3196913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3196987Z outputs = self.model( 2025-08-26T20:25:15.3197271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3197364Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3197646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3197735Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3198075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3198173Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3198442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3198562Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3198837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3198946Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3199330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3199489Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3199493Z 2025-08-26T20:25:15.3199618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3199836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3199953Z return mod(**inputs) 2025-08-26T20:25:15.3200239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3200313Z outputs = self.model( 2025-08-26T20:25:15.3200606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3200687Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3200990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3201079Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3201322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3201422Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3201696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3201815Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3202106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3202214Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3202543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3202665Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3202669Z 2025-08-26T20:25:15.3202788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3203012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3203085Z return mod(**inputs) 2025-08-26T20:25:15.3203364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3203439Z outputs = self.model( 2025-08-26T20:25:15.3203731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3203811Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3204140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3204230Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3204473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3204566Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3204891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3205012Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3205317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3205411Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3205415Z 2025-08-26T20:25:15.3205540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3205762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3205846Z return mod(**inputs) 2025-08-26T20:25:15.3206134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3206212Z outputs = self.model( 2025-08-26T20:25:15.3206501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3206590Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3206891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3206994Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3207238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3207335Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3207615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3207773Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3207777Z 2025-08-26T20:25:15.3207888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3208112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3208187Z return mod(**inputs) 2025-08-26T20:25:15.3208468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3208552Z outputs = self.model( 2025-08-26T20:25:15.3208817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3208901Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3209164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3209240Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3209485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3209567Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3209839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3209957Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3210173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3210251Z return self.act(input) 2025-08-26T20:25:15.3210254Z 2025-08-26T20:25:15.3210357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3210561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3210627Z return mod(**inputs) 2025-08-26T20:25:15.3210881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3210951Z outputs = self.model( 2025-08-26T20:25:15.3211198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3211279Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3211562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3211643Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3211862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3211942Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3212196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3212277Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3212283Z 2025-08-26T20:25:15.3212392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3212589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3212653Z return mod(**inputs) 2025-08-26T20:25:15.3212910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3212997Z outputs = self.model( 2025-08-26T20:25:15.3213253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3213325Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3213580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3213651Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3213873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3213977Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3214224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3214331Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3214583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3214740Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3214744Z 2025-08-26T20:25:15.3214856Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3215053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3215124Z return mod(**inputs) 2025-08-26T20:25:15.3215374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3215452Z outputs = self.model( 2025-08-26T20:25:15.3215705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3215777Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3216045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3216122Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3216368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3216453Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3216718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3216827Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3217081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3217168Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3217171Z 2025-08-26T20:25:15.3217274Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3217505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3217579Z return mod(**inputs) 2025-08-26T20:25:15.3217836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3217915Z outputs = self.model( 2025-08-26T20:25:15.3218180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3218265Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3218528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3218607Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3218852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3218935Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3219205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3219325Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3219573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3219671Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3219675Z 2025-08-26T20:25:15.3219759Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3219850Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3219933Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3220027Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3220140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3220341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3220418Z return mod(**inputs) 2025-08-26T20:25:15.3220686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3220761Z outputs = self.model( 2025-08-26T20:25:15.3221039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3221112Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3221368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3221439Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3221670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3221753Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3222003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3222113Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3222378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3222488Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3222809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3222950Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3222953Z 2025-08-26T20:25:15.3223057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3223257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3223332Z return mod(**inputs) 2025-08-26T20:25:15.3223585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3223693Z outputs = self.model( 2025-08-26T20:25:15.3223949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3224024Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3224287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3224357Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3224580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3224659Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3224907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3225014Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3225268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3225374Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3225686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3225803Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3225807Z 2025-08-26T20:25:15.3225911Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3226111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3226211Z return mod(**inputs) 2025-08-26T20:25:15.3226462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3226537Z outputs = self.model( 2025-08-26T20:25:15.3226787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3226861Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3227120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3227193Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3227430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3227509Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3227763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3227865Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3228115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3228204Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3228207Z 2025-08-26T20:25:15.3228313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3228520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3228587Z return mod(**inputs) 2025-08-26T20:25:15.3228837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3228915Z outputs = self.model( 2025-08-26T20:25:15.3229165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3229246Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3229495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3229566Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3229858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3229940Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3230195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3230304Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3230557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3230712Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3230716Z 2025-08-26T20:25:15.3230824Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3231046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3231116Z return mod(**inputs) 2025-08-26T20:25:15.3231393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3231465Z outputs = self.model( 2025-08-26T20:25:15.3231750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3231836Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3232099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3232184Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3232419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3232532Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3232795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3232911Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3233186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3233271Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3233275Z 2025-08-26T20:25:15.3233392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3233602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3233670Z return mod(**inputs) 2025-08-26T20:25:15.3233943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3234017Z outputs = self.model( 2025-08-26T20:25:15.3234290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3234368Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3234631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3234712Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3234947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3235037Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3235301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3235422Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3235684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3235778Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3235781Z 2025-08-26T20:25:15.3235872Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3235954Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3236043Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3236154Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3236265Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3236487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3236557Z return mod(**inputs) 2025-08-26T20:25:15.3236827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3236898Z outputs = self.model( 2025-08-26T20:25:15.3237161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3237249Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3237510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3237593Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3237830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3237932Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3238201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3238313Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3238581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3238684Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3239020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3239164Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3239168Z 2025-08-26T20:25:15.3239364Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3239597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3239671Z return mod(**inputs) 2025-08-26T20:25:15.3239948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3240021Z outputs = self.model( 2025-08-26T20:25:15.3240292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3240383Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3240657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3240748Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3240991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3241089Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3241360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3241478Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3241758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3241858Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3242161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3242270Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3242274Z 2025-08-26T20:25:15.3242376Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3242586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3242687Z return mod(**inputs) 2025-08-26T20:25:15.3242961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3243038Z outputs = self.model( 2025-08-26T20:25:15.3243320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3243400Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3243671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3243759Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3244003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3244095Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3244363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3244483Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3244788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3244876Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3244880Z 2025-08-26T20:25:15.3245000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3245218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3245289Z return mod(**inputs) 2025-08-26T20:25:15.3245584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3245659Z outputs = self.model( 2025-08-26T20:25:15.3245937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3246020Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3246298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3246377Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3246619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3246712Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3246979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3247117Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3247121Z 2025-08-26T20:25:15.3247233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3247451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3247530Z return mod(**inputs) 2025-08-26T20:25:15.3247804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3247886Z outputs = self.model( 2025-08-26T20:25:15.3248161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3248248Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3248519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3248598Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3248850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3248936Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3249213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3249378Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3249614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3249700Z return self.act(input) 2025-08-26T20:25:15.3249704Z 2025-08-26T20:25:15.3249816Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3250038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3250108Z return mod(**inputs) 2025-08-26T20:25:15.3250376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3250457Z outputs = self.model( 2025-08-26T20:25:15.3250730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3250817Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3251092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3251230Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3251475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3251561Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3251844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3251934Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3251938Z 2025-08-26T20:25:15.3252076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3252290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3252361Z return mod(**inputs) 2025-08-26T20:25:15.3252639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3252713Z outputs = self.model( 2025-08-26T20:25:15.3252995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3253071Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3253332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3253414Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3253649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3253741Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3254004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3254117Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3254383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3254544Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3254548Z 2025-08-26T20:25:15.3254667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3254877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3254953Z return mod(**inputs) 2025-08-26T20:25:15.3255217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3255291Z outputs = self.model( 2025-08-26T20:25:15.3255564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3255641Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3255944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3256025Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3256261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3256352Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3256615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3256729Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3256997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3257093Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3257096Z 2025-08-26T20:25:15.3257204Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3257419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3257497Z return mod(**inputs) 2025-08-26T20:25:15.3257762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3257863Z outputs = self.model( 2025-08-26T20:25:15.3258125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3258202Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3258474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3258575Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3258818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3258901Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3259179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3259288Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3259561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3259672Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3259676Z 2025-08-26T20:25:15.3259761Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3259852Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3259935Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3260016Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3260132Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3260344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3260420Z return mod(**inputs) 2025-08-26T20:25:15.3260686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3260759Z outputs = self.model( 2025-08-26T20:25:15.3261032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3261110Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3261379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3261454Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3261688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3261782Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3262043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3262155Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3262453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3262569Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3262879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3263020Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3263024Z 2025-08-26T20:25:15.3263142Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3263357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3263433Z return mod(**inputs) 2025-08-26T20:25:15.3263698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3263772Z outputs = self.model( 2025-08-26T20:25:15.3264046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3264142Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3264419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3264494Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3264740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3264822Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3265103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3265214Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3265474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3265589Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3265901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3266014Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3266018Z 2025-08-26T20:25:15.3266137Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3266345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3266422Z return mod(**inputs) 2025-08-26T20:25:15.3266686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3266759Z outputs = self.model( 2025-08-26T20:25:15.3267031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3267110Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3267384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3267460Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3267700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3267784Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3268048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3268166Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3268446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3268540Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3268543Z 2025-08-26T20:25:15.3268686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3268900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3268978Z return mod(**inputs) 2025-08-26T20:25:15.3269251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3269333Z outputs = self.model( 2025-08-26T20:25:15.3269608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3269698Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3269981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3270059Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3270306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3270395Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3270678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3270826Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3271097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3271270Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3271274Z 2025-08-26T20:25:15.3271386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3271637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3271709Z return mod(**inputs) 2025-08-26T20:25:15.3271987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3272066Z outputs = self.model( 2025-08-26T20:25:15.3272337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3272427Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3272697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3272784Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3273027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3273113Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3273392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3273522Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3273803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3273890Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3273896Z 2025-08-26T20:25:15.3274009Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3274236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3274307Z return mod(**inputs) 2025-08-26T20:25:15.3274588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3274664Z outputs = self.model( 2025-08-26T20:25:15.3274939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3275027Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3275299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3275431Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3275679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3275772Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3276042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3276159Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3276438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3276536Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3276540Z 2025-08-26T20:25:15.3276638Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3276727Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3276812Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3276903Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3277021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3277269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3277341Z return mod(**inputs) 2025-08-26T20:25:15.3277615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3277701Z outputs = self.model( 2025-08-26T20:25:15.3277975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3278086Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3278358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3278438Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3278691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3278779Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3279061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3279179Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3279556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3279669Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3279996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3280152Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3280157Z 2025-08-26T20:25:15.3280270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3280500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3280573Z return mod(**inputs) 2025-08-26T20:25:15.3280849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3280933Z outputs = self.model( 2025-08-26T20:25:15.3281274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3281363Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3281635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3281725Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3281965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3282050Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3282366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3282485Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3282771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3282876Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3283196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3283321Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3283326Z 2025-08-26T20:25:15.3283438Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3283660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3283741Z return mod(**inputs) 2025-08-26T20:25:15.3284015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3284107Z outputs = self.model( 2025-08-26T20:25:15.3284372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3284458Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3284723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3284806Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3285064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3285149Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3285430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3285545Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3285836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3285926Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3285930Z 2025-08-26T20:25:15.3286041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3286268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3286340Z return mod(**inputs) 2025-08-26T20:25:15.3286619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3286695Z outputs = self.model( 2025-08-26T20:25:15.3286984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3287064Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3287328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3287414Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3287650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3287741Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3288010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3288136Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3288142Z 2025-08-26T20:25:15.3288258Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3288469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3288547Z return mod(**inputs) 2025-08-26T20:25:15.3288847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3288922Z outputs = self.model( 2025-08-26T20:25:15.3289199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3289277Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3289553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3289629Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3289871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3289956Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3290258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3290391Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3290622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3290728Z return self.act(input) 2025-08-26T20:25:15.3290731Z 2025-08-26T20:25:15.3290841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3291066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3291141Z return mod(**inputs) 2025-08-26T20:25:15.3291410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3291511Z outputs = self.model( 2025-08-26T20:25:15.3291783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3291868Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3292139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3292217Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3292465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3292549Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3292825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3292914Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3292918Z 2025-08-26T20:25:15.3293030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3293260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3293332Z return mod(**inputs) 2025-08-26T20:25:15.3293626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3293702Z outputs = self.model( 2025-08-26T20:25:15.3293972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3294058Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3294328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3294411Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3294649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3294741Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3295008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3295113Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3295749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3295917Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3295923Z 2025-08-26T20:25:15.3296042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3296479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3296555Z return mod(**inputs) 2025-08-26T20:25:15.3296833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3296910Z outputs = self.model( 2025-08-26T20:25:15.3297189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3297268Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3297539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3297625Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3297863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3298010Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3298274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3298387Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3298648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3298763Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3298767Z 2025-08-26T20:25:15.3298887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3299100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3299180Z return mod(**inputs) 2025-08-26T20:25:15.3299446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3299520Z outputs = self.model( 2025-08-26T20:25:15.3299794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3299872Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3300144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3300224Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3300471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3300555Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3300819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3300934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3301199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3301300Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3301304Z 2025-08-26T20:25:15.3301392Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3301474Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3301564Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3301647Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3301767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3301979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3302049Z return mod(**inputs) 2025-08-26T20:25:15.3302418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3302495Z outputs = self.model( 2025-08-26T20:25:15.3302774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3302855Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3303122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3303206Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3303442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3303537Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3303799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3303909Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3304175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3304303Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3304626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3304767Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3304771Z 2025-08-26T20:25:15.3304891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3305101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3305193Z return mod(**inputs) 2025-08-26T20:25:15.3305466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3305538Z outputs = self.model( 2025-08-26T20:25:15.3305810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3305891Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3306160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3306237Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3306474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3306564Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3306826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3306937Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3307196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3307302Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3307619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3307737Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3307740Z 2025-08-26T20:25:15.3307857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3308066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3308136Z return mod(**inputs) 2025-08-26T20:25:15.3308407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3308483Z outputs = self.model( 2025-08-26T20:25:15.3308758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3308836Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3309140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3309222Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3309457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3309551Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3309812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3309923Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3310186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3310273Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3310276Z 2025-08-26T20:25:15.3310393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3310607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3310703Z return mod(**inputs) 2025-08-26T20:25:15.3310965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3311044Z outputs = self.model( 2025-08-26T20:25:15.3311304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3311381Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3311650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3311749Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3311988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3312072Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3312345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3312464Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3312708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3312866Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3312870Z 2025-08-26T20:25:15.3312973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3313179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3313245Z return mod(**inputs) 2025-08-26T20:25:15.3313496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3313577Z outputs = self.model( 2025-08-26T20:25:15.3313844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3313932Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3314197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3314272Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3314513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3314594Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3314865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3314981Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3315242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3315372Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3315378Z 2025-08-26T20:25:15.3315489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3315707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3315778Z return mod(**inputs) 2025-08-26T20:25:15.3316050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3316122Z outputs = self.model( 2025-08-26T20:25:15.3316388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3316476Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3316740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3316823Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3317062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3317163Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3317434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3317550Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3317820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3317913Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3317934Z 2025-08-26T20:25:15.3318020Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3318110Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3318192Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3318280Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3318397Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3318613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3318695Z return mod(**inputs) 2025-08-26T20:25:15.3318971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3319053Z outputs = self.model( 2025-08-26T20:25:15.3319394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3319489Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3319765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3319844Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3320098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3320188Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3320469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3320589Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3320852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3320963Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3321256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3321401Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3321405Z 2025-08-26T20:25:15.3321509Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3321760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3321833Z return mod(**inputs) 2025-08-26T20:25:15.3322102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3322187Z outputs = self.model( 2025-08-26T20:25:15.3322455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3322542Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3322804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3322882Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3323123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3323207Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3323480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3323594Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3323884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3323990Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3324283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3324399Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3324430Z 2025-08-26T20:25:15.3324535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3324741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3324807Z return mod(**inputs) 2025-08-26T20:25:15.3325063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3325147Z outputs = self.model( 2025-08-26T20:25:15.3325415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3325501Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3325766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3325841Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3326083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3326170Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3326440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3326553Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3326824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3326913Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3326917Z 2025-08-26T20:25:15.3327026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3327247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3327315Z return mod(**inputs) 2025-08-26T20:25:15.3327588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3327661Z outputs = self.model( 2025-08-26T20:25:15.3327925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3328010Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3328311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3328396Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3328633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3328715Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3328985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3329112Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3329116Z 2025-08-26T20:25:15.3329236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3329448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3329523Z return mod(**inputs) 2025-08-26T20:25:15.3329790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3329861Z outputs = self.model( 2025-08-26T20:25:15.3330133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3330229Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3330498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3330574Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3330809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3330951Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3331214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3331346Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3331576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3331650Z return self.act(input) 2025-08-26T20:25:15.3331660Z 2025-08-26T20:25:15.3331769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3331981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3332058Z return mod(**inputs) 2025-08-26T20:25:15.3332324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3332405Z outputs = self.model( 2025-08-26T20:25:15.3332674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3332753Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3333029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3333110Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3333353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3333440Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3333704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3333802Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3333806Z 2025-08-26T20:25:15.3333914Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3334139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3334210Z return mod(**inputs) 2025-08-26T20:25:15.3334476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3334559Z outputs = self.model( 2025-08-26T20:25:15.3334858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3334948Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3335212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3335296Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3335531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3335613Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3335885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3335998Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3336267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3336431Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3336454Z 2025-08-26T20:25:15.3336564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3336784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3336854Z return mod(**inputs) 2025-08-26T20:25:15.3337125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3337198Z outputs = self.model( 2025-08-26T20:25:15.3337489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3337567Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3337829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3337918Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3338153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3338245Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3338506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3338611Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3338885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3338972Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3338975Z 2025-08-26T20:25:15.3339091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3339302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3339379Z return mod(**inputs) 2025-08-26T20:25:15.3339648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3339723Z outputs = self.model( 2025-08-26T20:25:15.3339991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3340070Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3340340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3340416Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3340652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3340743Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3341006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3341151Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3341413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3341506Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3341517Z 2025-08-26T20:25:15.3341603Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3341686Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3341774Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3341853Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3341962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3342183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3342252Z return mod(**inputs) 2025-08-26T20:25:15.3342527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3342600Z outputs = self.model( 2025-08-26T20:25:15.3342862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3342968Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3343231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3343314Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3343548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3343657Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3343921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3344024Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3344294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3344397Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3344713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3344853Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3344856Z 2025-08-26T20:25:15.3344971Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3345182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3345254Z return mod(**inputs) 2025-08-26T20:25:15.3345530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3345603Z outputs = self.model( 2025-08-26T20:25:15.3345885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3345962Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3346249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3346333Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3346569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3346659Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3346922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3347028Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3347307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3347410Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3347759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3347880Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3347884Z 2025-08-26T20:25:15.3348001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3348211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3348282Z return mod(**inputs) 2025-08-26T20:25:15.3348554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3348630Z outputs = self.model( 2025-08-26T20:25:15.3348920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3348998Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3349279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3349394Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3349634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3349725Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3350004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3350103Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3350363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3350466Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3350470Z 2025-08-26T20:25:15.3350580Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3350780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3350854Z return mod(**inputs) 2025-08-26T20:25:15.3351106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3351175Z outputs = self.model( 2025-08-26T20:25:15.3351431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3351504Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3351760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3351835Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3352067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3352158Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3352423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3352547Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3352818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3352984Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3352988Z 2025-08-26T20:25:15.3353096Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3353305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3353385Z return mod(**inputs) 2025-08-26T20:25:15.3353661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3353742Z outputs = self.model( 2025-08-26T20:25:15.3354039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3354119Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3354397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3354473Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3354714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3354797Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3355057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3355180Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3355439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3355537Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3355540Z 2025-08-26T20:25:15.3355650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3355906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3355978Z return mod(**inputs) 2025-08-26T20:25:15.3356249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3356331Z outputs = self.model( 2025-08-26T20:25:15.3356601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3356707Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3356983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3357059Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3357317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3357403Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3357687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3357803Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3358085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3358178Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3358184Z 2025-08-26T20:25:15.3358270Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3358364Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3358447Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3358544Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3358654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3358873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3358953Z return mod(**inputs) 2025-08-26T20:25:15.3359224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3359388Z outputs = self.model( 2025-08-26T20:25:15.3359651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3359728Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3360006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3360088Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3360336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3360422Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3360737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3360856Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3361106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3361214Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3361511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3361654Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3361658Z 2025-08-26T20:25:15.3361761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3361977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3362059Z return mod(**inputs) 2025-08-26T20:25:15.3362328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3362433Z outputs = self.model( 2025-08-26T20:25:15.3362704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3362784Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3363060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3363136Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3363403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3363487Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3363750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3363876Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3364139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3364252Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3364563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3364684Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3364687Z 2025-08-26T20:25:15.3364799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3365009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3365090Z return mod(**inputs) 2025-08-26T20:25:15.3365353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3365435Z outputs = self.model( 2025-08-26T20:25:15.3365701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3365780Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3366050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3366128Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3366368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3366453Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3366723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3366835Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3367135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3367234Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3367238Z 2025-08-26T20:25:15.3367346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3367562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3367632Z return mod(**inputs) 2025-08-26T20:25:15.3367895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3367975Z outputs = self.model( 2025-08-26T20:25:15.3368241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3368325Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3368594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3368671Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3368918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3369021Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3369286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3369413Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3369417Z 2025-08-26T20:25:15.3369533Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3369762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3369832Z return mod(**inputs) 2025-08-26T20:25:15.3370101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3370178Z outputs = self.model( 2025-08-26T20:25:15.3370450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3370530Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3370791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3370876Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3371110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3371202Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3371469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3371602Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3371832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3371908Z return self.act(input) 2025-08-26T20:25:15.3371913Z 2025-08-26T20:25:15.3372032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3372242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3372319Z return mod(**inputs) 2025-08-26T20:25:15.3372583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3372656Z outputs = self.model( 2025-08-26T20:25:15.3372925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3373004Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3373276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3373352Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3373639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3373732Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3373995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3374089Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3374093Z 2025-08-26T20:25:15.3374201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3374416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3374487Z return mod(**inputs) 2025-08-26T20:25:15.3374749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3374828Z outputs = self.model( 2025-08-26T20:25:15.3375092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3375174Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3375457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3375533Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3375777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3375860Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3376126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3376251Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3376512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3376682Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3376686Z 2025-08-26T20:25:15.3376796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3377016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3377086Z return mod(**inputs) 2025-08-26T20:25:15.3377360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3377431Z outputs = self.model( 2025-08-26T20:25:15.3377699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3377788Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3378054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3378138Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3378375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3378461Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3378733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3378840Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3379111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3379197Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3379203Z 2025-08-26T20:25:15.3379317Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3379529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3379600Z return mod(**inputs) 2025-08-26T20:25:15.3379906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3379981Z outputs = self.model( 2025-08-26T20:25:15.3380254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3380332Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3380595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3380679Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3380919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3381014Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3381282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3381389Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3381668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3381783Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3381787Z 2025-08-26T20:25:15.3381881Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3381969Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3382069Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3382150Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3382259Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3382495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3382566Z return mod(**inputs) 2025-08-26T20:25:15.3382846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3382920Z outputs = self.model( 2025-08-26T20:25:15.3383194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3383284Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3383553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3383639Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3383879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3383964Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3384244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3384351Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3384628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3384736Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3385058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3385212Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3385216Z 2025-08-26T20:25:15.3385327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3385551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3385624Z return mod(**inputs) 2025-08-26T20:25:15.3385918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3385992Z outputs = self.model( 2025-08-26T20:25:15.3386276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3386402Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3386687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3386776Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3387016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3387100Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3387376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3387484Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3387760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3387867Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3388201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3388349Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3388353Z 2025-08-26T20:25:15.3388466Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3388690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3388762Z return mod(**inputs) 2025-08-26T20:25:15.3389043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3389144Z outputs = self.model( 2025-08-26T20:25:15.3389427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3389513Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3389801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3389889Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3390136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3390229Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3390500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3390607Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3390886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3390978Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3390982Z 2025-08-26T20:25:15.3391099Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3391318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3391392Z return mod(**inputs) 2025-08-26T20:25:15.3391685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3391764Z outputs = self.model( 2025-08-26T20:25:15.3392053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3392132Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3392416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3392503Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3392742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3392835Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3393141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3393271Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3393564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3393727Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3393731Z 2025-08-26T20:25:15.3393851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3394073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3394154Z return mod(**inputs) 2025-08-26T20:25:15.3394442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3394517Z outputs = self.model( 2025-08-26T20:25:15.3394802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3394884Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3395178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3395256Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3395504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3395591Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3395862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3396008Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3396495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3396600Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3396604Z 2025-08-26T20:25:15.3396723Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3396941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3397027Z return mod(**inputs) 2025-08-26T20:25:15.3397303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3397387Z outputs = self.model( 2025-08-26T20:25:15.3397660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3397741Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3398022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3398105Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3398360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3398447Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3398731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3398852Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3399124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3399274Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3399282Z 2025-08-26T20:25:15.3399377Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3399474Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3399559Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3399641Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3399764Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3400077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3400159Z return mod(**inputs) 2025-08-26T20:25:15.3400436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3400523Z outputs = self.model( 2025-08-26T20:25:15.3400794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3400872Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3401145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3401229Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3401464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3401558Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3401824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3401977Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3402240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3402356Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3402668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3402808Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3402842Z 2025-08-26T20:25:15.3402964Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3403175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3403253Z return mod(**inputs) 2025-08-26T20:25:15.3403521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3403595Z outputs = self.model( 2025-08-26T20:25:15.3403869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3403946Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3404218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3404294Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3404535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3404622Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3404881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3405003Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3405266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3405377Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3405687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3405800Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3405804Z 2025-08-26T20:25:15.3405921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3406134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3406210Z return mod(**inputs) 2025-08-26T20:25:15.3406475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3406551Z outputs = self.model( 2025-08-26T20:25:15.3406853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3406934Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3407207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3407284Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3407525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3407611Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3407872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3407992Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3408253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3408346Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3408369Z 2025-08-26T20:25:15.3408481Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3408693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3408769Z return mod(**inputs) 2025-08-26T20:25:15.3409034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3409113Z outputs = self.model( 2025-08-26T20:25:15.3409402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3409488Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3409751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3409834Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3410081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3410169Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3410441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3410572Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3410576Z 2025-08-26T20:25:15.3410689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3410912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3410987Z return mod(**inputs) 2025-08-26T20:25:15.3411263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3411339Z outputs = self.model( 2025-08-26T20:25:15.3411605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3411694Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3411963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3412050Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3412288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3412383Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3412652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3412781Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3413021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3413134Z return self.act(input) 2025-08-26T20:25:15.3413139Z 2025-08-26T20:25:15.3413256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3413472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3413542Z return mod(**inputs) 2025-08-26T20:25:15.3413817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3413892Z outputs = self.model( 2025-08-26T20:25:15.3414163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3414243Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3414506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3414591Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3414826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3414945Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3415208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3415302Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3415306Z 2025-08-26T20:25:15.3415416Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3415629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3415730Z return mod(**inputs) 2025-08-26T20:25:15.3415997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3416075Z outputs = self.model( 2025-08-26T20:25:15.3416342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3416420Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3416696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3416773Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3417016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3417099Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3417368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3417477Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3417743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3417911Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3417918Z 2025-08-26T20:25:15.3418031Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3418251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3418322Z return mod(**inputs) 2025-08-26T20:25:15.3418590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3418671Z outputs = self.model( 2025-08-26T20:25:15.3418934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3419021Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3419288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3419363Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3419637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3419723Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3419993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3420099Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3420370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3420453Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3420457Z 2025-08-26T20:25:15.3420565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3420784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3420854Z return mod(**inputs) 2025-08-26T20:25:15.3421129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3421200Z outputs = self.model( 2025-08-26T20:25:15.3421467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3421573Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3421832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3421916Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3422150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3422269Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3422528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3422633Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3422901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3422995Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3422999Z 2025-08-26T20:25:15.3423091Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3423174Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3423255Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3423342Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3423451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3423667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3423738Z return mod(**inputs) 2025-08-26T20:25:15.3424005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3424086Z outputs = self.model( 2025-08-26T20:25:15.3424351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3424438Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3424700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3424778Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3425022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3425105Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3425372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3425478Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3425747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3425888Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3426210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3426366Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3426370Z 2025-08-26T20:25:15.3426482Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3426708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3426782Z return mod(**inputs) 2025-08-26T20:25:15.3427070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3427167Z outputs = self.model( 2025-08-26T20:25:15.3427444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3427530Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3427819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3427919Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3428163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3428246Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3428520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3428624Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3428921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3429022Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3429333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3429460Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3429465Z 2025-08-26T20:25:15.3429573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3429794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3429864Z return mod(**inputs) 2025-08-26T20:25:15.3430139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3430218Z outputs = self.model( 2025-08-26T20:25:15.3430481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3430566Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3430841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3430926Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3431161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3431246Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3431515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3431619Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3431893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3431982Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3431986Z 2025-08-26T20:25:15.3432095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3432313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3432382Z return mod(**inputs) 2025-08-26T20:25:15.3432695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3432772Z outputs = self.model( 2025-08-26T20:25:15.3433055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3433134Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3433411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3433496Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3433737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3433827Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3434104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3434223Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3434511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3434734Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3434738Z 2025-08-26T20:25:15.3434857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3435072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3435144Z return mod(**inputs) 2025-08-26T20:25:15.3435458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3435531Z outputs = self.model( 2025-08-26T20:25:15.3435813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3435894Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3436191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3436271Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3436512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3436605Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3436875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3437000Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3437271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3437357Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3437361Z 2025-08-26T20:25:15.3437483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3437700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3437780Z return mod(**inputs) 2025-08-26T20:25:15.3438054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3438133Z outputs = self.model( 2025-08-26T20:25:15.3438406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3438487Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3438766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3438844Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3439094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3439212Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3439570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3439705Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3439975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3440080Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3440085Z 2025-08-26T20:25:15.3440172Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3440260Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3440355Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3440439Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3440562Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3440781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3440856Z return mod(**inputs) 2025-08-26T20:25:15.3441140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3441248Z outputs = self.model( 2025-08-26T20:25:15.3441527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3441609Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3441889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3441988Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3442231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3442324Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3442597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3442721Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3442994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3443103Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3443428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3443571Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3443577Z 2025-08-26T20:25:15.3443696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3443913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3443994Z return mod(**inputs) 2025-08-26T20:25:15.3444270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3444345Z outputs = self.model( 2025-08-26T20:25:15.3444624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3444705Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3444982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3445060Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3445301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3445397Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3445664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3445787Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3446096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3446208Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3446533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3446650Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3446654Z 2025-08-26T20:25:15.3446775Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3446994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3447074Z return mod(**inputs) 2025-08-26T20:25:15.3447344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3447419Z outputs = self.model( 2025-08-26T20:25:15.3447703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3447815Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3448096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3448174Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3448418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3448513Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3448805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3448931Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3449203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3449296Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3449307Z 2025-08-26T20:25:15.3449422Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3449643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3449723Z return mod(**inputs) 2025-08-26T20:25:15.3449997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3450076Z outputs = self.model( 2025-08-26T20:25:15.3450350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3450430Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3450718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3450795Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3451040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3451125Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3451386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3451519Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3451523Z 2025-08-26T20:25:15.3451632Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3451848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3451920Z return mod(**inputs) 2025-08-26T20:25:15.3452195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3452267Z outputs = self.model( 2025-08-26T20:25:15.3452567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3452658Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3452927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3453011Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3453248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3453332Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3453605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3453735Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3453973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3454052Z return self.act(input) 2025-08-26T20:25:15.3454056Z 2025-08-26T20:25:15.3454171Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3454409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3454480Z return mod(**inputs) 2025-08-26T20:25:15.3454748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3454820Z outputs = self.model( 2025-08-26T20:25:15.3455092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3455192Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3455455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3455540Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3455778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3455870Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3456135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3456222Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3456226Z 2025-08-26T20:25:15.3456342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3456552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3456632Z return mod(**inputs) 2025-08-26T20:25:15.3456900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3456973Z outputs = self.model( 2025-08-26T20:25:15.3457245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3457327Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3457600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3457678Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3457918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3458012Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3458259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3458371Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3458621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3458783Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3458787Z 2025-08-26T20:25:15.3458930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3459145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3459225Z return mod(**inputs) 2025-08-26T20:25:15.3459493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3459570Z outputs = self.model( 2025-08-26T20:25:15.3459820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3459899Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3460150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3460228Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3460457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3460539Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3460802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3460941Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3461205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3461307Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3461310Z 2025-08-26T20:25:15.3461413Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3461639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3461704Z return mod(**inputs) 2025-08-26T20:25:15.3461955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3462034Z outputs = self.model( 2025-08-26T20:25:15.3462284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3462367Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3462619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3462698Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3462922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3463001Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3463262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3463366Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3463641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3463733Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3463738Z 2025-08-26T20:25:15.3463824Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3463915Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3463998Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3464085Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3464194Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3464406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3464484Z return mod(**inputs) 2025-08-26T20:25:15.3464753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3464832Z outputs = self.model( 2025-08-26T20:25:15.3465140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3465216Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3465475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3465548Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3465780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3465863Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3466136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3466242Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3466513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3466619Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3466917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3467079Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3467082Z 2025-08-26T20:25:15.3467185Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3467386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3467459Z return mod(**inputs) 2025-08-26T20:25:15.3467718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3467826Z outputs = self.model( 2025-08-26T20:25:15.3468091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3468174Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3468454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3468532Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3468774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3468857Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3469132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3469237Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3469508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3469620Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3469938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3470062Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3470065Z 2025-08-26T20:25:15.3470176Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3470393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3470464Z return mod(**inputs) 2025-08-26T20:25:15.3470772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3470847Z outputs = self.model( 2025-08-26T20:25:15.3471097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3471178Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3471430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3471502Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3471779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3471862Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3472128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3472232Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3472494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3472600Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3472606Z 2025-08-26T20:25:15.3472710Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3472918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3472985Z return mod(**inputs) 2025-08-26T20:25:15.3473246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3473337Z outputs = self.model( 2025-08-26T20:25:15.3473615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3473703Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3473980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3474063Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3474300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3474400Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3474671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3474788Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3475054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3475216Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3475220Z 2025-08-26T20:25:15.3475336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3475549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3475618Z return mod(**inputs) 2025-08-26T20:25:15.3475890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3475964Z outputs = self.model( 2025-08-26T20:25:15.3476245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3476322Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3476599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3476685Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3476921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3477012Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3477282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3477397Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3477666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3477752Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3477755Z 2025-08-26T20:25:15.3477871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3478117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3478199Z return mod(**inputs) 2025-08-26T20:25:15.3478478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3478548Z outputs = self.model( 2025-08-26T20:25:15.3478820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3478897Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3479167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3479317Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3479562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3479653Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3479919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3480061Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3480320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3480419Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3480423Z 2025-08-26T20:25:15.3480508Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3480593Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3480704Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3480784Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3480901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3481115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3481186Z return mod(**inputs) 2025-08-26T20:25:15.3481466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3481540Z outputs = self.model( 2025-08-26T20:25:15.3481810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3481888Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3482152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3482237Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3482475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3482564Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3482828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3482944Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3483215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3483320Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3483640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3483781Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3483784Z 2025-08-26T20:25:15.3483900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3484113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3484182Z return mod(**inputs) 2025-08-26T20:25:15.3484453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3484559Z outputs = self.model( 2025-08-26T20:25:15.3484831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3484911Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3485173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3485257Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3485493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3485586Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3485847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3485958Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3486229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3486354Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3486672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3486785Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3486789Z 2025-08-26T20:25:15.3486905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3487117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3487209Z return mod(**inputs) 2025-08-26T20:25:15.3487484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3487557Z outputs = self.model( 2025-08-26T20:25:15.3487832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3487919Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3488170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3488250Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3488473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3488563Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3488824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3488948Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3489212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3489299Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3489305Z 2025-08-26T20:25:15.3489423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3489637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3489715Z return mod(**inputs) 2025-08-26T20:25:15.3489979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3490051Z outputs = self.model( 2025-08-26T20:25:15.3490320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3490399Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3490673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3490748Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3491023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3491116Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3491380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3491514Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3491518Z 2025-08-26T20:25:15.3491637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3491842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3491909Z return mod(**inputs) 2025-08-26T20:25:15.3492157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3492232Z outputs = self.model( 2025-08-26T20:25:15.3492481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3492565Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3492811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3492906Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3493151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3493236Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3493509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3493656Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3493891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3493966Z return self.act(input) 2025-08-26T20:25:15.3493970Z 2025-08-26T20:25:15.3494081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3494300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3494370Z return mod(**inputs) 2025-08-26T20:25:15.3494643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3494714Z outputs = self.model( 2025-08-26T20:25:15.3494976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3495063Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3495327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3495412Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3495646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3495733Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3495999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3496087Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3496091Z 2025-08-26T20:25:15.3496486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3496692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3496766Z return mod(**inputs) 2025-08-26T20:25:15.3497011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3497082Z outputs = self.model( 2025-08-26T20:25:15.3497342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3497422Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3497783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3497865Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3498111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3498205Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3498475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3498594Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3498878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3499039Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3499051Z 2025-08-26T20:25:15.3499162Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3499376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3499486Z return mod(**inputs) 2025-08-26T20:25:15.3499750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3499830Z outputs = self.model( 2025-08-26T20:25:15.3500094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3500172Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3500475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3500550Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3500802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3500884Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3501133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3501242Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3501490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3501580Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3501584Z 2025-08-26T20:25:15.3501685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3501891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3501958Z return mod(**inputs) 2025-08-26T20:25:15.3502206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3502282Z outputs = self.model( 2025-08-26T20:25:15.3502537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3502624Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3502891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3502969Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3503213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3503296Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3503568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3503674Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3503937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3504071Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3504075Z 2025-08-26T20:25:15.3504164Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3504259Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3504342Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3504431Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3504541Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3504756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3504834Z return mod(**inputs) 2025-08-26T20:25:15.3505107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3505186Z outputs = self.model( 2025-08-26T20:25:15.3505451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3505532Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3505804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3505898Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3506143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3506227Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3506489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3506620Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3506881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3506990Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3507304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3507454Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3507457Z 2025-08-26T20:25:15.3507565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3507775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3507852Z return mod(**inputs) 2025-08-26T20:25:15.3508121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3508202Z outputs = self.model( 2025-08-26T20:25:15.3508466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3508543Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3508819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3508896Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3509139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3509221Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3509483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3509594Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3509856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3509967Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3510282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3510444Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3510449Z 2025-08-26T20:25:15.3510564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3510785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3510866Z return mod(**inputs) 2025-08-26T20:25:15.3511151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3511231Z outputs = self.model( 2025-08-26T20:25:15.3511511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3511591Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3511874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3511951Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3512196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3512280Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3512571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-26T20:25:15.3512676Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:25:15.3512948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3513045Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3513066Z 2025-08-26T20:25:15.3513177Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3513395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3513463Z return mod(**inputs) 2025-08-26T20:25:15.3513737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3513816Z outputs = self.model( 2025-08-26T20:25:15.3514103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3514187Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3514461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3514538Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3514790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3514878Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3515160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3515280Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3515571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-26T20:25:15.3515738Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:25:15.3515742Z 2025-08-26T20:25:15.3515856Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3516080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3516151Z return mod(**inputs) 2025-08-26T20:25:15.3516446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3516522Z outputs = self.model( 2025-08-26T20:25:15.3516804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3516890Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3517208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3517295Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3517543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3517635Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3517918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3518037Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3518323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-26T20:25:15.3518412Z key_states = self.k_proj(current_states) 2025-08-26T20:25:15.3518416Z 2025-08-26T20:25:15.3518534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3518754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3518829Z return mod(**inputs) 2025-08-26T20:25:15.3519145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3519219Z outputs = self.model( 2025-08-26T20:25:15.3519575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3519658Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3519939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3520058Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3520310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3520407Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3520678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3520813Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3521083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-26T20:25:15.3521176Z value_states = self.v_proj(current_states) 2025-08-26T20:25:15.3521180Z 2025-08-26T20:25:15.3521275Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3521362Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3521455Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3521540Z cudagraph partition due to non gpu ops 2025-08-26T20:25:15.3521654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3521872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3521941Z return mod(**inputs) 2025-08-26T20:25:15.3522216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3522290Z outputs = self.model( 2025-08-26T20:25:15.3522557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3522644Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3522910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3522996Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3523231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3523317Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3523587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3523734Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3524007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3524114Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3524434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:25:15.3524576Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:15.3524579Z 2025-08-26T20:25:15.3524688Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3524909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3524978Z return mod(**inputs) 2025-08-26T20:25:15.3525250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3525322Z outputs = self.model( 2025-08-26T20:25:15.3525589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3525696Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3525963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3526050Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3526285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3526376Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3526657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3526770Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3527041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-26T20:25:15.3527144Z attn_output, attn_weights = attention_interface( 2025-08-26T20:25:15.3527461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:25:15.3527572Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:25:15.3527576Z 2025-08-26T20:25:15.3527683Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3527899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3527971Z return mod(**inputs) 2025-08-26T20:25:15.3528243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3528315Z outputs = self.model( 2025-08-26T20:25:15.3528585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3528665Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3528925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3529010Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3529245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3529333Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3529592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-26T20:25:15.3529706Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:25:15.3529974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-26T20:25:15.3530059Z attn_output = self.out_proj(attn_output) 2025-08-26T20:25:15.3530063Z 2025-08-26T20:25:15.3530215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3530430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3530502Z return mod(**inputs) 2025-08-26T20:25:15.3530773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3530845Z outputs = self.model( 2025-08-26T20:25:15.3531116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3531193Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3531461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3531537Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3531771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3531864Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3532127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3532281Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3532285Z 2025-08-26T20:25:15.3532396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3532608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3532685Z return mod(**inputs) 2025-08-26T20:25:15.3532979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3533057Z outputs = self.model( 2025-08-26T20:25:15.3533319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3533399Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3533671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3533750Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3533993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3534076Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3534344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-26T20:25:15.3534471Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:25:15.3534697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:15.3534777Z return self.act(input) 2025-08-26T20:25:15.3534781Z 2025-08-26T20:25:15.3534888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3535109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3535181Z return mod(**inputs) 2025-08-26T20:25:15.3535446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-26T20:25:15.3535526Z outputs = self.model( 2025-08-26T20:25:15.3535787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-26T20:25:15.3535872Z decoder_outputs = self.decoder( 2025-08-26T20:25:15.3536135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-26T20:25:15.3536212Z layer_outputs = decoder_layer( 2025-08-26T20:25:15.3536450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:15.3536534Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:15.3536869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-26T20:25:15.3536961Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:25:15.3536965Z 2025-08-26T20:25:15.3537081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3537292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3537361Z return mod(**inputs) 2025-08-26T20:25:15.3537634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1490, in forward 2025-08-26T20:25:15.3537720Z lm_logits = self.lm_head(outputs[0]) 2025-08-26T20:25:15.3537724Z 2025-08-26T20:25:15.3537839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:15.3538050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:15.3538122Z return mod(**inputs) 2025-08-26T20:25:15.3538393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1497, in forward 2025-08-26T20:25:15.3538602Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:25:15.3538606Z 2025-08-26T20:25:28.4587185Z Compilation time (from dynamo_timed): 28.01163485 2025-08-26T20:25:28.4693102Z pass 2025-08-26T20:25:28.4693535Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:25:28.4694425Z TIMING: _recursive_pre_grad_passes:0.01487 _recursive_joint_graph_passes:1.18885 _recursive_post_grad_passes:0.17641 async_compile.wait:0.89817 code_gen:11.26682 inductor_compile:14.50928 backend_compile:22.02974 gc:0.00086 entire_frame_compile:28.01163 total_wall_time:28.01163 2025-08-26T20:25:28.4698329Z STATS: call_* op count: 980 | FakeTensorMode.__torch_dispatch__:33505 | FakeTensor.__torch_dispatch__:11174 | ProxyTorchDispatchMode.__torch_dispatch__:12370 2025-08-26T20:25:28.4698897Z Dynamo produced 1 graphs covering 980 ops with 0 graph breaks (0 unique) 2025-08-26T20:25:34.4669240Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:25:34.4670250Z from pkg_resources import resource_filename 2025-08-26T20:25:35.0766090Z 2025-08-26T20:25:36.5252664Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:25:36.5257392Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:25:36.5267699Z cpu eval BertForMaskedLM 2025-08-26T20:25:37.0250229Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:25:37.2706385Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:25:37.5159389Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:25:45.3464108Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3464591Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3464853Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3465091Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3465341Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3465700Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3466771Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3467052Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3467270Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3467484Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3467686Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3467907Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3468558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3468989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3469377Z return mod(**inputs) 2025-08-26T20:25:45.3469814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3470286Z outputs = self.bert( 2025-08-26T20:25:45.3470690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3471131Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3471593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3472014Z layer_outputs = layer_module( 2025-08-26T20:25:45.3472433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3472870Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3473313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3473830Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3474390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3474803Z return func(*args, **kwargs) 2025-08-26T20:25:45.3475218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3475716Z self_outputs = self.self( 2025-08-26T20:25:45.3476175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3476600Z return func(*args, **kwargs) 2025-08-26T20:25:45.3477029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3477639Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3477946Z 2025-08-26T20:25:45.3478079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3478484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3478856Z return mod(**inputs) 2025-08-26T20:25:45.3479557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3480011Z outputs = self.bert( 2025-08-26T20:25:45.3480414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3480852Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3481279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3481708Z layer_outputs = layer_module( 2025-08-26T20:25:45.3482088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3482491Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3482938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3483381Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3483804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3484212Z return func(*args, **kwargs) 2025-08-26T20:25:45.3484663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3485081Z self_outputs = self.self( 2025-08-26T20:25:45.3485523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3485934Z return func(*args, **kwargs) 2025-08-26T20:25:45.3486322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3486773Z self.key(current_states) 2025-08-26T20:25:45.3486904Z 2025-08-26T20:25:45.3487020Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3487412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3487777Z return mod(**inputs) 2025-08-26T20:25:45.3488187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3488604Z outputs = self.bert( 2025-08-26T20:25:45.3488999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3489444Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3489864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3490303Z layer_outputs = layer_module( 2025-08-26T20:25:45.3490677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3491063Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3491472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3491906Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3492325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3492728Z return func(*args, **kwargs) 2025-08-26T20:25:45.3493137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3493548Z self_outputs = self.self( 2025-08-26T20:25:45.3493950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3494349Z return func(*args, **kwargs) 2025-08-26T20:25:45.3494744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3495149Z self.value(current_states) 2025-08-26T20:25:45.3495276Z 2025-08-26T20:25:45.3495365Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3495634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3496033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3496666Z return mod(**inputs) 2025-08-26T20:25:45.3497063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3497467Z outputs = self.bert( 2025-08-26T20:25:45.3497862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3498291Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3498718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3499127Z layer_outputs = layer_module( 2025-08-26T20:25:45.3499516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3499921Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3500353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3500787Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3501264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3501679Z return func(*args, **kwargs) 2025-08-26T20:25:45.3502085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3502500Z self_outputs = self.self( 2025-08-26T20:25:45.3502890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3503309Z return func(*args, **kwargs) 2025-08-26T20:25:45.3503718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3504213Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3504421Z 2025-08-26T20:25:45.3504547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3504947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3505309Z return mod(**inputs) 2025-08-26T20:25:45.3505740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3506158Z outputs = self.bert( 2025-08-26T20:25:45.3506553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3506971Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3507388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3507836Z layer_outputs = layer_module( 2025-08-26T20:25:45.3508222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3508617Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3509047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3509481Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3509904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3510312Z return func(*args, **kwargs) 2025-08-26T20:25:45.3510708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3511190Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3511670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3512103Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3512258Z 2025-08-26T20:25:45.3512383Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3512776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3513140Z return mod(**inputs) 2025-08-26T20:25:45.3513535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3513949Z outputs = self.bert( 2025-08-26T20:25:45.3514334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3514763Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3515176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3515601Z layer_outputs = layer_module( 2025-08-26T20:25:45.3515984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3516376Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3516863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3517294Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3517733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3518180Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3518639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3519159Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3519805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3520246Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3520402Z 2025-08-26T20:25:45.3520523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3520937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3521386Z return mod(**inputs) 2025-08-26T20:25:45.3521773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3522179Z outputs = self.bert( 2025-08-26T20:25:45.3522552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3522979Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3523385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3523819Z layer_outputs = layer_module( 2025-08-26T20:25:45.3524196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3524588Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3525008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3525448Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3525889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3526318Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3526765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3527267Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3527733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3528191Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3528604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3528990Z return self.act(input) 2025-08-26T20:25:45.3529124Z 2025-08-26T20:25:45.3529240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3529636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3530002Z return mod(**inputs) 2025-08-26T20:25:45.3530414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3530827Z outputs = self.bert( 2025-08-26T20:25:45.3531228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3531644Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3532093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3532556Z layer_outputs = layer_module( 2025-08-26T20:25:45.3532930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3533326Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3533743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3534166Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3534599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3535023Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3535462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3535960Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3536428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3536868Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3537014Z 2025-08-26T20:25:45.3537134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3537521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3537865Z return mod(**inputs) 2025-08-26T20:25:45.3538253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3538678Z outputs = self.bert( 2025-08-26T20:25:45.3539061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3539474Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3539875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3540285Z layer_outputs = layer_module( 2025-08-26T20:25:45.3540657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3541048Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3541455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3541885Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3542275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3542658Z return func(*args, **kwargs) 2025-08-26T20:25:45.3543030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3543408Z self_outputs = self.self( 2025-08-26T20:25:45.3543791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3544188Z return func(*args, **kwargs) 2025-08-26T20:25:45.3544584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3545146Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3545430Z 2025-08-26T20:25:45.3545542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3545933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3546292Z return mod(**inputs) 2025-08-26T20:25:45.3546659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3547037Z outputs = self.bert( 2025-08-26T20:25:45.3547439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3547850Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3548255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3548663Z layer_outputs = layer_module( 2025-08-26T20:25:45.3549026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3549414Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3549827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3550250Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3550651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3551047Z return func(*args, **kwargs) 2025-08-26T20:25:45.3551445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3551872Z self_outputs = self.self( 2025-08-26T20:25:45.3552257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3552648Z return func(*args, **kwargs) 2025-08-26T20:25:45.3553040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3553444Z self.key(current_states) 2025-08-26T20:25:45.3553567Z 2025-08-26T20:25:45.3553706Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3554094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3554436Z return mod(**inputs) 2025-08-26T20:25:45.3554827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3555232Z outputs = self.bert( 2025-08-26T20:25:45.3555613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3556021Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3556425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3556829Z layer_outputs = layer_module( 2025-08-26T20:25:45.3557200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3557587Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3557994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3558427Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3558836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3559305Z return func(*args, **kwargs) 2025-08-26T20:25:45.3559713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3560129Z self_outputs = self.self( 2025-08-26T20:25:45.3560527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3560954Z return func(*args, **kwargs) 2025-08-26T20:25:45.3561347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3561751Z self.value(current_states) 2025-08-26T20:25:45.3561886Z 2025-08-26T20:25:45.3561975Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3562238Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3562665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3563010Z return mod(**inputs) 2025-08-26T20:25:45.3563397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3563798Z outputs = self.bert( 2025-08-26T20:25:45.3564183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3564595Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3564992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3565401Z layer_outputs = layer_module( 2025-08-26T20:25:45.3565772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3566156Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3566571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3567000Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3567411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3567805Z return func(*args, **kwargs) 2025-08-26T20:25:45.3568199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3568603Z self_outputs = self.self( 2025-08-26T20:25:45.3569022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3569426Z return func(*args, **kwargs) 2025-08-26T20:25:45.3569826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3570318Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3570520Z 2025-08-26T20:25:45.3570636Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3571031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3571384Z return mod(**inputs) 2025-08-26T20:25:45.3571794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3572198Z outputs = self.bert( 2025-08-26T20:25:45.3572609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3573036Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3573442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3573855Z layer_outputs = layer_module( 2025-08-26T20:25:45.3574227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3574622Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3575041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3575459Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3575883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3576286Z return func(*args, **kwargs) 2025-08-26T20:25:45.3576690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3577174Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3577650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3578128Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3578286Z 2025-08-26T20:25:45.3578400Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3578786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3579133Z return mod(**inputs) 2025-08-26T20:25:45.3579532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3579935Z outputs = self.bert( 2025-08-26T20:25:45.3580314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3580728Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3581124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3581529Z layer_outputs = layer_module( 2025-08-26T20:25:45.3581898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3582311Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3582729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3583155Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3583585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3584027Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3584482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3584969Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3585429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3585842Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3586005Z 2025-08-26T20:25:45.3586116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3586503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3586854Z return mod(**inputs) 2025-08-26T20:25:45.3587237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3587632Z outputs = self.bert( 2025-08-26T20:25:45.3588016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3588426Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3588824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3589222Z layer_outputs = layer_module( 2025-08-26T20:25:45.3589595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3589981Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3590387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3590808Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3591234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3591662Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3592115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3592617Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3593114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3593574Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3593996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3594376Z return self.act(input) 2025-08-26T20:25:45.3594500Z 2025-08-26T20:25:45.3594624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3595020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3595383Z return mod(**inputs) 2025-08-26T20:25:45.3595777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3596334Z outputs = self.bert( 2025-08-26T20:25:45.3596745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3597168Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3597588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3598058Z layer_outputs = layer_module( 2025-08-26T20:25:45.3598437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3598948Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3599453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3599929Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3600375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3600816Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3601267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3601793Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3602281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3602718Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3602871Z 2025-08-26T20:25:45.3602995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3603390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3603754Z return mod(**inputs) 2025-08-26T20:25:45.3604152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3604569Z outputs = self.bert( 2025-08-26T20:25:45.3604958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3605384Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3605800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3606232Z layer_outputs = layer_module( 2025-08-26T20:25:45.3606616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3607004Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3607427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3607864Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3608292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3608705Z return func(*args, **kwargs) 2025-08-26T20:25:45.3609156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3609577Z self_outputs = self.self( 2025-08-26T20:25:45.3609973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3610392Z return func(*args, **kwargs) 2025-08-26T20:25:45.3610785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3611355Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3611649Z 2025-08-26T20:25:45.3611775Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3612158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3612508Z return mod(**inputs) 2025-08-26T20:25:45.3612899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3613342Z outputs = self.bert( 2025-08-26T20:25:45.3613727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3614137Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3614533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3614931Z layer_outputs = layer_module( 2025-08-26T20:25:45.3615299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3615703Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3616113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3616526Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3616939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3617339Z return func(*args, **kwargs) 2025-08-26T20:25:45.3617732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3618139Z self_outputs = self.self( 2025-08-26T20:25:45.3618516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3618914Z return func(*args, **kwargs) 2025-08-26T20:25:45.3619311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3619720Z self.key(current_states) 2025-08-26T20:25:45.3619842Z 2025-08-26T20:25:45.3619963Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3620349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3620702Z return mod(**inputs) 2025-08-26T20:25:45.3621088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3621497Z outputs = self.bert( 2025-08-26T20:25:45.3621875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3622289Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3622702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3623135Z layer_outputs = layer_module( 2025-08-26T20:25:45.3623507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3623896Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3624377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3624801Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3625215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3625612Z return func(*args, **kwargs) 2025-08-26T20:25:45.3626013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3626441Z self_outputs = self.self( 2025-08-26T20:25:45.3626840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3627246Z return func(*args, **kwargs) 2025-08-26T20:25:45.3627639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3628065Z self.value(current_states) 2025-08-26T20:25:45.3628209Z 2025-08-26T20:25:45.3628301Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3628584Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3628963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3629310Z return mod(**inputs) 2025-08-26T20:25:45.3629692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3630105Z outputs = self.bert( 2025-08-26T20:25:45.3630483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3630916Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3631322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3631743Z layer_outputs = layer_module( 2025-08-26T20:25:45.3632121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3632508Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3632952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3633388Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3633818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3634234Z return func(*args, **kwargs) 2025-08-26T20:25:45.3634644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3635070Z self_outputs = self.self( 2025-08-26T20:25:45.3635468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3635893Z return func(*args, **kwargs) 2025-08-26T20:25:45.3636310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3636790Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3637008Z 2025-08-26T20:25:45.3637124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3637524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3637886Z return mod(**inputs) 2025-08-26T20:25:45.3638295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3638727Z outputs = self.bert( 2025-08-26T20:25:45.3639133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3639654Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3640120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3640561Z layer_outputs = layer_module( 2025-08-26T20:25:45.3640961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3641349Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3641764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3642197Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3642603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3643007Z return func(*args, **kwargs) 2025-08-26T20:25:45.3643403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3643883Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3644360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3644771Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3644929Z 2025-08-26T20:25:45.3645042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3645434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3645786Z return mod(**inputs) 2025-08-26T20:25:45.3646184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3646594Z outputs = self.bert( 2025-08-26T20:25:45.3646979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3647396Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3647795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3648207Z layer_outputs = layer_module( 2025-08-26T20:25:45.3648581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3648967Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3649386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3649812Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3650249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3650678Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3651119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3651613Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3652068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3652487Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3652639Z 2025-08-26T20:25:45.3652751Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3653143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3653492Z return mod(**inputs) 2025-08-26T20:25:45.3653873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3654281Z outputs = self.bert( 2025-08-26T20:25:45.3654665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3655119Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3655519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3655931Z layer_outputs = layer_module( 2025-08-26T20:25:45.3656304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3656689Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3657101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3657518Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3657949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3658367Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3658811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3659294Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3659765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3660214Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3660620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3660987Z return self.act(input) 2025-08-26T20:25:45.3661126Z 2025-08-26T20:25:45.3661242Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3661648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3662017Z return mod(**inputs) 2025-08-26T20:25:45.3662410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3662831Z outputs = self.bert( 2025-08-26T20:25:45.3663230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3663648Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3664056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3664469Z layer_outputs = layer_module( 2025-08-26T20:25:45.3664836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3665230Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3665644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3666083Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3666520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3666943Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3667386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3667894Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3668365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3668802Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3668952Z 2025-08-26T20:25:45.3669064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3669453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3669799Z return mod(**inputs) 2025-08-26T20:25:45.3670218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3670620Z outputs = self.bert( 2025-08-26T20:25:45.3671005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3671416Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3671828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3672251Z layer_outputs = layer_module( 2025-08-26T20:25:45.3672618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3673022Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3673456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3673890Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3674315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3674745Z return func(*args, **kwargs) 2025-08-26T20:25:45.3675176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3675595Z self_outputs = self.self( 2025-08-26T20:25:45.3676002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3676407Z return func(*args, **kwargs) 2025-08-26T20:25:45.3676844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3677417Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3677710Z 2025-08-26T20:25:45.3677837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3678240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3678603Z return mod(**inputs) 2025-08-26T20:25:45.3679003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3679525Z outputs = self.bert( 2025-08-26T20:25:45.3679927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3680358Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3680786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3681211Z layer_outputs = layer_module( 2025-08-26T20:25:45.3681601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3682020Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3682455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3682903Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3683341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3683766Z return func(*args, **kwargs) 2025-08-26T20:25:45.3684185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3684604Z self_outputs = self.self( 2025-08-26T20:25:45.3685006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3685427Z return func(*args, **kwargs) 2025-08-26T20:25:45.3685871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3686287Z self.key(current_states) 2025-08-26T20:25:45.3686425Z 2025-08-26T20:25:45.3686543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3686945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3687312Z return mod(**inputs) 2025-08-26T20:25:45.3687698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3688097Z outputs = self.bert( 2025-08-26T20:25:45.3688481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3688894Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3689301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3689717Z layer_outputs = layer_module( 2025-08-26T20:25:45.3690097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3690514Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3690922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3691348Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3691748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3692147Z return func(*args, **kwargs) 2025-08-26T20:25:45.3692554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3693015Z self_outputs = self.self( 2025-08-26T20:25:45.3693400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3693804Z return func(*args, **kwargs) 2025-08-26T20:25:45.3694201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3694612Z self.value(current_states) 2025-08-26T20:25:45.3694738Z 2025-08-26T20:25:45.3694833Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3695087Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3695474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3695816Z return mod(**inputs) 2025-08-26T20:25:45.3696319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3696740Z outputs = self.bert( 2025-08-26T20:25:45.3697119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3697537Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3697945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3698358Z layer_outputs = layer_module( 2025-08-26T20:25:45.3698722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3699116Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3699531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3699957Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3700369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3700757Z return func(*args, **kwargs) 2025-08-26T20:25:45.3701227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3701638Z self_outputs = self.self( 2025-08-26T20:25:45.3702028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3702427Z return func(*args, **kwargs) 2025-08-26T20:25:45.3702812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3703292Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3703501Z 2025-08-26T20:25:45.3703616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3704011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3704358Z return mod(**inputs) 2025-08-26T20:25:45.3704755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3705171Z outputs = self.bert( 2025-08-26T20:25:45.3705554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3705994Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3706401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3706842Z layer_outputs = layer_module( 2025-08-26T20:25:45.3707226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3707629Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3708067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3708492Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3708913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3709320Z return func(*args, **kwargs) 2025-08-26T20:25:45.3709724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3710215Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3710688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3711130Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3711285Z 2025-08-26T20:25:45.3711407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3711807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3712160Z return mod(**inputs) 2025-08-26T20:25:45.3712556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3712973Z outputs = self.bert( 2025-08-26T20:25:45.3713366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3713787Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3714206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3714638Z layer_outputs = layer_module( 2025-08-26T20:25:45.3715027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3715429Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3715842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3716279Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3716774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3717206Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3717650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3718136Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3718597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3719018Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3719165Z 2025-08-26T20:25:45.3719348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3719743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3720092Z return mod(**inputs) 2025-08-26T20:25:45.3720479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3720886Z outputs = self.bert( 2025-08-26T20:25:45.3721267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3721695Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3722102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3722511Z layer_outputs = layer_module( 2025-08-26T20:25:45.3722886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3723289Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3723690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3724114Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3724549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3724971Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3725403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3725889Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3726346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3726797Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3727205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3727565Z return self.act(input) 2025-08-26T20:25:45.3727693Z 2025-08-26T20:25:45.3727806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3728196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3728548Z return mod(**inputs) 2025-08-26T20:25:45.3728931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3729328Z outputs = self.bert( 2025-08-26T20:25:45.3729714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3730126Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3730531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3730932Z layer_outputs = layer_module( 2025-08-26T20:25:45.3731306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3731691Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3732144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3732573Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3733003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3733428Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3733877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3734355Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3734802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3735202Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3735359Z 2025-08-26T20:25:45.3735473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3735860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3736244Z return mod(**inputs) 2025-08-26T20:25:45.3736644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3737052Z outputs = self.bert( 2025-08-26T20:25:45.3737428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3737824Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3738206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3738610Z layer_outputs = layer_module( 2025-08-26T20:25:45.3738972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3739357Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3739772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3740164Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3740558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3740935Z return func(*args, **kwargs) 2025-08-26T20:25:45.3741313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3741698Z self_outputs = self.self( 2025-08-26T20:25:45.3742060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3742439Z return func(*args, **kwargs) 2025-08-26T20:25:45.3742826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3743402Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3743669Z 2025-08-26T20:25:45.3743781Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3744143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3744477Z return mod(**inputs) 2025-08-26T20:25:45.3744858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3745260Z outputs = self.bert( 2025-08-26T20:25:45.3745656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3746075Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3746480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3746928Z layer_outputs = layer_module( 2025-08-26T20:25:45.3747301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3747663Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3748051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3748447Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3748855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3749250Z return func(*args, **kwargs) 2025-08-26T20:25:45.3749634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3750051Z self_outputs = self.self( 2025-08-26T20:25:45.3750465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3750862Z return func(*args, **kwargs) 2025-08-26T20:25:45.3751271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3751680Z self.key(current_states) 2025-08-26T20:25:45.3751809Z 2025-08-26T20:25:45.3751919Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3752308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3752655Z return mod(**inputs) 2025-08-26T20:25:45.3753051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3753462Z outputs = self.bert( 2025-08-26T20:25:45.3753846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3754320Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3754725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3755145Z layer_outputs = layer_module( 2025-08-26T20:25:45.3755537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3755923Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3756331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3756742Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3757152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3757558Z return func(*args, **kwargs) 2025-08-26T20:25:45.3757964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3758384Z self_outputs = self.self( 2025-08-26T20:25:45.3758772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3759186Z return func(*args, **kwargs) 2025-08-26T20:25:45.3759692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3760119Z self.value(current_states) 2025-08-26T20:25:45.3760253Z 2025-08-26T20:25:45.3760346Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3762650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3763357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3763925Z return mod(**inputs) 2025-08-26T20:25:45.3764395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3765153Z outputs = self.bert( 2025-08-26T20:25:45.3765592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3766108Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3766673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3767093Z layer_outputs = layer_module( 2025-08-26T20:25:45.3767485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3767886Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3768350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3768788Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3769206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3769623Z return func(*args, **kwargs) 2025-08-26T20:25:45.3770087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3770505Z self_outputs = self.self( 2025-08-26T20:25:45.3770906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3771318Z return func(*args, **kwargs) 2025-08-26T20:25:45.3771789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3772394Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3772610Z 2025-08-26T20:25:45.3772738Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3773141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3773499Z return mod(**inputs) 2025-08-26T20:25:45.3773903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3774311Z outputs = self.bert( 2025-08-26T20:25:45.3774706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3775129Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3775555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3775996Z layer_outputs = layer_module( 2025-08-26T20:25:45.3776385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3776799Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3777218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3777638Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3778185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3778800Z return func(*args, **kwargs) 2025-08-26T20:25:45.3779373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3779879Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3780372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3780835Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3780994Z 2025-08-26T20:25:45.3781123Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3781573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3781952Z return mod(**inputs) 2025-08-26T20:25:45.3782361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3782806Z outputs = self.bert( 2025-08-26T20:25:45.3783219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3783666Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3784091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3784532Z layer_outputs = layer_module( 2025-08-26T20:25:45.3784943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3785363Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3785850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3786310Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3786782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3787235Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3787685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3788197Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3788697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3789150Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3789306Z 2025-08-26T20:25:45.3789431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3789832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3790196Z return mod(**inputs) 2025-08-26T20:25:45.3790598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3791013Z outputs = self.bert( 2025-08-26T20:25:45.3791400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3791836Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3792257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3792680Z layer_outputs = layer_module( 2025-08-26T20:25:45.3793070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3793464Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3793893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3794329Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3794780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3795210Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3795669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3796358Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3797005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3797492Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3798045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3798438Z return self.act(input) 2025-08-26T20:25:45.3798572Z 2025-08-26T20:25:45.3798695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3799102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3799725Z return mod(**inputs) 2025-08-26T20:25:45.3800125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3800557Z outputs = self.bert( 2025-08-26T20:25:45.3800955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3801389Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3801823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3802259Z layer_outputs = layer_module( 2025-08-26T20:25:45.3802652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3803137Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3803571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3804014Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3804455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3804884Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3805365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3805875Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3806342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3806766Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3806924Z 2025-08-26T20:25:45.3807039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3807470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3807813Z return mod(**inputs) 2025-08-26T20:25:45.3808197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3808605Z outputs = self.bert( 2025-08-26T20:25:45.3808990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3809412Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3809809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3810222Z layer_outputs = layer_module( 2025-08-26T20:25:45.3810601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3810996Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3811400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3811837Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3812255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3812665Z return func(*args, **kwargs) 2025-08-26T20:25:45.3813065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3813467Z self_outputs = self.self( 2025-08-26T20:25:45.3813922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3814352Z return func(*args, **kwargs) 2025-08-26T20:25:45.3814750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3815366Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3815653Z 2025-08-26T20:25:45.3815768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3816161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3816511Z return mod(**inputs) 2025-08-26T20:25:45.3816893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3817294Z outputs = self.bert( 2025-08-26T20:25:45.3817679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3818094Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3818527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3818954Z layer_outputs = layer_module( 2025-08-26T20:25:45.3819323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3819714Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3820130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3820574Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3820984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3821404Z return func(*args, **kwargs) 2025-08-26T20:25:45.3821808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3822224Z self_outputs = self.self( 2025-08-26T20:25:45.3822614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3823014Z return func(*args, **kwargs) 2025-08-26T20:25:45.3823407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3823809Z self.key(current_states) 2025-08-26T20:25:45.3823932Z 2025-08-26T20:25:45.3824050Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3824443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3824786Z return mod(**inputs) 2025-08-26T20:25:45.3825167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3825573Z outputs = self.bert( 2025-08-26T20:25:45.3825958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3826382Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3826773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3827176Z layer_outputs = layer_module( 2025-08-26T20:25:45.3827556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3827950Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3828353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3828777Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3829232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3829612Z return func(*args, **kwargs) 2025-08-26T20:25:45.3829989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3830402Z self_outputs = self.self( 2025-08-26T20:25:45.3830798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3831231Z return func(*args, **kwargs) 2025-08-26T20:25:45.3831643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3832061Z self.value(current_states) 2025-08-26T20:25:45.3832210Z 2025-08-26T20:25:45.3832301Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3832562Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3832954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3833309Z return mod(**inputs) 2025-08-26T20:25:45.3833734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3834145Z outputs = self.bert( 2025-08-26T20:25:45.3834528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3834963Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3835359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3835794Z layer_outputs = layer_module( 2025-08-26T20:25:45.3836173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3836566Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3836994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3837436Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3837870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3838295Z return func(*args, **kwargs) 2025-08-26T20:25:45.3838706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3839135Z self_outputs = self.self( 2025-08-26T20:25:45.3839680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3840130Z return func(*args, **kwargs) 2025-08-26T20:25:45.3840547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3841050Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3841256Z 2025-08-26T20:25:45.3841377Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3841751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3842090Z return mod(**inputs) 2025-08-26T20:25:45.3842456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3842849Z outputs = self.bert( 2025-08-26T20:25:45.3843209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3843627Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3844032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3844440Z layer_outputs = layer_module( 2025-08-26T20:25:45.3845969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3846383Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3846802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3847229Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3847646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3848047Z return func(*args, **kwargs) 2025-08-26T20:25:45.3848454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3848949Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3849420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3849853Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3850005Z 2025-08-26T20:25:45.3850120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3850542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3850901Z return mod(**inputs) 2025-08-26T20:25:45.3851297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3851706Z outputs = self.bert( 2025-08-26T20:25:45.3852093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3852553Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3852968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3853390Z layer_outputs = layer_module( 2025-08-26T20:25:45.3853931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3854524Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3854953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3855399Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3855830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3856253Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3856697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3857190Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3857651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3858077Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3858242Z 2025-08-26T20:25:45.3858359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3858763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3859133Z return mod(**inputs) 2025-08-26T20:25:45.3859521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3859919Z outputs = self.bert( 2025-08-26T20:25:45.3860302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3860716Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3861122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3861531Z layer_outputs = layer_module( 2025-08-26T20:25:45.3861950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3862351Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3862787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3863229Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3863677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3864104Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3864562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3865066Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3865538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3866009Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3866460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3866844Z return self.act(input) 2025-08-26T20:25:45.3866968Z 2025-08-26T20:25:45.3867092Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3867497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3867868Z return mod(**inputs) 2025-08-26T20:25:45.3868340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3868763Z outputs = self.bert( 2025-08-26T20:25:45.3869178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3869598Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3870023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3870447Z layer_outputs = layer_module( 2025-08-26T20:25:45.3870838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3871252Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3871714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3872153Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3872602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3873036Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3873484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3874004Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3874483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3874931Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3875083Z 2025-08-26T20:25:45.3875208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3875601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3875961Z return mod(**inputs) 2025-08-26T20:25:45.3876358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3876778Z outputs = self.bert( 2025-08-26T20:25:45.3877168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3877762Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3878307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3878723Z layer_outputs = layer_module( 2025-08-26T20:25:45.3879113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3879640Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3880082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3880528Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3880969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3881389Z return func(*args, **kwargs) 2025-08-26T20:25:45.3881789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3882209Z self_outputs = self.self( 2025-08-26T20:25:45.3882625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3883021Z return func(*args, **kwargs) 2025-08-26T20:25:45.3883405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3883955Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3884269Z 2025-08-26T20:25:45.3884384Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3884775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3885130Z return mod(**inputs) 2025-08-26T20:25:45.3885519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3885943Z outputs = self.bert( 2025-08-26T20:25:45.3886342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3886768Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3887182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3887595Z layer_outputs = layer_module( 2025-08-26T20:25:45.3887981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3888371Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3888782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3889220Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3889641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3889731Z return func(*args, **kwargs) 2025-08-26T20:25:45.3890009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3890096Z self_outputs = self.self( 2025-08-26T20:25:45.3890369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3890444Z return func(*args, **kwargs) 2025-08-26T20:25:45.3890717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3890798Z self.key(current_states) 2025-08-26T20:25:45.3890801Z 2025-08-26T20:25:45.3890922Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3891221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3891297Z return mod(**inputs) 2025-08-26T20:25:45.3891582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3891655Z outputs = self.bert( 2025-08-26T20:25:45.3891934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3892017Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3892287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3892379Z layer_outputs = layer_module( 2025-08-26T20:25:45.3892623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3892720Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3892998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3893103Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3893400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3893476Z return func(*args, **kwargs) 2025-08-26T20:25:45.3893746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3893821Z self_outputs = self.self( 2025-08-26T20:25:45.3894087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3894184Z return func(*args, **kwargs) 2025-08-26T20:25:45.3894456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3894546Z self.value(current_states) 2025-08-26T20:25:45.3894551Z 2025-08-26T20:25:45.3894645Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3894770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3894993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3895066Z return mod(**inputs) 2025-08-26T20:25:45.3895353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3895427Z outputs = self.bert( 2025-08-26T20:25:45.3895712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3895797Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3896071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3896290Z layer_outputs = layer_module( 2025-08-26T20:25:45.3896667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3896766Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3897037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3897136Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3897400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3897477Z return func(*args, **kwargs) 2025-08-26T20:25:45.3897761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3897837Z self_outputs = self.self( 2025-08-26T20:25:45.3898116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3898196Z return func(*args, **kwargs) 2025-08-26T20:25:45.3898574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3898737Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3898741Z 2025-08-26T20:25:45.3898858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3899087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3899161Z return mod(**inputs) 2025-08-26T20:25:45.3899450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3899526Z outputs = self.bert( 2025-08-26T20:25:45.3899804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3899894Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3900167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3900287Z layer_outputs = layer_module( 2025-08-26T20:25:45.3900536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3900622Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3900907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3900997Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3901307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3901383Z return func(*args, **kwargs) 2025-08-26T20:25:45.3901651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3901803Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3902074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3902176Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3902180Z 2025-08-26T20:25:45.3902295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3902523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3902596Z return mod(**inputs) 2025-08-26T20:25:45.3902869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3902953Z outputs = self.bert( 2025-08-26T20:25:45.3903225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3903312Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3903587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3903668Z layer_outputs = layer_module( 2025-08-26T20:25:45.3903935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3904023Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3904299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3904393Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3904682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3904777Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3905083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3905261Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3905539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3905638Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3905642Z 2025-08-26T20:25:45.3905756Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3905977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3906059Z return mod(**inputs) 2025-08-26T20:25:45.3906337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3906416Z outputs = self.bert( 2025-08-26T20:25:45.3906694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3906779Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3907063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3907174Z layer_outputs = layer_module( 2025-08-26T20:25:45.3907424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3907510Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3907793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3907886Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3908195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3908289Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3908600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3908762Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3909034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3909158Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3909401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3909479Z return self.act(input) 2025-08-26T20:25:45.3909484Z 2025-08-26T20:25:45.3909604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3909829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3909907Z return mod(**inputs) 2025-08-26T20:25:45.3910189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3910264Z outputs = self.bert( 2025-08-26T20:25:45.3910547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3910632Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3910919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3910995Z layer_outputs = layer_module( 2025-08-26T20:25:45.3911232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3911325Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3911587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3911685Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3912002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3912091Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3912397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3912540Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3912813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3912903Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3912909Z 2025-08-26T20:25:45.3913026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3913240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3913310Z return mod(**inputs) 2025-08-26T20:25:45.3913585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3913655Z outputs = self.bert( 2025-08-26T20:25:45.3913948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3914027Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3914290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3914373Z layer_outputs = layer_module( 2025-08-26T20:25:45.3914610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3914723Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3914988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3915077Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3915354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3915432Z return func(*args, **kwargs) 2025-08-26T20:25:45.3915703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3915780Z self_outputs = self.self( 2025-08-26T20:25:45.3916048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3916123Z return func(*args, **kwargs) 2025-08-26T20:25:45.3916392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3916627Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3916632Z 2025-08-26T20:25:45.3916744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3936147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3936368Z return mod(**inputs) 2025-08-26T20:25:45.3936701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3936783Z outputs = self.bert( 2025-08-26T20:25:45.3937052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3937147Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3937409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3937506Z layer_outputs = layer_module( 2025-08-26T20:25:45.3937747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3937840Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3938257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3938368Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3938630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3938707Z return func(*args, **kwargs) 2025-08-26T20:25:45.3938961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3939052Z self_outputs = self.self( 2025-08-26T20:25:45.3939318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3939402Z return func(*args, **kwargs) 2025-08-26T20:25:45.3939673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3939754Z self.key(current_states) 2025-08-26T20:25:45.3939767Z 2025-08-26T20:25:45.3939894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3940155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3940241Z return mod(**inputs) 2025-08-26T20:25:45.3940511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3940594Z outputs = self.bert( 2025-08-26T20:25:45.3940860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3940997Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3941275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3941353Z layer_outputs = layer_module( 2025-08-26T20:25:45.3941610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3941701Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3941965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3942062Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3942319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3942404Z return func(*args, **kwargs) 2025-08-26T20:25:45.3942667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3942746Z self_outputs = self.self( 2025-08-26T20:25:45.3943012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3943087Z return func(*args, **kwargs) 2025-08-26T20:25:45.3943364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3943444Z self.value(current_states) 2025-08-26T20:25:45.3943448Z 2025-08-26T20:25:45.3943547Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3943663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3943884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3943966Z return mod(**inputs) 2025-08-26T20:25:45.3944244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3944328Z outputs = self.bert( 2025-08-26T20:25:45.3944604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3944696Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3945001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3945081Z layer_outputs = layer_module( 2025-08-26T20:25:45.3945349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3945435Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3945697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3945792Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3946049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3946139Z return func(*args, **kwargs) 2025-08-26T20:25:45.3946411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3946496Z self_outputs = self.self( 2025-08-26T20:25:45.3946751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3946845Z return func(*args, **kwargs) 2025-08-26T20:25:45.3947119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3947265Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3947270Z 2025-08-26T20:25:45.3947393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3947628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3947699Z return mod(**inputs) 2025-08-26T20:25:45.3947967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3948038Z outputs = self.bert( 2025-08-26T20:25:45.3948314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3948395Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3948663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3948739Z layer_outputs = layer_module( 2025-08-26T20:25:45.3948973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3949063Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3949327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3949420Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3949683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3949760Z return func(*args, **kwargs) 2025-08-26T20:25:45.3950043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3950187Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3950458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3950550Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3950554Z 2025-08-26T20:25:45.3950673Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3950888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3950959Z return mod(**inputs) 2025-08-26T20:25:45.3951235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3951306Z outputs = self.bert( 2025-08-26T20:25:45.3951613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3951697Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3951965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3952052Z layer_outputs = layer_module( 2025-08-26T20:25:45.3952296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3952393Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3952682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3952777Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3953083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3953172Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3953513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3953652Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3953946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3954040Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3954044Z 2025-08-26T20:25:45.3954160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3954411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3954484Z return mod(**inputs) 2025-08-26T20:25:45.3954766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3954844Z outputs = self.bert( 2025-08-26T20:25:45.3955121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3955213Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3955494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3955581Z layer_outputs = layer_module( 2025-08-26T20:25:45.3955825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3955915Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3956203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3956296Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3956611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3956695Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3957016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3957152Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3957423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3957557Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3957796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3957882Z return self.act(input) 2025-08-26T20:25:45.3957886Z 2025-08-26T20:25:45.3958001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3958260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3958344Z return mod(**inputs) 2025-08-26T20:25:45.3958624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3958705Z outputs = self.bert( 2025-08-26T20:25:45.3958981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3959069Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3959451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3959540Z layer_outputs = layer_module( 2025-08-26T20:25:45.3959792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3959880Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3960163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3960257Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3960571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3960669Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3960978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3961145Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3961427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3961515Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3961528Z 2025-08-26T20:25:45.3961642Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3961857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3961939Z return mod(**inputs) 2025-08-26T20:25:45.3962204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3962284Z outputs = self.bert( 2025-08-26T20:25:45.3962554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3962635Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3962916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3962996Z layer_outputs = layer_module( 2025-08-26T20:25:45.3963244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3963329Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3963603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3963702Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3963967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3964051Z return func(*args, **kwargs) 2025-08-26T20:25:45.3964320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3964397Z self_outputs = self.self( 2025-08-26T20:25:45.3964673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3964748Z return func(*args, **kwargs) 2025-08-26T20:25:45.3965011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3965274Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3965280Z 2025-08-26T20:25:45.3965402Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3965620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3965694Z return mod(**inputs) 2025-08-26T20:25:45.3965973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3966046Z outputs = self.bert( 2025-08-26T20:25:45.3966330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3966412Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3966684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3966776Z layer_outputs = layer_module( 2025-08-26T20:25:45.3967022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3967163Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3967432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3967528Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3967794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3967890Z return func(*args, **kwargs) 2025-08-26T20:25:45.3968172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3968259Z self_outputs = self.self( 2025-08-26T20:25:45.3968528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3968601Z return func(*args, **kwargs) 2025-08-26T20:25:45.3968865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3968951Z self.key(current_states) 2025-08-26T20:25:45.3968954Z 2025-08-26T20:25:45.3969065Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3969285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3969355Z return mod(**inputs) 2025-08-26T20:25:45.3969632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3969713Z outputs = self.bert( 2025-08-26T20:25:45.3969989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3970076Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3970348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3970435Z layer_outputs = layer_module( 2025-08-26T20:25:45.3970681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3970768Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3971047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3971135Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3971410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3971485Z return func(*args, **kwargs) 2025-08-26T20:25:45.3971756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3971878Z self_outputs = self.self( 2025-08-26T20:25:45.3972147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3972232Z return func(*args, **kwargs) 2025-08-26T20:25:45.3972502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.3972583Z self.value(current_states) 2025-08-26T20:25:45.3972595Z 2025-08-26T20:25:45.3972688Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.3972804Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3973032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3973103Z return mod(**inputs) 2025-08-26T20:25:45.3973384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3973461Z outputs = self.bert( 2025-08-26T20:25:45.3973738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3973846Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3974114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3974199Z layer_outputs = layer_module( 2025-08-26T20:25:45.3974441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3974527Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3974829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3974916Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3975194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3975270Z return func(*args, **kwargs) 2025-08-26T20:25:45.3975542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3975629Z self_outputs = self.self( 2025-08-26T20:25:45.3975895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3975980Z return func(*args, **kwargs) 2025-08-26T20:25:45.3976253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.3976405Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.3976415Z 2025-08-26T20:25:45.3976528Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3976746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3976832Z return mod(**inputs) 2025-08-26T20:25:45.3977109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3977192Z outputs = self.bert( 2025-08-26T20:25:45.3977467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3977547Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3977828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3977906Z layer_outputs = layer_module( 2025-08-26T20:25:45.3978159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3978244Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3978515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3978645Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3978911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3978994Z return func(*args, **kwargs) 2025-08-26T20:25:45.3979263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.3979415Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.3979683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.3979779Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3979782Z 2025-08-26T20:25:45.3979902Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3980120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3980203Z return mod(**inputs) 2025-08-26T20:25:45.3980474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3980564Z outputs = self.bert( 2025-08-26T20:25:45.3980846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3980925Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3981207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3981305Z layer_outputs = layer_module( 2025-08-26T20:25:45.3981551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3981645Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3981920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3982021Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3982311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3982403Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3982712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3982849Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3983127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.3983232Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3983236Z 2025-08-26T20:25:45.3983354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3983568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3983640Z return mod(**inputs) 2025-08-26T20:25:45.3983915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3983987Z outputs = self.bert( 2025-08-26T20:25:45.3984260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3984339Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3984608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3984687Z layer_outputs = layer_module( 2025-08-26T20:25:45.3984923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3985015Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3985307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3985406Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3985689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3985772Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3986078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.3986207Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.3986482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.3986604Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.3986837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.3986924Z return self.act(input) 2025-08-26T20:25:45.3986928Z 2025-08-26T20:25:45.3987039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3987279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3987351Z return mod(**inputs) 2025-08-26T20:25:45.3987625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3987697Z outputs = self.bert( 2025-08-26T20:25:45.3987963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3988066Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3988333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3988418Z layer_outputs = layer_module( 2025-08-26T20:25:45.3988658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3988745Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3989032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.3989124Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.3989424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.3989507Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.3989822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.3989970Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.3990256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.3990354Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.3990360Z 2025-08-26T20:25:45.3990473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3990701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3990774Z return mod(**inputs) 2025-08-26T20:25:45.3991050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3991132Z outputs = self.bert( 2025-08-26T20:25:45.3991407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3991497Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3991776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3991853Z layer_outputs = layer_module( 2025-08-26T20:25:45.3992154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3992243Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3992518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3992607Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3992876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3992953Z return func(*args, **kwargs) 2025-08-26T20:25:45.3993222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3993309Z self_outputs = self.self( 2025-08-26T20:25:45.3993584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3993671Z return func(*args, **kwargs) 2025-08-26T20:25:45.3993959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.3994210Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.3994221Z 2025-08-26T20:25:45.3994336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3994566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3994644Z return mod(**inputs) 2025-08-26T20:25:45.3994933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3995011Z outputs = self.bert( 2025-08-26T20:25:45.3995277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3995358Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3995631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3995708Z layer_outputs = layer_module( 2025-08-26T20:25:45.3995951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.3996034Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.3996592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.3996756Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.3997081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3997168Z return func(*args, **kwargs) 2025-08-26T20:25:45.3997444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.3997523Z self_outputs = self.self( 2025-08-26T20:25:45.3997812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.3997888Z return func(*args, **kwargs) 2025-08-26T20:25:45.3998172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.3998252Z self.key(current_states) 2025-08-26T20:25:45.3998256Z 2025-08-26T20:25:45.3998380Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.3998601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.3998677Z return mod(**inputs) 2025-08-26T20:25:45.3998966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.3999038Z outputs = self.bert( 2025-08-26T20:25:45.3999541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.3999631Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.3999903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.3999992Z layer_outputs = layer_module( 2025-08-26T20:25:45.4000236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4000327Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4000612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4000702Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4000980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4001058Z return func(*args, **kwargs) 2025-08-26T20:25:45.4001336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4001449Z self_outputs = self.self( 2025-08-26T20:25:45.4001725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4001810Z return func(*args, **kwargs) 2025-08-26T20:25:45.4002087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.4002206Z self.value(current_states) 2025-08-26T20:25:45.4002210Z 2025-08-26T20:25:45.4002300Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.4002415Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4002643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4002718Z return mod(**inputs) 2025-08-26T20:25:45.4003004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4003078Z outputs = self.bert( 2025-08-26T20:25:45.4003355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4003446Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4003716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4003805Z layer_outputs = layer_module( 2025-08-26T20:25:45.4004055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4004148Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4004429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4004518Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4004798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4004873Z return func(*args, **kwargs) 2025-08-26T20:25:45.4005158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4005234Z self_outputs = self.self( 2025-08-26T20:25:45.4005500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4005586Z return func(*args, **kwargs) 2025-08-26T20:25:45.4005858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.4006013Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.4006016Z 2025-08-26T20:25:45.4006165Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4006387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4006466Z return mod(**inputs) 2025-08-26T20:25:45.4006738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4006819Z outputs = self.bert( 2025-08-26T20:25:45.4007093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4007182Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4007452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4007531Z layer_outputs = layer_module( 2025-08-26T20:25:45.4007789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4007875Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4008171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4008260Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4008526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4008610Z return func(*args, **kwargs) 2025-08-26T20:25:45.4008878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.4009049Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.4009322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.4009415Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4009430Z 2025-08-26T20:25:45.4009547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4009768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4009848Z return mod(**inputs) 2025-08-26T20:25:45.4010121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4010202Z outputs = self.bert( 2025-08-26T20:25:45.4010479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4010561Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4010837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4010916Z layer_outputs = layer_module( 2025-08-26T20:25:45.4011170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4011257Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4011530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4011632Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4011918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4012012Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4012320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.4012464Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.4012737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.4012860Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4012864Z 2025-08-26T20:25:45.4012987Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4013212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4013291Z return mod(**inputs) 2025-08-26T20:25:45.4013575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4013646Z outputs = self.bert( 2025-08-26T20:25:45.4013920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4013998Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4014267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4014341Z layer_outputs = layer_module( 2025-08-26T20:25:45.4014580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4014688Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4014948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4015043Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4015321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4015408Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4015728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.4015857Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.4016129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.4016251Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.4016490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.4016565Z return self.act(input) 2025-08-26T20:25:45.4016569Z 2025-08-26T20:25:45.4016679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4016897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4016968Z return mod(**inputs) 2025-08-26T20:25:45.4017241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4017314Z outputs = self.bert( 2025-08-26T20:25:45.4017589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4017668Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4017936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4018023Z layer_outputs = layer_module( 2025-08-26T20:25:45.4018262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4018352Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4018617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4018707Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4018996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4019077Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4019426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.4019573Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.4019840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.4019936Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4019940Z 2025-08-26T20:25:45.4020049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4020271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4020341Z return mod(**inputs) 2025-08-26T20:25:45.4020620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4020692Z outputs = self.bert( 2025-08-26T20:25:45.4020960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4021048Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4021313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4021416Z layer_outputs = layer_module( 2025-08-26T20:25:45.4021655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4021739Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4022011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4022121Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4022384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4022459Z return func(*args, **kwargs) 2025-08-26T20:25:45.4022723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4022805Z self_outputs = self.self( 2025-08-26T20:25:45.4023062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4023144Z return func(*args, **kwargs) 2025-08-26T20:25:45.4023406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.4023635Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.4023639Z 2025-08-26T20:25:45.4023751Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4023962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4024052Z return mod(**inputs) 2025-08-26T20:25:45.4024305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4024379Z outputs = self.bert( 2025-08-26T20:25:45.4024629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4024703Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4024958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4025031Z layer_outputs = layer_module( 2025-08-26T20:25:45.4025260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4025341Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4025594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4025675Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4025952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4026034Z return func(*args, **kwargs) 2025-08-26T20:25:45.4026285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4026363Z self_outputs = self.self( 2025-08-26T20:25:45.4026619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4026692Z return func(*args, **kwargs) 2025-08-26T20:25:45.4026968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.4027047Z self.key(current_states) 2025-08-26T20:25:45.4027050Z 2025-08-26T20:25:45.4027168Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4027383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4027455Z return mod(**inputs) 2025-08-26T20:25:45.4027732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4027822Z outputs = self.bert( 2025-08-26T20:25:45.4028096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4028175Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4028446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4028540Z layer_outputs = layer_module( 2025-08-26T20:25:45.4028777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4028870Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4029135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4029229Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4029487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4029561Z return func(*args, **kwargs) 2025-08-26T20:25:45.4029832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4029906Z self_outputs = self.self( 2025-08-26T20:25:45.4030169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4030243Z return func(*args, **kwargs) 2025-08-26T20:25:45.4030502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.4030588Z self.value(current_states) 2025-08-26T20:25:45.4030592Z 2025-08-26T20:25:45.4030681Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.4030800Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4031010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4031087Z return mod(**inputs) 2025-08-26T20:25:45.4031351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4031421Z outputs = self.bert( 2025-08-26T20:25:45.4031689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4031769Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4032038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4032113Z layer_outputs = layer_module( 2025-08-26T20:25:45.4032382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4032477Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4032741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4032834Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4033086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4033160Z return func(*args, **kwargs) 2025-08-26T20:25:45.4033431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4033508Z self_outputs = self.self( 2025-08-26T20:25:45.4033773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4033845Z return func(*args, **kwargs) 2025-08-26T20:25:45.4034110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.4034279Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.4034282Z 2025-08-26T20:25:45.4034395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4034616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4034686Z return mod(**inputs) 2025-08-26T20:25:45.4034962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4035055Z outputs = self.bert( 2025-08-26T20:25:45.4035321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4035408Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4035674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4035759Z layer_outputs = layer_module( 2025-08-26T20:25:45.4035998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4036080Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4036351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4036438Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4036702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4036779Z return func(*args, **kwargs) 2025-08-26T20:25:45.4037048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.4037190Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.4037456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.4037554Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4037558Z 2025-08-26T20:25:45.4037668Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4037886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4037956Z return mod(**inputs) 2025-08-26T20:25:45.4038225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4038309Z outputs = self.bert( 2025-08-26T20:25:45.4038582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4038670Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4038973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4039055Z layer_outputs = layer_module( 2025-08-26T20:25:45.4039403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4039496Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4039784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4039879Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4040176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4040265Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4040571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.4040718Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.4040995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.4041119Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4041124Z 2025-08-26T20:25:45.4041236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4041454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4041537Z return mod(**inputs) 2025-08-26T20:25:45.4041816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4041929Z outputs = self.bert( 2025-08-26T20:25:45.4042206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4042295Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4042573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4042653Z layer_outputs = layer_module( 2025-08-26T20:25:45.4042909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4042994Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4043273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4043366Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4043657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4043748Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4044055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.4044198Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.4044472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.4044599Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.4044844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.4044922Z return self.act(input) 2025-08-26T20:25:45.4044926Z 2025-08-26T20:25:45.4045048Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4045270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4045350Z return mod(**inputs) 2025-08-26T20:25:45.4045629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4045744Z outputs = self.bert( 2025-08-26T20:25:45.4046030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4046112Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4046389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4046467Z layer_outputs = layer_module( 2025-08-26T20:25:45.4046709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4046800Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4047071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4047168Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4047455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4047545Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4047868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.4048014Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.4048291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.4048381Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4048385Z 2025-08-26T20:25:45.4048522Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4048739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4048811Z return mod(**inputs) 2025-08-26T20:25:45.4049094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4049172Z outputs = self.bert( 2025-08-26T20:25:45.4049450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4049533Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4049800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4049885Z layer_outputs = layer_module( 2025-08-26T20:25:45.4050129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4050227Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4050493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4050590Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4050857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4050934Z return func(*args, **kwargs) 2025-08-26T20:25:45.4051213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4051292Z self_outputs = self.self( 2025-08-26T20:25:45.4051572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4051648Z return func(*args, **kwargs) 2025-08-26T20:25:45.4051926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:25:45.4052171Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:25:45.4052175Z 2025-08-26T20:25:45.4052291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4052554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4052629Z return mod(**inputs) 2025-08-26T20:25:45.4052917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4053001Z outputs = self.bert( 2025-08-26T20:25:45.4053268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4053355Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4053618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4053718Z layer_outputs = layer_module( 2025-08-26T20:25:45.4053955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4054040Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4054315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4054425Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4054693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4054768Z return func(*args, **kwargs) 2025-08-26T20:25:45.4055031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4055113Z self_outputs = self.self( 2025-08-26T20:25:45.4055374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4055476Z return func(*args, **kwargs) 2025-08-26T20:25:45.4055740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:25:45.4055818Z self.key(current_states) 2025-08-26T20:25:45.4055831Z 2025-08-26T20:25:45.4055943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4056156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4056235Z return mod(**inputs) 2025-08-26T20:25:45.4056502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4056581Z outputs = self.bert( 2025-08-26T20:25:45.4056848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4056927Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4057198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4057275Z layer_outputs = layer_module( 2025-08-26T20:25:45.4057523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4057607Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4057872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4057968Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4058227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4058308Z return func(*args, **kwargs) 2025-08-26T20:25:45.4058572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4058648Z self_outputs = self.self( 2025-08-26T20:25:45.4058929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4059004Z return func(*args, **kwargs) 2025-08-26T20:25:45.4059321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:25:45.4059405Z self.value(current_states) 2025-08-26T20:25:45.4059410Z 2025-08-26T20:25:45.4059517Z cudagraph partition due to non gpu ops 2025-08-26T20:25:45.4059629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4059842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4059920Z return mod(**inputs) 2025-08-26T20:25:45.4060188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4060270Z outputs = self.bert( 2025-08-26T20:25:45.4060544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4060624Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4060913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4061007Z layer_outputs = layer_module( 2025-08-26T20:25:45.4061254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4061336Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4061605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4061699Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4061974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4062053Z return func(*args, **kwargs) 2025-08-26T20:25:45.4062311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:25:45.4062384Z self_outputs = self.self( 2025-08-26T20:25:45.4062650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4062724Z return func(*args, **kwargs) 2025-08-26T20:25:45.4062993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:25:45.4063134Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:25:45.4063138Z 2025-08-26T20:25:45.4063255Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4063465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4063537Z return mod(**inputs) 2025-08-26T20:25:45.4063811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4063882Z outputs = self.bert( 2025-08-26T20:25:45.4064157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4064238Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4064500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4064583Z layer_outputs = layer_module( 2025-08-26T20:25:45.4064819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4064912Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4065173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:25:45.4065268Z self_attention_outputs = self.attention( 2025-08-26T20:25:45.4065526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:25:45.4065641Z return func(*args, **kwargs) 2025-08-26T20:25:45.4065914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:25:45.4066057Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:25:45.4066326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:25:45.4066415Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4066419Z 2025-08-26T20:25:45.4066532Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4066754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4066827Z return mod(**inputs) 2025-08-26T20:25:45.4067102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4067173Z outputs = self.bert( 2025-08-26T20:25:45.4067443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4067557Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4067831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4067916Z layer_outputs = layer_module( 2025-08-26T20:25:45.4068167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4068260Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4068556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4068648Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4068937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4069023Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4069340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.4069475Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.4069752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:25:45.4069850Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4069853Z 2025-08-26T20:25:45.4069966Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4070198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4070271Z return mod(**inputs) 2025-08-26T20:25:45.4070552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4070624Z outputs = self.bert( 2025-08-26T20:25:45.4070903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4070992Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4071265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4071349Z layer_outputs = layer_module( 2025-08-26T20:25:45.4071593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4071678Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4071960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4072051Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4072385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4072470Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4072777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:25:45.4072917Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:25:45.4073191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:25:45.4073324Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:25:45.4073561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:25:45.4073649Z return self.act(input) 2025-08-26T20:25:45.4073653Z 2025-08-26T20:25:45.4073766Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4073986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4074070Z return mod(**inputs) 2025-08-26T20:25:45.4074347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-26T20:25:45.4074465Z outputs = self.bert( 2025-08-26T20:25:45.4074744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:25:45.4074825Z encoder_outputs = self.encoder( 2025-08-26T20:25:45.4075112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:25:45.4075213Z layer_outputs = layer_module( 2025-08-26T20:25:45.4075469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:25:45.4075556Z return super().__call__(*args, **kwargs) 2025-08-26T20:25:45.4075833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:25:45.4075932Z layer_output = apply_chunking_to_forward( 2025-08-26T20:25:45.4076223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:25:45.4076315Z return forward_fn(*input_tensors) 2025-08-26T20:25:45.4076629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:25:45.4076783Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:25:45.4077058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:25:45.4077150Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4077153Z 2025-08-26T20:25:45.4077275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4077500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4077578Z return mod(**inputs) 2025-08-26T20:25:45.4077858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-08-26T20:25:45.4077963Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:25:45.4078244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-08-26T20:25:45.4078369Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:25:45.4078654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward 2025-08-26T20:25:45.4078759Z hidden_states = self.transform(hidden_states) 2025-08-26T20:25:45.4079051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward 2025-08-26T20:25:45.4079140Z hidden_states = self.dense(hidden_states) 2025-08-26T20:25:45.4079187Z 2025-08-26T20:25:45.4079392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4079628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4079703Z return mod(**inputs) 2025-08-26T20:25:45.4079985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-08-26T20:25:45.4080086Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:25:45.4080354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-08-26T20:25:45.4080486Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:25:45.4080755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward 2025-08-26T20:25:45.4080863Z hidden_states = self.decoder(hidden_states) 2025-08-26T20:25:45.4080871Z 2025-08-26T20:25:45.4080986Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:25:45.4081239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:25:45.4081313Z return mod(**inputs) 2025-08-26T20:25:45.4081593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1328, in forward 2025-08-26T20:25:45.4081817Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:25:45.4081821Z 2025-08-26T20:25:54.1385576Z Compilation time (from dynamo_timed): 15.226656461 2025-08-26T20:25:54.1467643Z pass 2025-08-26T20:25:54.1470653Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:25:54.1471500Z TIMING: _recursive_pre_grad_passes:0.00718 _recursive_joint_graph_passes:0.37849 _recursive_post_grad_passes:0.07789 async_compile.wait:0.81405 code_gen:7.5569 inductor_compile:8.81314 backend_compile:12.05791 gc:0.0002 entire_frame_compile:15.22666 total_wall_time:15.22666 2025-08-26T20:25:54.1472405Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:12331 | FakeTensor.__torch_dispatch__:4342 | ProxyTorchDispatchMode.__torch_dispatch__:4495 2025-08-26T20:25:54.1472904Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-08-26T20:25:59.4645446Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:25:59.4646654Z from pkg_resources import resource_filename 2025-08-26T20:26:00.0385472Z 2025-08-26T20:26:01.2438978Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:26:01.2439627Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:26:01.2455182Z cpu eval BertForQuestionAnswering 2025-08-26T20:26:01.6469346Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:01.8465903Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:02.0434485Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:09.8799742Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8800103Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8800373Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8800642Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8800872Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8801116Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8801349Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8801577Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8801794Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8802510Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8802749Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8803020Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8803342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8803753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8804129Z return mod(**inputs) 2025-08-26T20:26:09.8804566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8804990Z outputs = self.bert( 2025-08-26T20:26:09.8805372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8805833Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8806250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8806663Z layer_outputs = layer_module( 2025-08-26T20:26:09.8807120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8807524Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8807945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8808376Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8808800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8809280Z return func(*args, **kwargs) 2025-08-26T20:26:09.8809680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8810106Z self_outputs = self.self( 2025-08-26T20:26:09.8810510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8810974Z return func(*args, **kwargs) 2025-08-26T20:26:09.8811409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.8812012Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.8812308Z 2025-08-26T20:26:09.8812428Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8812834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8813200Z return mod(**inputs) 2025-08-26T20:26:09.8813596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8814010Z outputs = self.bert( 2025-08-26T20:26:09.8814398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8814840Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8815254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8815671Z layer_outputs = layer_module( 2025-08-26T20:26:09.8816056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8816496Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8816917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8817350Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8817771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8818184Z return func(*args, **kwargs) 2025-08-26T20:26:09.8818647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8819069Z self_outputs = self.self( 2025-08-26T20:26:09.8819446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8819844Z return func(*args, **kwargs) 2025-08-26T20:26:09.8820240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.8820643Z self.key(current_states) 2025-08-26T20:26:09.8820769Z 2025-08-26T20:26:09.8820890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8821279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8821634Z return mod(**inputs) 2025-08-26T20:26:09.8822059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8822464Z outputs = self.bert( 2025-08-26T20:26:09.8822870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8823266Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8823657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8824046Z layer_outputs = layer_module( 2025-08-26T20:26:09.8824407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8824808Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8825218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8825644Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8826056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8826460Z return func(*args, **kwargs) 2025-08-26T20:26:09.8826853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8827244Z self_outputs = self.self( 2025-08-26T20:26:09.8827614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8827991Z return func(*args, **kwargs) 2025-08-26T20:26:09.8828359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.8828757Z self.value(current_states) 2025-08-26T20:26:09.8828891Z 2025-08-26T20:26:09.8828979Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8829237Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8829628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8829969Z return mod(**inputs) 2025-08-26T20:26:09.8830355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8830760Z outputs = self.bert( 2025-08-26T20:26:09.8831153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8831562Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8831966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8832374Z layer_outputs = layer_module( 2025-08-26T20:26:09.8832746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8833134Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8833580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8833997Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8834400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8834817Z return func(*args, **kwargs) 2025-08-26T20:26:09.8835222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8835630Z self_outputs = self.self( 2025-08-26T20:26:09.8836038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8836442Z return func(*args, **kwargs) 2025-08-26T20:26:09.8836850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.8837340Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.8837552Z 2025-08-26T20:26:09.8837693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8838102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8838461Z return mod(**inputs) 2025-08-26T20:26:09.8838858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8839572Z outputs = self.bert( 2025-08-26T20:26:09.8839991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8840447Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8840863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8841285Z layer_outputs = layer_module( 2025-08-26T20:26:09.8841654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8842042Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8842448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8842865Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8843272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8843663Z return func(*args, **kwargs) 2025-08-26T20:26:09.8844058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.8844533Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.8844992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.8845417Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8845576Z 2025-08-26T20:26:09.8845690Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8846071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8846416Z return mod(**inputs) 2025-08-26T20:26:09.8846797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8847190Z outputs = self.bert( 2025-08-26T20:26:09.8847568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8847982Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8848382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8848795Z layer_outputs = layer_module( 2025-08-26T20:26:09.8849202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8849599Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8850009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8850444Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8850850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8851253Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8851673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.8852164Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.8852627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.8853040Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8853240Z 2025-08-26T20:26:09.8853354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8853740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8854089Z return mod(**inputs) 2025-08-26T20:26:09.8854464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8854839Z outputs = self.bert( 2025-08-26T20:26:09.8855219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8855605Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8855987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8856368Z layer_outputs = layer_module( 2025-08-26T20:26:09.8856723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8857090Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8857494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8857913Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8858337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8858768Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8859184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.8859649Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.8860080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.8860495Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.8860880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.8861223Z return self.act(input) 2025-08-26T20:26:09.8861338Z 2025-08-26T20:26:09.8861451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8861807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8862143Z return mod(**inputs) 2025-08-26T20:26:09.8862504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8862882Z outputs = self.bert( 2025-08-26T20:26:09.8863238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8863650Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8864035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8864420Z layer_outputs = layer_module( 2025-08-26T20:26:09.8864770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8865129Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8865519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8865917Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8866322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8866715Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8867123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.8867619Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.8868073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.8868620Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8868758Z 2025-08-26T20:26:09.8868873Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8869229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8869583Z return mod(**inputs) 2025-08-26T20:26:09.8869957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8870356Z outputs = self.bert( 2025-08-26T20:26:09.8870732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8871142Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8871542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8871961Z layer_outputs = layer_module( 2025-08-26T20:26:09.8872334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8872713Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8873127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8873545Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8873959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8874367Z return func(*args, **kwargs) 2025-08-26T20:26:09.8874758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8875167Z self_outputs = self.self( 2025-08-26T20:26:09.8875561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8875960Z return func(*args, **kwargs) 2025-08-26T20:26:09.8876347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.8876905Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.8877200Z 2025-08-26T20:26:09.8877312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8877709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8878059Z return mod(**inputs) 2025-08-26T20:26:09.8878477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8878886Z outputs = self.bert( 2025-08-26T20:26:09.8879352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8879784Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8880190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8880614Z layer_outputs = layer_module( 2025-08-26T20:26:09.8880993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8881387Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8881806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8882225Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8882641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8883062Z return func(*args, **kwargs) 2025-08-26T20:26:09.8883460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8883861Z self_outputs = self.self( 2025-08-26T20:26:09.8884239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8884633Z return func(*args, **kwargs) 2025-08-26T20:26:09.8885047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.8885458Z self.key(current_states) 2025-08-26T20:26:09.8885579Z 2025-08-26T20:26:09.8885699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8886080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8886436Z return mod(**inputs) 2025-08-26T20:26:09.8886823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8887225Z outputs = self.bert( 2025-08-26T20:26:09.8887599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8888008Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8888408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8888814Z layer_outputs = layer_module( 2025-08-26T20:26:09.8889182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8889560Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8889969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8890386Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8890792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8891188Z return func(*args, **kwargs) 2025-08-26T20:26:09.8891578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8891985Z self_outputs = self.self( 2025-08-26T20:26:09.8892371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8892776Z return func(*args, **kwargs) 2025-08-26T20:26:09.8893162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.8893629Z self.value(current_states) 2025-08-26T20:26:09.8893767Z 2025-08-26T20:26:09.8893856Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8894119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8894500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8894850Z return mod(**inputs) 2025-08-26T20:26:09.8895238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8895646Z outputs = self.bert( 2025-08-26T20:26:09.8896031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8896633Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8897048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8897468Z layer_outputs = layer_module( 2025-08-26T20:26:09.8897852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8898304Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8898723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8899150Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8899565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8899971Z return func(*args, **kwargs) 2025-08-26T20:26:09.8900393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8900783Z self_outputs = self.self( 2025-08-26T20:26:09.8901150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8901525Z return func(*args, **kwargs) 2025-08-26T20:26:09.8901894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.8902334Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.8902529Z 2025-08-26T20:26:09.8902636Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8903001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8903332Z return mod(**inputs) 2025-08-26T20:26:09.8903706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8904109Z outputs = self.bert( 2025-08-26T20:26:09.8904490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8904898Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8905299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8905700Z layer_outputs = layer_module( 2025-08-26T20:26:09.8906072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8906454Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8906845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8907255Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8907654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8908054Z return func(*args, **kwargs) 2025-08-26T20:26:09.8908442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.8908997Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.8909437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.8909836Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8909984Z 2025-08-26T20:26:09.8910089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8910458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8910789Z return mod(**inputs) 2025-08-26T20:26:09.8911147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8911529Z outputs = self.bert( 2025-08-26T20:26:09.8911893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8912639Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8913047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8913483Z layer_outputs = layer_module( 2025-08-26T20:26:09.8913870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8914275Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8914703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8915161Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8915621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8916071Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8916541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.8917052Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.8917535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.8917992Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8918155Z 2025-08-26T20:26:09.8918273Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8918683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8919052Z return mod(**inputs) 2025-08-26T20:26:09.8919517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8919945Z outputs = self.bert( 2025-08-26T20:26:09.8920394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8920825Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8921244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8921673Z layer_outputs = layer_module( 2025-08-26T20:26:09.8922061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8922461Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8922933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8923373Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8923821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8924268Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8924772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.8925255Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.8925685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.8926126Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.8926505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.8926844Z return self.act(input) 2025-08-26T20:26:09.8926957Z 2025-08-26T20:26:09.8927060Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8927418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8927741Z return mod(**inputs) 2025-08-26T20:26:09.8928095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8928474Z outputs = self.bert( 2025-08-26T20:26:09.8928873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8929252Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8929628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8930011Z layer_outputs = layer_module( 2025-08-26T20:26:09.8930363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8930737Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8931114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8931502Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8931900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8932285Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8932686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.8933149Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.8933582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.8933968Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8934108Z 2025-08-26T20:26:09.8934213Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8934570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8934893Z return mod(**inputs) 2025-08-26T20:26:09.8935248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8935621Z outputs = self.bert( 2025-08-26T20:26:09.8935960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8936337Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8936703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8937077Z layer_outputs = layer_module( 2025-08-26T20:26:09.8937412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8937772Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8938146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8938530Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8938936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8939301Z return func(*args, **kwargs) 2025-08-26T20:26:09.8939666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8940039Z self_outputs = self.self( 2025-08-26T20:26:09.8940396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8940752Z return func(*args, **kwargs) 2025-08-26T20:26:09.8941118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.8941632Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.8941894Z 2025-08-26T20:26:09.8942011Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8942378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8942719Z return mod(**inputs) 2025-08-26T20:26:09.8943084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8943467Z outputs = self.bert( 2025-08-26T20:26:09.8943831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8944220Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8944639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8945020Z layer_outputs = layer_module( 2025-08-26T20:26:09.8945370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8945739Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8946120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8946516Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8946904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8947281Z return func(*args, **kwargs) 2025-08-26T20:26:09.8947653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8948027Z self_outputs = self.self( 2025-08-26T20:26:09.8948393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8948768Z return func(*args, **kwargs) 2025-08-26T20:26:09.8949143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.8949525Z self.key(current_states) 2025-08-26T20:26:09.8949643Z 2025-08-26T20:26:09.8949750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8950119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8950452Z return mod(**inputs) 2025-08-26T20:26:09.8950815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8951192Z outputs = self.bert( 2025-08-26T20:26:09.8951553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8951949Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8952334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8952749Z layer_outputs = layer_module( 2025-08-26T20:26:09.8953143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8953535Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8953948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8954376Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8954781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8955189Z return func(*args, **kwargs) 2025-08-26T20:26:09.8955584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8956000Z self_outputs = self.self( 2025-08-26T20:26:09.8956389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8956791Z return func(*args, **kwargs) 2025-08-26T20:26:09.8957195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.8957627Z self.value(current_states) 2025-08-26T20:26:09.8957754Z 2025-08-26T20:26:09.8957851Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.8958102Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8958489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8958837Z return mod(**inputs) 2025-08-26T20:26:09.8959317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8959745Z outputs = self.bert( 2025-08-26T20:26:09.8960126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8960541Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8960929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8961340Z layer_outputs = layer_module( 2025-08-26T20:26:09.8961708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8962104Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8962523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8962955Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8963365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8963765Z return func(*args, **kwargs) 2025-08-26T20:26:09.8964166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.8964576Z self_outputs = self.self( 2025-08-26T20:26:09.8964977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8965381Z return func(*args, **kwargs) 2025-08-26T20:26:09.8965774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.8966244Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.8966450Z 2025-08-26T20:26:09.8966561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8966948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8967301Z return mod(**inputs) 2025-08-26T20:26:09.8967686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8968146Z outputs = self.bert( 2025-08-26T20:26:09.8968535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8968954Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8969341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8969721Z layer_outputs = layer_module( 2025-08-26T20:26:09.8970069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8970438Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8970825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.8971212Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.8971600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.8971976Z return func(*args, **kwargs) 2025-08-26T20:26:09.8972365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.8972795Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.8973230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.8973628Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8973770Z 2025-08-26T20:26:09.8973906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8974270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8974593Z return mod(**inputs) 2025-08-26T20:26:09.8974957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8975344Z outputs = self.bert( 2025-08-26T20:26:09.8975704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8976093Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8976463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8976848Z layer_outputs = layer_module( 2025-08-26T20:26:09.8977198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8977563Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8977938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8978334Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8978742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8979139Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8979554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.8980009Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.8980439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.8980834Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8980976Z 2025-08-26T20:26:09.8981091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8981460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8981802Z return mod(**inputs) 2025-08-26T20:26:09.8982226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8982613Z outputs = self.bert( 2025-08-26T20:26:09.8982974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8983353Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8983735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8984116Z layer_outputs = layer_module( 2025-08-26T20:26:09.8984467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8984835Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8985218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8985612Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8986023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8986450Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8986860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.8987320Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.8987749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.8988173Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.8988576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.8988913Z return self.act(input) 2025-08-26T20:26:09.8989033Z 2025-08-26T20:26:09.8989139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8989509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8989837Z return mod(**inputs) 2025-08-26T20:26:09.8990201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8990572Z outputs = self.bert( 2025-08-26T20:26:09.8990930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8991320Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8991699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.8992079Z layer_outputs = layer_module( 2025-08-26T20:26:09.8992454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.8992819Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.8993212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.8993629Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.8994051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.8994473Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.8994906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.8995412Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.8995872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.8996450Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.8996608Z 2025-08-26T20:26:09.8996720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.8997206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.8997567Z return mod(**inputs) 2025-08-26T20:26:09.8997953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.8998370Z outputs = self.bert( 2025-08-26T20:26:09.8998764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.8999190Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.8999650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9000061Z layer_outputs = layer_module( 2025-08-26T20:26:09.9000432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9000830Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9001233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9001665Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9002050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9002429Z return func(*args, **kwargs) 2025-08-26T20:26:09.9002814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9003253Z self_outputs = self.self( 2025-08-26T20:26:09.9003639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9004047Z return func(*args, **kwargs) 2025-08-26T20:26:09.9004454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9005008Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9005275Z 2025-08-26T20:26:09.9005391Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9005772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9006134Z return mod(**inputs) 2025-08-26T20:26:09.9006516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9006922Z outputs = self.bert( 2025-08-26T20:26:09.9007305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9007709Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9008113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9008520Z layer_outputs = layer_module( 2025-08-26T20:26:09.9008888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9009269Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9009678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9010095Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9010502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9010902Z return func(*args, **kwargs) 2025-08-26T20:26:09.9011287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9011701Z self_outputs = self.self( 2025-08-26T20:26:09.9012116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9012516Z return func(*args, **kwargs) 2025-08-26T20:26:09.9012903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9013315Z self.key(current_states) 2025-08-26T20:26:09.9013442Z 2025-08-26T20:26:09.9013553Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9013937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9014284Z return mod(**inputs) 2025-08-26T20:26:09.9014659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9015058Z outputs = self.bert( 2025-08-26T20:26:09.9015437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9015850Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9016245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9016729Z layer_outputs = layer_module( 2025-08-26T20:26:09.9017108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9017479Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9017867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9018288Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9018681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9019063Z return func(*args, **kwargs) 2025-08-26T20:26:09.9019444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9019829Z self_outputs = self.self( 2025-08-26T20:26:09.9020190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9020571Z return func(*args, **kwargs) 2025-08-26T20:26:09.9020946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9021333Z self.value(current_states) 2025-08-26T20:26:09.9021454Z 2025-08-26T20:26:09.9021539Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9021790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9022159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9022495Z return mod(**inputs) 2025-08-26T20:26:09.9022879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9023291Z outputs = self.bert( 2025-08-26T20:26:09.9023657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9024051Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9024429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9024810Z layer_outputs = layer_module( 2025-08-26T20:26:09.9025164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9025537Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9025928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9026329Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9026746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9027126Z return func(*args, **kwargs) 2025-08-26T20:26:09.9027500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9027886Z self_outputs = self.self( 2025-08-26T20:26:09.9028249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9028612Z return func(*args, **kwargs) 2025-08-26T20:26:09.9028982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9029428Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9029615Z 2025-08-26T20:26:09.9029726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9030088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9030425Z return mod(**inputs) 2025-08-26T20:26:09.9030774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9031173Z outputs = self.bert( 2025-08-26T20:26:09.9031521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9031890Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9032262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9032676Z layer_outputs = layer_module( 2025-08-26T20:26:09.9033043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9033425Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9033845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9034267Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9034685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9035099Z return func(*args, **kwargs) 2025-08-26T20:26:09.9035489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9035976Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9036443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9036872Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9037026Z 2025-08-26T20:26:09.9037150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9037543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9037899Z return mod(**inputs) 2025-08-26T20:26:09.9038293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9038702Z outputs = self.bert( 2025-08-26T20:26:09.9039080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9039591Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9039996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9040419Z layer_outputs = layer_module( 2025-08-26T20:26:09.9040767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9041116Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9041544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9041937Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9042342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9042740Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9043149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9043619Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9044040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9044426Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9044561Z 2025-08-26T20:26:09.9044671Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9045025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9045368Z return mod(**inputs) 2025-08-26T20:26:09.9045721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9046092Z outputs = self.bert( 2025-08-26T20:26:09.9046435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9046816Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9047187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9047595Z layer_outputs = layer_module( 2025-08-26T20:26:09.9047943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9048303Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9048705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9049091Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9049486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9049867Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9050278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9050740Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9051174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9051603Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9051987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9052340Z return self.act(input) 2025-08-26T20:26:09.9052457Z 2025-08-26T20:26:09.9052564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9052925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9053247Z return mod(**inputs) 2025-08-26T20:26:09.9053603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9053986Z outputs = self.bert( 2025-08-26T20:26:09.9054348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9054742Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9055115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9055505Z layer_outputs = layer_module( 2025-08-26T20:26:09.9055900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9056281Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9056668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9057065Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9057474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9057868Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9058306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9058805Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9059247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9059646Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9059814Z 2025-08-26T20:26:09.9059923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9060289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9060612Z return mod(**inputs) 2025-08-26T20:26:09.9060974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9061354Z outputs = self.bert( 2025-08-26T20:26:09.9061735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9062125Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9062500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9062892Z layer_outputs = layer_module( 2025-08-26T20:26:09.9063245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9063621Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9064036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9064447Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9064837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9065215Z return func(*args, **kwargs) 2025-08-26T20:26:09.9065594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9065964Z self_outputs = self.self( 2025-08-26T20:26:09.9066322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9066690Z return func(*args, **kwargs) 2025-08-26T20:26:09.9067097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9067617Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9067898Z 2025-08-26T20:26:09.9068010Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9068400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9068764Z return mod(**inputs) 2025-08-26T20:26:09.9069151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9069557Z outputs = self.bert( 2025-08-26T20:26:09.9069995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9070414Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9070822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9071235Z layer_outputs = layer_module( 2025-08-26T20:26:09.9071601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9071989Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9072404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9072830Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9073251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9073666Z return func(*args, **kwargs) 2025-08-26T20:26:09.9074074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9074531Z self_outputs = self.self( 2025-08-26T20:26:09.9074932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9075328Z return func(*args, **kwargs) 2025-08-26T20:26:09.9075735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9076158Z self.key(current_states) 2025-08-26T20:26:09.9076285Z 2025-08-26T20:26:09.9076428Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9076832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9077186Z return mod(**inputs) 2025-08-26T20:26:09.9077579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9078007Z outputs = self.bert( 2025-08-26T20:26:09.9078393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9078808Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9079216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9079717Z layer_outputs = layer_module( 2025-08-26T20:26:09.9080104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9080504Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9080917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9081363Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9081791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9082189Z return func(*args, **kwargs) 2025-08-26T20:26:09.9082577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9082953Z self_outputs = self.self( 2025-08-26T20:26:09.9083316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9083702Z return func(*args, **kwargs) 2025-08-26T20:26:09.9084093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9084493Z self.value(current_states) 2025-08-26T20:26:09.9084628Z 2025-08-26T20:26:09.9084716Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9084970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9085397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9085755Z return mod(**inputs) 2025-08-26T20:26:09.9086147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9086565Z outputs = self.bert( 2025-08-26T20:26:09.9086954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9087377Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9087786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9088210Z layer_outputs = layer_module( 2025-08-26T20:26:09.9088595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9088993Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9089419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9089852Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9090268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9090675Z return func(*args, **kwargs) 2025-08-26T20:26:09.9091078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9091486Z self_outputs = self.self( 2025-08-26T20:26:09.9091879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9092304Z return func(*args, **kwargs) 2025-08-26T20:26:09.9092707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9093267Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9093477Z 2025-08-26T20:26:09.9093593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9093999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9094356Z return mod(**inputs) 2025-08-26T20:26:09.9094758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9095177Z outputs = self.bert( 2025-08-26T20:26:09.9095561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9095987Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9096589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9097038Z layer_outputs = layer_module( 2025-08-26T20:26:09.9097419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9097831Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9098249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9098680Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9099093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9099552Z return func(*args, **kwargs) 2025-08-26T20:26:09.9099959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9100433Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9100913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9101443Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9101605Z 2025-08-26T20:26:09.9101722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9102123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9102482Z return mod(**inputs) 2025-08-26T20:26:09.9102875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9103284Z outputs = self.bert( 2025-08-26T20:26:09.9103682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9104098Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9104501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9104906Z layer_outputs = layer_module( 2025-08-26T20:26:09.9105274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9105690Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9106095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9106512Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9106950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9107385Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9107862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9108363Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9108833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9109243Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9109402Z 2025-08-26T20:26:09.9109512Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9109894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9110241Z return mod(**inputs) 2025-08-26T20:26:09.9110620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9111011Z outputs = self.bert( 2025-08-26T20:26:09.9111386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9111795Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9112193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9112587Z layer_outputs = layer_module( 2025-08-26T20:26:09.9112956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9113337Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9113743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9114170Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9114588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9115013Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9115451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9115947Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9116441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9116897Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9117319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9117693Z return self.act(input) 2025-08-26T20:26:09.9117816Z 2025-08-26T20:26:09.9117940Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9118338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9118689Z return mod(**inputs) 2025-08-26T20:26:09.9119084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9119566Z outputs = self.bert( 2025-08-26T20:26:09.9119965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9120390Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9120806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9121256Z layer_outputs = layer_module( 2025-08-26T20:26:09.9121644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9122044Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9122463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9122917Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9123360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9123790Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9124236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9124767Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9125255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9125685Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9125838Z 2025-08-26T20:26:09.9125960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9126351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9126712Z return mod(**inputs) 2025-08-26T20:26:09.9127105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9127521Z outputs = self.bert( 2025-08-26T20:26:09.9127916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9128335Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9128756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9129177Z layer_outputs = layer_module( 2025-08-26T20:26:09.9129563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9129957Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9130380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9130805Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9131229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9131636Z return func(*args, **kwargs) 2025-08-26T20:26:09.9132078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9132494Z self_outputs = self.self( 2025-08-26T20:26:09.9132889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9133293Z return func(*args, **kwargs) 2025-08-26T20:26:09.9133678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9134235Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9134527Z 2025-08-26T20:26:09.9134638Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9135026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9135375Z return mod(**inputs) 2025-08-26T20:26:09.9135751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9136155Z outputs = self.bert( 2025-08-26T20:26:09.9136543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9136936Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9137332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9137726Z layer_outputs = layer_module( 2025-08-26T20:26:09.9138099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9138507Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9138917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9139344Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9139745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9140145Z return func(*args, **kwargs) 2025-08-26T20:26:09.9140540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9140954Z self_outputs = self.self( 2025-08-26T20:26:09.9141334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9141741Z return func(*args, **kwargs) 2025-08-26T20:26:09.9142130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9142533Z self.key(current_states) 2025-08-26T20:26:09.9142649Z 2025-08-26T20:26:09.9142760Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9143122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9143453Z return mod(**inputs) 2025-08-26T20:26:09.9143837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9144241Z outputs = self.bert( 2025-08-26T20:26:09.9144614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9145024Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9145425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9145830Z layer_outputs = layer_module( 2025-08-26T20:26:09.9146198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9146574Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9147016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9147441Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9147853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9148247Z return func(*args, **kwargs) 2025-08-26T20:26:09.9148647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9149057Z self_outputs = self.self( 2025-08-26T20:26:09.9149447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9149850Z return func(*args, **kwargs) 2025-08-26T20:26:09.9150239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9150653Z self.value(current_states) 2025-08-26T20:26:09.9150794Z 2025-08-26T20:26:09.9150886Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9151169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9151553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9151895Z return mod(**inputs) 2025-08-26T20:26:09.9152277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9152683Z outputs = self.bert( 2025-08-26T20:26:09.9153061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9153480Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9153890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9154308Z layer_outputs = layer_module( 2025-08-26T20:26:09.9154692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9155091Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9155512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9155929Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9156334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9156732Z return func(*args, **kwargs) 2025-08-26T20:26:09.9157129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9157553Z self_outputs = self.self( 2025-08-26T20:26:09.9157945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9158366Z return func(*args, **kwargs) 2025-08-26T20:26:09.9158766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9159320Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9159543Z 2025-08-26T20:26:09.9159660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9160065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9160420Z return mod(**inputs) 2025-08-26T20:26:09.9160813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9161233Z outputs = self.bert( 2025-08-26T20:26:09.9161612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9162022Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9162493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9162890Z layer_outputs = layer_module( 2025-08-26T20:26:09.9163269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9163665Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9164085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9164519Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9164938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9165343Z return func(*args, **kwargs) 2025-08-26T20:26:09.9165741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9166229Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9166708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9167174Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9167338Z 2025-08-26T20:26:09.9167454Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9167857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9168234Z return mod(**inputs) 2025-08-26T20:26:09.9168618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9169058Z outputs = self.bert( 2025-08-26T20:26:09.9169463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9169890Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9170308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9170724Z layer_outputs = layer_module( 2025-08-26T20:26:09.9171106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9171516Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9171946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9172381Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9172828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9173274Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9173733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9174239Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9174703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9175160Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9175316Z 2025-08-26T20:26:09.9175430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9175828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9176202Z return mod(**inputs) 2025-08-26T20:26:09.9176591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9177007Z outputs = self.bert( 2025-08-26T20:26:09.9177400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9177864Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9178275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9178698Z layer_outputs = layer_module( 2025-08-26T20:26:09.9179080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9179470Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9179889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9180313Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9180752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9181198Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9181654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9182157Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9182640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9183106Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9183533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9183912Z return self.act(input) 2025-08-26T20:26:09.9184036Z 2025-08-26T20:26:09.9184169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9184572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9184933Z return mod(**inputs) 2025-08-26T20:26:09.9185337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9185760Z outputs = self.bert( 2025-08-26T20:26:09.9186149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9186577Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9186991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9187424Z layer_outputs = layer_module( 2025-08-26T20:26:09.9187803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9188208Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9188629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9189062Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9189506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9189935Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9190390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9190907Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9191385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9191814Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9191970Z 2025-08-26T20:26:09.9192083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9192481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9192838Z return mod(**inputs) 2025-08-26T20:26:09.9193284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9193709Z outputs = self.bert( 2025-08-26T20:26:09.9194099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9194521Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9194944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9195365Z layer_outputs = layer_module( 2025-08-26T20:26:09.9195742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9196149Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9196714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9197145Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9197566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9198026Z return func(*args, **kwargs) 2025-08-26T20:26:09.9198435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9198861Z self_outputs = self.self( 2025-08-26T20:26:09.9199303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9199727Z return func(*args, **kwargs) 2025-08-26T20:26:09.9200126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9200736Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9201026Z 2025-08-26T20:26:09.9201150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9201556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9201907Z return mod(**inputs) 2025-08-26T20:26:09.9202305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9202720Z outputs = self.bert( 2025-08-26T20:26:09.9203105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9203517Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9203919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9204331Z layer_outputs = layer_module( 2025-08-26T20:26:09.9204705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9205104Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9205518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9205943Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9206354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9206430Z return func(*args, **kwargs) 2025-08-26T20:26:09.9206708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9206787Z self_outputs = self.self( 2025-08-26T20:26:09.9207061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9207139Z return func(*args, **kwargs) 2025-08-26T20:26:09.9207408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9207548Z self.key(current_states) 2025-08-26T20:26:09.9207553Z 2025-08-26T20:26:09.9207674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9207905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9207978Z return mod(**inputs) 2025-08-26T20:26:09.9208258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9208330Z outputs = self.bert( 2025-08-26T20:26:09.9208604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9208697Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9208976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9209061Z layer_outputs = layer_module( 2025-08-26T20:26:09.9209309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9209417Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9209698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9209789Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9210120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9210198Z return func(*args, **kwargs) 2025-08-26T20:26:09.9210500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9210587Z self_outputs = self.self( 2025-08-26T20:26:09.9210863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9210951Z return func(*args, **kwargs) 2025-08-26T20:26:09.9211236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9211326Z self.value(current_states) 2025-08-26T20:26:09.9211331Z 2025-08-26T20:26:09.9211421Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9211537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9211764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9211838Z return mod(**inputs) 2025-08-26T20:26:09.9212120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9212195Z outputs = self.bert( 2025-08-26T20:26:09.9212469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9212560Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9212841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9212931Z layer_outputs = layer_module( 2025-08-26T20:26:09.9213175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9213260Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9213548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9213636Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9213923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9213999Z return func(*args, **kwargs) 2025-08-26T20:26:09.9214281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9214389Z self_outputs = self.self( 2025-08-26T20:26:09.9214668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9214752Z return func(*args, **kwargs) 2025-08-26T20:26:09.9215073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9215222Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9215226Z 2025-08-26T20:26:09.9215338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9215549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9215627Z return mod(**inputs) 2025-08-26T20:26:09.9215892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9215971Z outputs = self.bert( 2025-08-26T20:26:09.9216236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9216334Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9216615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9216693Z layer_outputs = layer_module( 2025-08-26T20:26:09.9216936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9217018Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9217318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9217405Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9217671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9217756Z return func(*args, **kwargs) 2025-08-26T20:26:09.9218017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9218164Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9218425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9218515Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9218519Z 2025-08-26T20:26:09.9218638Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9218853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9218929Z return mod(**inputs) 2025-08-26T20:26:09.9219194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9219275Z outputs = self.bert( 2025-08-26T20:26:09.9219541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9219621Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9219890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9219968Z layer_outputs = layer_module( 2025-08-26T20:26:09.9220210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9220294Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9220557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9220657Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9220973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9221064Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9221365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9221498Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9221769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9221856Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9221860Z 2025-08-26T20:26:09.9221979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9222192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9222270Z return mod(**inputs) 2025-08-26T20:26:09.9222536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9222611Z outputs = self.bert( 2025-08-26T20:26:09.9222885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9222988Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9223257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9223333Z layer_outputs = layer_module( 2025-08-26T20:26:09.9223568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9223689Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9223947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9224042Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9224321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9224409Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9224711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9224841Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9225117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9225238Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9225478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9225552Z return self.act(input) 2025-08-26T20:26:09.9225556Z 2025-08-26T20:26:09.9225668Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9225895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9225965Z return mod(**inputs) 2025-08-26T20:26:09.9226246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9226320Z outputs = self.bert( 2025-08-26T20:26:09.9226590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9226677Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9226945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9227032Z layer_outputs = layer_module( 2025-08-26T20:26:09.9227270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9227362Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9227662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9227754Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9228039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9228119Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9228426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9228568Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9228834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9228928Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9228931Z 2025-08-26T20:26:09.9229040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9229262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9229363Z return mod(**inputs) 2025-08-26T20:26:09.9229618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9229683Z outputs = self.bert( 2025-08-26T20:26:09.9229932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9230009Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9230253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9230349Z layer_outputs = layer_module( 2025-08-26T20:26:09.9230565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9230645Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9230894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9230976Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9231218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9231287Z return func(*args, **kwargs) 2025-08-26T20:26:09.9231525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9231601Z self_outputs = self.self( 2025-08-26T20:26:09.9231839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9231914Z return func(*args, **kwargs) 2025-08-26T20:26:09.9232159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9232376Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9232381Z 2025-08-26T20:26:09.9232485Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9232684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9232758Z return mod(**inputs) 2025-08-26T20:26:09.9233008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9233081Z outputs = self.bert( 2025-08-26T20:26:09.9233331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9233404Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9233660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9233766Z layer_outputs = layer_module( 2025-08-26T20:26:09.9234008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9234093Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9234353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9234446Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9234703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9234787Z return func(*args, **kwargs) 2025-08-26T20:26:09.9235053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9235136Z self_outputs = self.self( 2025-08-26T20:26:09.9235398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9235472Z return func(*args, **kwargs) 2025-08-26T20:26:09.9235770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9235846Z self.key(current_states) 2025-08-26T20:26:09.9235850Z 2025-08-26T20:26:09.9235971Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9236187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9236257Z return mod(**inputs) 2025-08-26T20:26:09.9236531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9236621Z outputs = self.bert( 2025-08-26T20:26:09.9236900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9236977Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9237251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9237336Z layer_outputs = layer_module( 2025-08-26T20:26:09.9237573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9237663Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9237930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9238022Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9238286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9238360Z return func(*args, **kwargs) 2025-08-26T20:26:09.9238646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9238723Z self_outputs = self.self( 2025-08-26T20:26:09.9238992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9239065Z return func(*args, **kwargs) 2025-08-26T20:26:09.9239418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9239518Z self.value(current_states) 2025-08-26T20:26:09.9239523Z 2025-08-26T20:26:09.9239616Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9239740Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9239963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9240036Z return mod(**inputs) 2025-08-26T20:26:09.9240319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9240432Z outputs = self.bert( 2025-08-26T20:26:09.9240720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9240793Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9241042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9241112Z layer_outputs = layer_module( 2025-08-26T20:26:09.9241330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9241416Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9241659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9241746Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9241985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9242053Z return func(*args, **kwargs) 2025-08-26T20:26:09.9243381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9243451Z self_outputs = self.self( 2025-08-26T20:26:09.9243693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9243764Z return func(*args, **kwargs) 2025-08-26T20:26:09.9244005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9244174Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9244178Z 2025-08-26T20:26:09.9244282Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9244492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9244561Z return mod(**inputs) 2025-08-26T20:26:09.9244824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9244893Z outputs = self.bert( 2025-08-26T20:26:09.9245146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9245229Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9245476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9245557Z layer_outputs = layer_module( 2025-08-26T20:26:09.9245777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9245855Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9246114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9246196Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9246449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9246518Z return func(*args, **kwargs) 2025-08-26T20:26:09.9246766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9246904Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9247153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9247246Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9247249Z 2025-08-26T20:26:09.9247352Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9247560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9247659Z return mod(**inputs) 2025-08-26T20:26:09.9247911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9247985Z outputs = self.bert( 2025-08-26T20:26:09.9248236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9248319Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9248565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9248638Z layer_outputs = layer_module( 2025-08-26T20:26:09.9248868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9248946Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9249202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9249289Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9249596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9249677Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9249965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9250100Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9250353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9250464Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9250468Z 2025-08-26T20:26:09.9250575Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9250783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9250859Z return mod(**inputs) 2025-08-26T20:26:09.9251119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9251196Z outputs = self.bert( 2025-08-26T20:26:09.9251453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9251529Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9251790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9251867Z layer_outputs = layer_module( 2025-08-26T20:26:09.9252104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9252184Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9252450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9252537Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9252809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9252897Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9253182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9253315Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9253574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9253691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9253921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9254022Z return self.act(input) 2025-08-26T20:26:09.9254026Z 2025-08-26T20:26:09.9254143Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9254346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9254421Z return mod(**inputs) 2025-08-26T20:26:09.9254679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9254748Z outputs = self.bert( 2025-08-26T20:26:09.9255013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9255092Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9255355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9255429Z layer_outputs = layer_module( 2025-08-26T20:26:09.9255659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9255768Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9256016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9256101Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9256348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9256421Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9256707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9256835Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9257079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9257157Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9257162Z 2025-08-26T20:26:09.9257266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9257455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9257518Z return mod(**inputs) 2025-08-26T20:26:09.9257764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9257829Z outputs = self.bert( 2025-08-26T20:26:09.9258074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9258145Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9258383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9258466Z layer_outputs = layer_module( 2025-08-26T20:26:09.9258687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9258776Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9259030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9259118Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9259356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9259428Z return func(*args, **kwargs) 2025-08-26T20:26:09.9259674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9259744Z self_outputs = self.self( 2025-08-26T20:26:09.9260021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9260091Z return func(*args, **kwargs) 2025-08-26T20:26:09.9260330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9260540Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9260544Z 2025-08-26T20:26:09.9260645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9260851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9260916Z return mod(**inputs) 2025-08-26T20:26:09.9261156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9261226Z outputs = self.bert( 2025-08-26T20:26:09.9261468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9261550Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9261790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9261898Z layer_outputs = layer_module( 2025-08-26T20:26:09.9262115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9262189Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9262433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9262526Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9262766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9262837Z return func(*args, **kwargs) 2025-08-26T20:26:09.9263075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9263155Z self_outputs = self.self( 2025-08-26T20:26:09.9263394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9263470Z return func(*args, **kwargs) 2025-08-26T20:26:09.9263713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9263785Z self.key(current_states) 2025-08-26T20:26:09.9263796Z 2025-08-26T20:26:09.9263901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9264102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9264178Z return mod(**inputs) 2025-08-26T20:26:09.9264424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9264503Z outputs = self.bert( 2025-08-26T20:26:09.9264752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9264828Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9265084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9265159Z layer_outputs = layer_module( 2025-08-26T20:26:09.9265393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9265475Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9265724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9265817Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9266089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9266170Z return func(*args, **kwargs) 2025-08-26T20:26:09.9266417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9266488Z self_outputs = self.self( 2025-08-26T20:26:09.9266735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9266808Z return func(*args, **kwargs) 2025-08-26T20:26:09.9267074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9267154Z self.value(current_states) 2025-08-26T20:26:09.9267158Z 2025-08-26T20:26:09.9267253Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9267365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9267580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9267660Z return mod(**inputs) 2025-08-26T20:26:09.9267924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9268028Z outputs = self.bert( 2025-08-26T20:26:09.9268280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9268354Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9268605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9268699Z layer_outputs = layer_module( 2025-08-26T20:26:09.9268930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9269008Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9269262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9269359Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9269598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9269674Z return func(*args, **kwargs) 2025-08-26T20:26:09.9269914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9269989Z self_outputs = self.self( 2025-08-26T20:26:09.9270227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9270296Z return func(*args, **kwargs) 2025-08-26T20:26:09.9270546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9270678Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9270684Z 2025-08-26T20:26:09.9270793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9270988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9271054Z return mod(**inputs) 2025-08-26T20:26:09.9271307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9271371Z outputs = self.bert( 2025-08-26T20:26:09.9271623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9271697Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9271936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9272039Z layer_outputs = layer_module( 2025-08-26T20:26:09.9272295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9272386Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9272632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9272720Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9272978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9273046Z return func(*args, **kwargs) 2025-08-26T20:26:09.9273295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9273425Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9273673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9273756Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9273760Z 2025-08-26T20:26:09.9273862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9274082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9274149Z return mod(**inputs) 2025-08-26T20:26:09.9274406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9274473Z outputs = self.bert( 2025-08-26T20:26:09.9274721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9274821Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9275068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9275149Z layer_outputs = layer_module( 2025-08-26T20:26:09.9275371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9275457Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9275704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9275788Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9276056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9276134Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9276416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9276539Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9276786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9276881Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9276885Z 2025-08-26T20:26:09.9276990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9277197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9277263Z return mod(**inputs) 2025-08-26T20:26:09.9277520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9277586Z outputs = self.bert( 2025-08-26T20:26:09.9277838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9277920Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9278166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9278246Z layer_outputs = layer_module( 2025-08-26T20:26:09.9278529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9278612Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9278871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9278955Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9279313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9279405Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9279707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9279847Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9280116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9280244Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9280496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9280580Z return self.act(input) 2025-08-26T20:26:09.9280585Z 2025-08-26T20:26:09.9280694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9280908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9280988Z return mod(**inputs) 2025-08-26T20:26:09.9281255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9281355Z outputs = self.bert( 2025-08-26T20:26:09.9281623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9281698Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9281959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9282036Z layer_outputs = layer_module( 2025-08-26T20:26:09.9282264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9282343Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9282603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9282688Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9282953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9283036Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9283318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9283461Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9283713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9283795Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9283799Z 2025-08-26T20:26:09.9283912Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9284112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9284185Z return mod(**inputs) 2025-08-26T20:26:09.9284439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9284506Z outputs = self.bert( 2025-08-26T20:26:09.9284772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9284879Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9285128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9285200Z layer_outputs = layer_module( 2025-08-26T20:26:09.9285424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9285502Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9285741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9285831Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9286067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9286142Z return func(*args, **kwargs) 2025-08-26T20:26:09.9286383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9286451Z self_outputs = self.self( 2025-08-26T20:26:09.9286711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9286780Z return func(*args, **kwargs) 2025-08-26T20:26:09.9287031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9287235Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9287255Z 2025-08-26T20:26:09.9287366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9287562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9287627Z return mod(**inputs) 2025-08-26T20:26:09.9287884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9287949Z outputs = self.bert( 2025-08-26T20:26:09.9288209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9288283Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9288528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9288610Z layer_outputs = layer_module( 2025-08-26T20:26:09.9288833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9288922Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9289169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9289250Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9289508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9289578Z return func(*args, **kwargs) 2025-08-26T20:26:09.9289828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9289899Z self_outputs = self.self( 2025-08-26T20:26:09.9290146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9290215Z return func(*args, **kwargs) 2025-08-26T20:26:09.9290462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9290542Z self.key(current_states) 2025-08-26T20:26:09.9290546Z 2025-08-26T20:26:09.9290650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9290914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9290985Z return mod(**inputs) 2025-08-26T20:26:09.9291238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9291315Z outputs = self.bert( 2025-08-26T20:26:09.9291563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9291643Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9291889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9291962Z layer_outputs = layer_module( 2025-08-26T20:26:09.9292191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9292269Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9292527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9292609Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9292876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9292946Z return func(*args, **kwargs) 2025-08-26T20:26:09.9293197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9293275Z self_outputs = self.self( 2025-08-26T20:26:09.9293518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9293614Z return func(*args, **kwargs) 2025-08-26T20:26:09.9293863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9293937Z self.value(current_states) 2025-08-26T20:26:09.9293941Z 2025-08-26T20:26:09.9294034Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9294141Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9294351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9294418Z return mod(**inputs) 2025-08-26T20:26:09.9294671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9294746Z outputs = self.bert( 2025-08-26T20:26:09.9294997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9295081Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9295329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9295401Z layer_outputs = layer_module( 2025-08-26T20:26:09.9295635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9295714Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9295968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9296051Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9296420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9296495Z return func(*args, **kwargs) 2025-08-26T20:26:09.9296742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9296826Z self_outputs = self.self( 2025-08-26T20:26:09.9297069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9297147Z return func(*args, **kwargs) 2025-08-26T20:26:09.9297473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9297614Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9297618Z 2025-08-26T20:26:09.9297730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9297930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9298003Z return mod(**inputs) 2025-08-26T20:26:09.9298252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9298327Z outputs = self.bert( 2025-08-26T20:26:09.9298578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9298652Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9298921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9299020Z layer_outputs = layer_module( 2025-08-26T20:26:09.9299253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9299334Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9299586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9299681Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9299927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9300039Z return func(*args, **kwargs) 2025-08-26T20:26:09.9300287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9300422Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9300678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9300763Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9300766Z 2025-08-26T20:26:09.9300878Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9301081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9301155Z return mod(**inputs) 2025-08-26T20:26:09.9301405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9301474Z outputs = self.bert( 2025-08-26T20:26:09.9301734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9301808Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9302064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9302138Z layer_outputs = layer_module( 2025-08-26T20:26:09.9302363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9302449Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9302696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9302789Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9303052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9303129Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9303415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9303569Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9303827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9303912Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9303915Z 2025-08-26T20:26:09.9304024Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9304224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9304290Z return mod(**inputs) 2025-08-26T20:26:09.9304547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9304614Z outputs = self.bert( 2025-08-26T20:26:09.9304876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9304950Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9305201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9305298Z layer_outputs = layer_module( 2025-08-26T20:26:09.9305520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9305606Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9305854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9305943Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9306222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9306298Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9306583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9306706Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9306957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9307071Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9307284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9307363Z return self.act(input) 2025-08-26T20:26:09.9307367Z 2025-08-26T20:26:09.9307479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9307680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9307744Z return mod(**inputs) 2025-08-26T20:26:09.9307985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9308059Z outputs = self.bert( 2025-08-26T20:26:09.9308301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9308381Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9308620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9308705Z layer_outputs = layer_module( 2025-08-26T20:26:09.9308913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9308990Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9309229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9309307Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9309593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9309671Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9309943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9310084Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9310326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9310412Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9310416Z 2025-08-26T20:26:09.9310519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9310723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9310789Z return mod(**inputs) 2025-08-26T20:26:09.9311035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9311111Z outputs = self.bert( 2025-08-26T20:26:09.9311362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9311462Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9311711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9311784Z layer_outputs = layer_module( 2025-08-26T20:26:09.9312015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9312112Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9312369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9312456Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9312715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9312800Z return func(*args, **kwargs) 2025-08-26T20:26:09.9313060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9313142Z self_outputs = self.self( 2025-08-26T20:26:09.9313397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9313477Z return func(*args, **kwargs) 2025-08-26T20:26:09.9313738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9313959Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9313963Z 2025-08-26T20:26:09.9314081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9314298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9314376Z return mod(**inputs) 2025-08-26T20:26:09.9314643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9314713Z outputs = self.bert( 2025-08-26T20:26:09.9314985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9315061Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9315330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9315408Z layer_outputs = layer_module( 2025-08-26T20:26:09.9315648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9315731Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9316024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9316122Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9316377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9316458Z return func(*args, **kwargs) 2025-08-26T20:26:09.9316717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9316791Z self_outputs = self.self( 2025-08-26T20:26:09.9317062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9317135Z return func(*args, **kwargs) 2025-08-26T20:26:09.9317405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9317485Z self.key(current_states) 2025-08-26T20:26:09.9317490Z 2025-08-26T20:26:09.9317601Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9317841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9317911Z return mod(**inputs) 2025-08-26T20:26:09.9318182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9318253Z outputs = self.bert( 2025-08-26T20:26:09.9318523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9318628Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9318890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9318976Z layer_outputs = layer_module( 2025-08-26T20:26:09.9319215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9319373Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9319655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9319745Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9320021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9320095Z return func(*args, **kwargs) 2025-08-26T20:26:09.9320373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9320452Z self_outputs = self.self( 2025-08-26T20:26:09.9320718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9320801Z return func(*args, **kwargs) 2025-08-26T20:26:09.9321062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9321160Z self.value(current_states) 2025-08-26T20:26:09.9321163Z 2025-08-26T20:26:09.9321246Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9321361Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9321560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9321626Z return mod(**inputs) 2025-08-26T20:26:09.9321886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9321955Z outputs = self.bert( 2025-08-26T20:26:09.9322218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9322292Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9322573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9322657Z layer_outputs = layer_module( 2025-08-26T20:26:09.9322882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9322966Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9323222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9323300Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9323546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9323614Z return func(*args, **kwargs) 2025-08-26T20:26:09.9323864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9323938Z self_outputs = self.self( 2025-08-26T20:26:09.9324180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9324277Z return func(*args, **kwargs) 2025-08-26T20:26:09.9324523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9324664Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9324668Z 2025-08-26T20:26:09.9324773Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9324980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9325066Z return mod(**inputs) 2025-08-26T20:26:09.9325322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9325396Z outputs = self.bert( 2025-08-26T20:26:09.9325655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9325737Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9325987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9326059Z layer_outputs = layer_module( 2025-08-26T20:26:09.9326293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9326372Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9326633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9326715Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9326961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9327039Z return func(*args, **kwargs) 2025-08-26T20:26:09.9327293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9327431Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9327682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9327772Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9327775Z 2025-08-26T20:26:09.9327878Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9328082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9328157Z return mod(**inputs) 2025-08-26T20:26:09.9328413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9328486Z outputs = self.bert( 2025-08-26T20:26:09.9328771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9328850Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9329103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9329176Z layer_outputs = layer_module( 2025-08-26T20:26:09.9329404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9329483Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9329739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9329824Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9330089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9330177Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9330454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9330604Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9330856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9330939Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9330942Z 2025-08-26T20:26:09.9331054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9331275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9331347Z return mod(**inputs) 2025-08-26T20:26:09.9331599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9331674Z outputs = self.bert( 2025-08-26T20:26:09.9331925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9331999Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9332254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9332326Z layer_outputs = layer_module( 2025-08-26T20:26:09.9332556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9332636Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9332888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9332981Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9333246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9333332Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9333613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9333736Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9333992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9334105Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9334326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9334400Z return self.act(input) 2025-08-26T20:26:09.9334404Z 2025-08-26T20:26:09.9334514Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9334742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9334811Z return mod(**inputs) 2025-08-26T20:26:09.9335071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9335139Z outputs = self.bert( 2025-08-26T20:26:09.9335400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9335475Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9335722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9335803Z layer_outputs = layer_module( 2025-08-26T20:26:09.9336023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9336108Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9336358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9336442Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9336734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9336810Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9337094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9337230Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9337504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9337584Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9337587Z 2025-08-26T20:26:09.9337687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9337893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9337958Z return mod(**inputs) 2025-08-26T20:26:09.9338207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9338271Z outputs = self.bert( 2025-08-26T20:26:09.9338514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9338592Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9338830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9338909Z layer_outputs = layer_module( 2025-08-26T20:26:09.9339126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9339211Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9339456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9339538Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9339782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9339850Z return func(*args, **kwargs) 2025-08-26T20:26:09.9340100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9340168Z self_outputs = self.self( 2025-08-26T20:26:09.9340404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9340482Z return func(*args, **kwargs) 2025-08-26T20:26:09.9340723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-26T20:26:09.9340967Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:26:09.9340972Z 2025-08-26T20:26:09.9341076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9341279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9341345Z return mod(**inputs) 2025-08-26T20:26:09.9341589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9341663Z outputs = self.bert( 2025-08-26T20:26:09.9341905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9341988Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9342236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9342308Z layer_outputs = layer_module( 2025-08-26T20:26:09.9342543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9342654Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9342907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9342988Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9343236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9343315Z return func(*args, **kwargs) 2025-08-26T20:26:09.9343585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9343661Z self_outputs = self.self( 2025-08-26T20:26:09.9343895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9343964Z return func(*args, **kwargs) 2025-08-26T20:26:09.9344211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-26T20:26:09.9344285Z self.key(current_states) 2025-08-26T20:26:09.9344289Z 2025-08-26T20:26:09.9344397Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9344593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9344664Z return mod(**inputs) 2025-08-26T20:26:09.9344907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9344974Z outputs = self.bert( 2025-08-26T20:26:09.9345224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9345295Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9345545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9345616Z layer_outputs = layer_module( 2025-08-26T20:26:09.9345831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9345918Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9346155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9346243Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9346479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9346546Z return func(*args, **kwargs) 2025-08-26T20:26:09.9346794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9346890Z self_outputs = self.self( 2025-08-26T20:26:09.9347136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9347207Z return func(*args, **kwargs) 2025-08-26T20:26:09.9347454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-26T20:26:09.9347527Z self.value(current_states) 2025-08-26T20:26:09.9347531Z 2025-08-26T20:26:09.9347611Z cudagraph partition due to non gpu ops 2025-08-26T20:26:09.9347719Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9347917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9347989Z return mod(**inputs) 2025-08-26T20:26:09.9348234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9348299Z outputs = self.bert( 2025-08-26T20:26:09.9348551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9348639Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9348891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9348961Z layer_outputs = layer_module( 2025-08-26T20:26:09.9349178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9349261Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9349517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9349602Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9349835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9349913Z return func(*args, **kwargs) 2025-08-26T20:26:09.9350155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-26T20:26:09.9350227Z self_outputs = self.self( 2025-08-26T20:26:09.9350468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9350535Z return func(*args, **kwargs) 2025-08-26T20:26:09.9350780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-26T20:26:09.9350911Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:09.9350914Z 2025-08-26T20:26:09.9351017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9351219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9351283Z return mod(**inputs) 2025-08-26T20:26:09.9351537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9351604Z outputs = self.bert( 2025-08-26T20:26:09.9351846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9351925Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9352165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9352242Z layer_outputs = layer_module( 2025-08-26T20:26:09.9352464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9352547Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9352794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-26T20:26:09.9352910Z self_attention_outputs = self.attention( 2025-08-26T20:26:09.9353161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:26:09.9353230Z return func(*args, **kwargs) 2025-08-26T20:26:09.9353477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-26T20:26:09.9353607Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:26:09.9353855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-26T20:26:09.9353950Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9353954Z 2025-08-26T20:26:09.9354057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9354265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9354336Z return mod(**inputs) 2025-08-26T20:26:09.9354592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9354679Z outputs = self.bert( 2025-08-26T20:26:09.9354931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9355014Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9355259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9355337Z layer_outputs = layer_module( 2025-08-26T20:26:09.9355591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9355673Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9355950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9356041Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9356333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9356415Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9356709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9356849Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9357113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-26T20:26:09.9357213Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9357217Z 2025-08-26T20:26:09.9357326Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9357548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9357619Z return mod(**inputs) 2025-08-26T20:26:09.9357886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9357975Z outputs = self.bert( 2025-08-26T20:26:09.9358242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9358327Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9358591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9358669Z layer_outputs = layer_module( 2025-08-26T20:26:09.9358912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9358996Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9359576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9359679Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9359989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9360073Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9360376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-26T20:26:09.9360514Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:26:09.9360781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-26T20:26:09.9360910Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:26:09.9361139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:09.9361218Z return self.act(input) 2025-08-26T20:26:09.9361222Z 2025-08-26T20:26:09.9361342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9361574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9361651Z return mod(**inputs) 2025-08-26T20:26:09.9361918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-26T20:26:09.9361989Z outputs = self.bert( 2025-08-26T20:26:09.9362261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-26T20:26:09.9362357Z encoder_outputs = self.encoder( 2025-08-26T20:26:09.9362626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-26T20:26:09.9362703Z layer_outputs = layer_module( 2025-08-26T20:26:09.9362949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:09.9363034Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:09.9363300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-26T20:26:09.9363396Z layer_output = apply_chunking_to_forward( 2025-08-26T20:26:09.9363673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:26:09.9363765Z return forward_fn(*input_tensors) 2025-08-26T20:26:09.9364063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-26T20:26:09.9364207Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:26:09.9364477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-26T20:26:09.9364566Z hidden_states = self.dense(hidden_states) 2025-08-26T20:26:09.9364570Z 2025-08-26T20:26:09.9364689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9364900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9364977Z return mod(**inputs) 2025-08-26T20:26:09.9365245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1781, in forward 2025-08-26T20:26:09.9365335Z logits = self.qa_outputs(sequence_output) 2025-08-26T20:26:09.9365341Z 2025-08-26T20:26:09.9365458Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9365672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9365748Z return mod(**inputs) 2025-08-26T20:26:09.9366055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1799, in forward 2025-08-26T20:26:09.9366170Z start_loss = loss_fct(start_logits, start_positions) 2025-08-26T20:26:09.9366175Z 2025-08-26T20:26:09.9366290Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:09.9366498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:09.9366575Z return mod(**inputs) 2025-08-26T20:26:09.9366841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1800, in forward 2025-08-26T20:26:09.9366945Z end_loss = loss_fct(end_logits, end_positions) 2025-08-26T20:26:09.9366950Z 2025-08-26T20:26:17.6877961Z Compilation time (from dynamo_timed): 14.35831903 2025-08-26T20:26:17.6878268Z pass 2025-08-26T20:26:17.6878591Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:17.6879562Z TIMING: _recursive_pre_grad_passes:0.00725 _recursive_joint_graph_passes:0.37822 _recursive_post_grad_passes:0.08081 async_compile.wait:0.00215 code_gen:7.07483 inductor_compile:8.34351 backend_compile:11.57186 gc:0.00127 entire_frame_compile:14.35832 total_wall_time:14.35832 2025-08-26T20:26:17.6880857Z STATS: call_* op count: 296 | FakeTensorMode.__torch_dispatch__:12365 | FakeTensor.__torch_dispatch__:4381 | ProxyTorchDispatchMode.__torch_dispatch__:4531 2025-08-26T20:26:17.6883848Z Dynamo produced 1 graphs covering 296 ops with 0 graph breaks (0 unique) 2025-08-26T20:26:23.0292947Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:26:23.0294309Z from pkg_resources import resource_filename 2025-08-26T20:26:23.6084162Z 2025-08-26T20:26:43.1701956Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:26:43.1703525Z loading model: 0it [00:19, ?it/s] 2025-08-26T20:26:43.1736383Z cpu eval BlenderbotForCausalLM 2025-08-26T20:26:43.3807159Z Compilation time (from dynamo_timed): 0 2025-08-26T20:26:43.3807494Z pass_due_to_skip 2025-08-26T20:26:43.3815190Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:43.3815593Z TIMING: total_wall_time:0 2025-08-26T20:26:43.3815827Z STATS: call_* op count: 0 2025-08-26T20:26:43.3816140Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-08-26T20:26:48.3311523Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:26:48.3315833Z from pkg_resources import resource_filename 2025-08-26T20:26:48.9380664Z 2025-08-26T20:26:49.8393629Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:26:49.8393978Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:26:49.8406981Z cpu eval BlenderbotSmallForCausalLM 2025-08-26T20:26:50.0078361Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:50.0617797Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:50.1133559Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:26:56.1918713Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1919091Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1919498Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1919744Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1919981Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1920218Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1920863Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1921100Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1921370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1921799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1922167Z return mod(**inputs) 2025-08-26T20:26:56.1922684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1923203Z outputs = self.model.decoder( 2025-08-26T20:26:56.1923717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1924237Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1924611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1924982Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1925435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1926029Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1926546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.1927215Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.1927518Z 2025-08-26T20:26:56.1927698Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1928081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1928419Z return mod(**inputs) 2025-08-26T20:26:56.1928854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1929337Z outputs = self.model.decoder( 2025-08-26T20:26:56.1929814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1930261Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1930623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1930992Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1931445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1931921Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1932394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.1932867Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.1933014Z 2025-08-26T20:26:56.1933130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1933516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1933853Z return mod(**inputs) 2025-08-26T20:26:56.1934278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1934759Z outputs = self.model.decoder( 2025-08-26T20:26:56.1935255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1935707Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1936825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1937940Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1938948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1939511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1940067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.1940570Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.1940732Z 2025-08-26T20:26:56.1940835Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1941084Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1941310Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1941532Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1941794Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1942251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1942786Z return mod(**inputs) 2025-08-26T20:26:56.1943256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1943823Z outputs = self.model.decoder( 2025-08-26T20:26:56.1944403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1944888Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1945278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1945735Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1946225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1946783Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1947433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.1947996Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.1948529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.1949327Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.1949561Z 2025-08-26T20:26:56.1949698Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1950170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1950527Z return mod(**inputs) 2025-08-26T20:26:56.1951007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1951497Z outputs = self.model.decoder( 2025-08-26T20:26:56.1951992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1952464Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1952848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1953251Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1953730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1954236Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1954724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.1955222Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.1955750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.1956252Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.1956428Z 2025-08-26T20:26:56.1956550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1956932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1957294Z return mod(**inputs) 2025-08-26T20:26:56.1957749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1958239Z outputs = self.model.decoder( 2025-08-26T20:26:56.1958706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1959197Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1959939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1960361Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1960892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1961386Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1961897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.1962446Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.1962594Z 2025-08-26T20:26:56.1962718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1963168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1963563Z return mod(**inputs) 2025-08-26T20:26:56.1964033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1964516Z outputs = self.model.decoder( 2025-08-26T20:26:56.1964983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1965460Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1965831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1966230Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1966707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.1967245Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.1967449Z 2025-08-26T20:26:56.1967572Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1967954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1968316Z return mod(**inputs) 2025-08-26T20:26:56.1968792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1969265Z outputs = self.model.decoder( 2025-08-26T20:26:56.1969722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1970200Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1970576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1970979Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1971502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.1972016Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.1972438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.1972813Z return self.act(input) 2025-08-26T20:26:56.1972933Z 2025-08-26T20:26:56.1973052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1973448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1973795Z return mod(**inputs) 2025-08-26T20:26:56.1974250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1974726Z outputs = self.model.decoder( 2025-08-26T20:26:56.1975195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1975664Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1976068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1976467Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1976956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.1977447Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.1977606Z 2025-08-26T20:26:56.1977744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1978138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1978500Z return mod(**inputs) 2025-08-26T20:26:56.1978969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1979461Z outputs = self.model.decoder( 2025-08-26T20:26:56.1979943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1980426Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1980800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1981188Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1981666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1982180Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1982688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.1983262Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.1983494Z 2025-08-26T20:26:56.1983617Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1984011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1984363Z return mod(**inputs) 2025-08-26T20:26:56.1984826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1985323Z outputs = self.model.decoder( 2025-08-26T20:26:56.1985808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1986283Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1986673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1987116Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1987616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1988145Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1988657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.1989164Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.1989325Z 2025-08-26T20:26:56.1989447Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1989854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1990227Z return mod(**inputs) 2025-08-26T20:26:56.1990695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1991189Z outputs = self.model.decoder( 2025-08-26T20:26:56.1991695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1992180Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1992563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.1992965Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.1993458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.1993991Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.1994495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.1994995Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.1995156Z 2025-08-26T20:26:56.1995246Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1995486Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1995721Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1995948Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.1996416Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.1996848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.1997213Z return mod(**inputs) 2025-08-26T20:26:56.1997684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.1998175Z outputs = self.model.decoder( 2025-08-26T20:26:56.1998663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.1999165Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.1999686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2000112Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2000612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2001114Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2001614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2002117Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2002593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.2003242Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.2003457Z 2025-08-26T20:26:56.2003571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2003966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2004320Z return mod(**inputs) 2025-08-26T20:26:56.2004771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2005249Z outputs = self.model.decoder( 2025-08-26T20:26:56.2005719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2006193Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2006570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2006961Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2007447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2008024Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2008549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2009043Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2009513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.2010044Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.2010233Z 2025-08-26T20:26:56.2010343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2010736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2011090Z return mod(**inputs) 2025-08-26T20:26:56.2011548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2012047Z outputs = self.model.decoder( 2025-08-26T20:26:56.2012524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2013010Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2013396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2013792Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2014270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2014782Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2015253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.2015720Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.2015876Z 2025-08-26T20:26:56.2015990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2016379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2016728Z return mod(**inputs) 2025-08-26T20:26:56.2017173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2017634Z outputs = self.model.decoder( 2025-08-26T20:26:56.2018075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2018518Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2018927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2019296Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2019739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2020233Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2020417Z 2025-08-26T20:26:56.2020523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2020893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2021241Z return mod(**inputs) 2025-08-26T20:26:56.2021684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2022150Z outputs = self.model.decoder( 2025-08-26T20:26:56.2022599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2023125Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2023477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2023849Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2024300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2024825Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2025229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.2025593Z return self.act(input) 2025-08-26T20:26:56.2025721Z 2025-08-26T20:26:56.2025830Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2026210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2026553Z return mod(**inputs) 2025-08-26T20:26:56.2026986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2027446Z outputs = self.model.decoder( 2025-08-26T20:26:56.2027918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2028398Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2028795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2029175Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2029656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.2030150Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.2030308Z 2025-08-26T20:26:56.2030451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2030854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2031198Z return mod(**inputs) 2025-08-26T20:26:56.2031662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2032145Z outputs = self.model.decoder( 2025-08-26T20:26:56.2032624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2033105Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2033482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2033943Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2034432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2034944Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2035451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.2036011Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.2036249Z 2025-08-26T20:26:56.2036366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2036762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2037118Z return mod(**inputs) 2025-08-26T20:26:56.2037577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2038056Z outputs = self.model.decoder( 2025-08-26T20:26:56.2038573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2039047Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2039509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2039932Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2040433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2040983Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2041581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.2042148Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.2042339Z 2025-08-26T20:26:56.2042641Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2043120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2043569Z return mod(**inputs) 2025-08-26T20:26:56.2044174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2044693Z outputs = self.model.decoder( 2025-08-26T20:26:56.2045240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2045806Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2046261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2046736Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2047273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2047820Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2048351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.2048921Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.2049088Z 2025-08-26T20:26:56.2049227Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2049478Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2049821Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2050107Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2050423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2051629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2052044Z return mod(**inputs) 2025-08-26T20:26:56.2052562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2053106Z outputs = self.model.decoder( 2025-08-26T20:26:56.2053623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2054134Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2054587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2055082Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2055687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2056224Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2056734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2068678Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2069159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.2069658Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.2069860Z 2025-08-26T20:26:56.2070019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2070525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2070879Z return mod(**inputs) 2025-08-26T20:26:56.2071356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2071855Z outputs = self.model.decoder( 2025-08-26T20:26:56.2072317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2072767Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2073140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2073536Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2074027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2074536Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2075046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2075569Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2076062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.2076582Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.2076760Z 2025-08-26T20:26:56.2076886Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2077275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2077637Z return mod(**inputs) 2025-08-26T20:26:56.2078095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2078589Z outputs = self.model.decoder( 2025-08-26T20:26:56.2079064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2079701Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2080093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2080494Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2080983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2081497Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2082001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.2082497Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.2082657Z 2025-08-26T20:26:56.2082777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2083178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2083530Z return mod(**inputs) 2025-08-26T20:26:56.2083994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2084488Z outputs = self.model.decoder( 2025-08-26T20:26:56.2084954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2085421Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2085791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2086207Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2086679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2087202Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2087395Z 2025-08-26T20:26:56.2087516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2087900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2088246Z return mod(**inputs) 2025-08-26T20:26:56.2088693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2089169Z outputs = self.model.decoder( 2025-08-26T20:26:56.2089630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2090105Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2090482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2090873Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2091349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2091863Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2092280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.2092649Z return self.act(input) 2025-08-26T20:26:56.2092768Z 2025-08-26T20:26:56.2092890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2093279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2093636Z return mod(**inputs) 2025-08-26T20:26:56.2094062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2094511Z outputs = self.model.decoder( 2025-08-26T20:26:56.2094985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2095472Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2095843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2096387Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2096874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.2097355Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.2097508Z 2025-08-26T20:26:56.2097627Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2097987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2098308Z return mod(**inputs) 2025-08-26T20:26:56.2098722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2099240Z outputs = self.model.decoder( 2025-08-26T20:26:56.2099664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2100098Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2100439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2100795Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2101266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2101717Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2102178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.2102696Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.2102907Z 2025-08-26T20:26:56.2103020Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2103380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2103699Z return mod(**inputs) 2025-08-26T20:26:56.2104123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2104570Z outputs = self.model.decoder( 2025-08-26T20:26:56.2105008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2105445Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2105795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2106180Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2106655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2107152Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2107648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.2108119Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.2108265Z 2025-08-26T20:26:56.2108370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2108734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2109062Z return mod(**inputs) 2025-08-26T20:26:56.2109526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2109983Z outputs = self.model.decoder( 2025-08-26T20:26:56.2110429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2110898Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2111273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2111659Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2112136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2112636Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2113127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.2113622Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.2113776Z 2025-08-26T20:26:56.2113887Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2114121Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2114348Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2114571Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2114815Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2115200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2115546Z return mod(**inputs) 2025-08-26T20:26:56.2116037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2116501Z outputs = self.model.decoder( 2025-08-26T20:26:56.2116973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2117454Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2117842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2118242Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2118722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2119302Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2119823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2120320Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2120790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.2121301Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.2121506Z 2025-08-26T20:26:56.2121619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2122002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2122353Z return mod(**inputs) 2025-08-26T20:26:56.2122799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2123265Z outputs = self.model.decoder( 2025-08-26T20:26:56.2123731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2124202Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2124582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2124969Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2125407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2125863Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2126317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2126773Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2127216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.2127704Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.2127882Z 2025-08-26T20:26:56.2127994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2128380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2128728Z return mod(**inputs) 2025-08-26T20:26:56.2129194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2129627Z outputs = self.model.decoder( 2025-08-26T20:26:56.2130059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2130494Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2130841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2131211Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2131645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2132104Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2132557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.2133001Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.2133138Z 2025-08-26T20:26:56.2133243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2133603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2133923Z return mod(**inputs) 2025-08-26T20:26:56.2134332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2134762Z outputs = self.model.decoder( 2025-08-26T20:26:56.2135188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2135623Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2135971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2136333Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2136765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2137252Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2137434Z 2025-08-26T20:26:56.2137540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2137909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2138242Z return mod(**inputs) 2025-08-26T20:26:56.2138661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2139143Z outputs = self.model.decoder( 2025-08-26T20:26:56.2139585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2140035Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2140378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2140728Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2141158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2141633Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2142014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.2142350Z return self.act(input) 2025-08-26T20:26:56.2142469Z 2025-08-26T20:26:56.2142573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2142928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2143271Z return mod(**inputs) 2025-08-26T20:26:56.2143685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2144118Z outputs = self.model.decoder( 2025-08-26T20:26:56.2144545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2144994Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2145336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2145703Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2146146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.2146598Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.2146747Z 2025-08-26T20:26:56.2146852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2147217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2147539Z return mod(**inputs) 2025-08-26T20:26:56.2147963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2148414Z outputs = self.model.decoder( 2025-08-26T20:26:56.2148855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2149298Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2149654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2150019Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2150465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2150934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2151402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.2151917Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.2152134Z 2025-08-26T20:26:56.2152240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2152603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2152933Z return mod(**inputs) 2025-08-26T20:26:56.2153390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2153837Z outputs = self.model.decoder( 2025-08-26T20:26:56.2154276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2154718Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2155070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2155433Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2155883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2156354Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2156828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.2157278Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.2157435Z 2025-08-26T20:26:56.2157541Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2157905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2158231Z return mod(**inputs) 2025-08-26T20:26:56.2158649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2159125Z outputs = self.model.decoder( 2025-08-26T20:26:56.2159689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2160177Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2160575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2160980Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2161435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2161902Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2162373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.2162835Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.2162985Z 2025-08-26T20:26:56.2163076Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2163290Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2163510Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2163727Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2163965Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2164334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2164669Z return mod(**inputs) 2025-08-26T20:26:56.2165120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2165593Z outputs = self.model.decoder( 2025-08-26T20:26:56.2166061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2166533Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2166891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2167258Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2167709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2168226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2168720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2169227Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2169702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.2170218Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.2170419Z 2025-08-26T20:26:56.2170537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2170918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2171267Z return mod(**inputs) 2025-08-26T20:26:56.2171719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2172198Z outputs = self.model.decoder( 2025-08-26T20:26:56.2172699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2173167Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2173544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2173936Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2174415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2174926Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2175426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2175904Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2176342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.2176816Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.2176975Z 2025-08-26T20:26:56.2177079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2177434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2177756Z return mod(**inputs) 2025-08-26T20:26:56.2178173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2178609Z outputs = self.model.decoder( 2025-08-26T20:26:56.2179034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2179468Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2179816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2180184Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2180643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2181095Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2181552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.2182025Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.2182171Z 2025-08-26T20:26:56.2182289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2182714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2183060Z return mod(**inputs) 2025-08-26T20:26:56.2183507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2183964Z outputs = self.model.decoder( 2025-08-26T20:26:56.2184403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2184837Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2185191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2185561Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2186035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2186544Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2186718Z 2025-08-26T20:26:56.2186824Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2187211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2187545Z return mod(**inputs) 2025-08-26T20:26:56.2187951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2188391Z outputs = self.model.decoder( 2025-08-26T20:26:56.2188810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2189268Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2189615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2189982Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2190437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2190914Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2191294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.2191636Z return self.act(input) 2025-08-26T20:26:56.2191750Z 2025-08-26T20:26:56.2191863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2192220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2192555Z return mod(**inputs) 2025-08-26T20:26:56.2192982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2193462Z outputs = self.model.decoder( 2025-08-26T20:26:56.2193945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2194518Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2194894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2195288Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2195767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.2196407Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.2196564Z 2025-08-26T20:26:56.2196677Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2197065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2197417Z return mod(**inputs) 2025-08-26T20:26:56.2197946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2198432Z outputs = self.model.decoder( 2025-08-26T20:26:56.2198894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2199441Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2199819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2200216Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2200697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2201169Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2201641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.2202203Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.2202451Z 2025-08-26T20:26:56.2202573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2202952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2203299Z return mod(**inputs) 2025-08-26T20:26:56.2203752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2204269Z outputs = self.model.decoder( 2025-08-26T20:26:56.2204721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2205167Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2205522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2205892Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2206341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2206810Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2207295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.2207767Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.2207912Z 2025-08-26T20:26:56.2208017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2208378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2208705Z return mod(**inputs) 2025-08-26T20:26:56.2209120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2209573Z outputs = self.model.decoder( 2025-08-26T20:26:56.2210012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2210455Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2210800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2211172Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2211619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2212090Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2212592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.2213050Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.2213220Z 2025-08-26T20:26:56.2213303Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2213521Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2213733Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2213934Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2214170Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2214537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2214867Z return mod(**inputs) 2025-08-26T20:26:56.2215323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2215798Z outputs = self.model.decoder( 2025-08-26T20:26:56.2216276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2216751Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2217156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2217512Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2217956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2218423Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2218935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2219423Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2219892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.2220426Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.2220628Z 2025-08-26T20:26:56.2220733Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2221096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2221425Z return mod(**inputs) 2025-08-26T20:26:56.2221842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2222290Z outputs = self.model.decoder( 2025-08-26T20:26:56.2222731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2223174Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2223528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2223886Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2224341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2224797Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2225265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2225729Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2226172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.2226637Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.2226808Z 2025-08-26T20:26:56.2226913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2227322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2227650Z return mod(**inputs) 2025-08-26T20:26:56.2228100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2228740Z outputs = self.model.decoder( 2025-08-26T20:26:56.2229291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2229811Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2230160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2230542Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2231017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2231518Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2232021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.2232549Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.2232710Z 2025-08-26T20:26:56.2232823Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2233222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2233585Z return mod(**inputs) 2025-08-26T20:26:56.2234095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2234578Z outputs = self.model.decoder( 2025-08-26T20:26:56.2235062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2235546Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2235930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2236323Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2236810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2237345Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2237544Z 2025-08-26T20:26:56.2237659Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2238058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2238409Z return mod(**inputs) 2025-08-26T20:26:56.2238889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2239675Z outputs = self.model.decoder( 2025-08-26T20:26:56.2240181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2240675Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2241040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2241463Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2241909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2242398Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2242808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.2243181Z return self.act(input) 2025-08-26T20:26:56.2243313Z 2025-08-26T20:26:56.2243483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2243889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2244248Z return mod(**inputs) 2025-08-26T20:26:56.2244702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2245199Z outputs = self.model.decoder( 2025-08-26T20:26:56.2245693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2246198Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2246589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2246990Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2247486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.2248004Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.2248157Z 2025-08-26T20:26:56.2248280Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2248679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2249031Z return mod(**inputs) 2025-08-26T20:26:56.2249489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2250015Z outputs = self.model.decoder( 2025-08-26T20:26:56.2250489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2250973Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2251355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2251746Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2252179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2252630Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2253072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.2253577Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.2253784Z 2025-08-26T20:26:56.2253887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2254241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2254562Z return mod(**inputs) 2025-08-26T20:26:56.2254982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2255430Z outputs = self.model.decoder( 2025-08-26T20:26:56.2255866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2256309Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2256662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2257015Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2257459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2257927Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2258425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.2258890Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.2259028Z 2025-08-26T20:26:56.2259134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2259499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2259825Z return mod(**inputs) 2025-08-26T20:26:56.2260251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2260689Z outputs = self.model.decoder( 2025-08-26T20:26:56.2261146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2261613Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2261990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2262378Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2262862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2263357Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2263847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.2264325Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.2264497Z 2025-08-26T20:26:56.2264594Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2264827Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2265063Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2265292Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2265554Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2265947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2266319Z return mod(**inputs) 2025-08-26T20:26:56.2266787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2267261Z outputs = self.model.decoder( 2025-08-26T20:26:56.2267723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2268206Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2268593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2268991Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2269485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2269999Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2270506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2271019Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2271508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.2272040Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.2272244Z 2025-08-26T20:26:56.2272365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2272755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2273117Z return mod(**inputs) 2025-08-26T20:26:56.2273620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2274119Z outputs = self.model.decoder( 2025-08-26T20:26:56.2274602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2275101Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2275486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2275884Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2276371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2276871Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2277380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2277892Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2278396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.2278898Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.2279076Z 2025-08-26T20:26:56.2279189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2279749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2280154Z return mod(**inputs) 2025-08-26T20:26:56.2280622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2281128Z outputs = self.model.decoder( 2025-08-26T20:26:56.2281624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2282114Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2282503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2282915Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2283397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2283917Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2284414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.2284898Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.2285043Z 2025-08-26T20:26:56.2285163Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2285557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2285919Z return mod(**inputs) 2025-08-26T20:26:56.2286385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2286881Z outputs = self.model.decoder( 2025-08-26T20:26:56.2287370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2287859Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2288239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2288633Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2289109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2289681Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2289870Z 2025-08-26T20:26:56.2289985Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2290375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2290723Z return mod(**inputs) 2025-08-26T20:26:56.2291185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2291669Z outputs = self.model.decoder( 2025-08-26T20:26:56.2292143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2292628Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2293020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2293426Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2293906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2294455Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2294883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.2295259Z return self.act(input) 2025-08-26T20:26:56.2295381Z 2025-08-26T20:26:56.2295502Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2295919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2296482Z return mod(**inputs) 2025-08-26T20:26:56.2297090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2297608Z outputs = self.model.decoder( 2025-08-26T20:26:56.2298084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2298562Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2298941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2299326Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2299774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.2300226Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.2300374Z 2025-08-26T20:26:56.2300478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2300843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2301173Z return mod(**inputs) 2025-08-26T20:26:56.2301601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2302045Z outputs = self.model.decoder( 2025-08-26T20:26:56.2302486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2302949Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2303332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2303701Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2304142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2304634Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2305210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:26:56.2305730Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:26:56.2305934Z 2025-08-26T20:26:56.2306048Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2306404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2306732Z return mod(**inputs) 2025-08-26T20:26:56.2307157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2307607Z outputs = self.model.decoder( 2025-08-26T20:26:56.2308035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2308475Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2308832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2309239Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2309685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2310148Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2310616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:26:56.2311096Z key_states = self.k_proj(current_states) 2025-08-26T20:26:56.2311233Z 2025-08-26T20:26:56.2311347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2311710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2312038Z return mod(**inputs) 2025-08-26T20:26:56.2312463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2312910Z outputs = self.model.decoder( 2025-08-26T20:26:56.2313382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2313859Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2314227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2314628Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2315124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2315635Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2316137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:26:56.2316640Z value_states = self.v_proj(current_states) 2025-08-26T20:26:56.2316803Z 2025-08-26T20:26:56.2316895Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2317136Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2317372Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2317592Z cudagraph partition due to non gpu ops 2025-08-26T20:26:56.2317849Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2318245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2318608Z return mod(**inputs) 2025-08-26T20:26:56.2319064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2319636Z outputs = self.model.decoder( 2025-08-26T20:26:56.2320186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2320699Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2321091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2321495Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2321987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2322505Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2323015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2323533Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2324016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:26:56.2324548Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:26:56.2324800Z 2025-08-26T20:26:56.2324916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2325318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2325685Z return mod(**inputs) 2025-08-26T20:26:56.2326143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2326675Z outputs = self.model.decoder( 2025-08-26T20:26:56.2327161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2327658Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2328039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2328458Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2328936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2329417Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2329880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:26:56.2330340Z attn_output, attn_weights = attention_interface( 2025-08-26T20:26:56.2330789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:26:56.2331250Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:26:56.2331412Z 2025-08-26T20:26:56.2331526Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2331897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2332222Z return mod(**inputs) 2025-08-26T20:26:56.2332645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2333091Z outputs = self.model.decoder( 2025-08-26T20:26:56.2333531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2333977Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2334348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2334745Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2335261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:26:56.2335756Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:26:56.2336235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:26:56.2336685Z attn_output = self.out_proj(attn_output) 2025-08-26T20:26:56.2336834Z 2025-08-26T20:26:56.2336940Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2337301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2337628Z return mod(**inputs) 2025-08-26T20:26:56.2338043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2338489Z outputs = self.model.decoder( 2025-08-26T20:26:56.2338929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2339372Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2339750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2340108Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2340579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2341093Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2341267Z 2025-08-26T20:26:56.2341407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2341766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2342086Z return mod(**inputs) 2025-08-26T20:26:56.2342507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2342957Z outputs = self.model.decoder( 2025-08-26T20:26:56.2343397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2343831Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2344186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2344551Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2344999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:26:56.2345485Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:26:56.2345872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:26:56.2346224Z return self.act(input) 2025-08-26T20:26:56.2346345Z 2025-08-26T20:26:56.2346450Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2346829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2347152Z return mod(**inputs) 2025-08-26T20:26:56.2347559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-26T20:26:56.2348002Z outputs = self.model.decoder( 2025-08-26T20:26:56.2348463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:26:56.2348940Z layer_outputs = decoder_layer( 2025-08-26T20:26:56.2349313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:26:56.2349691Z return super().__call__(*args, **kwargs) 2025-08-26T20:26:56.2350204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:26:56.2350658Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:26:56.2350798Z 2025-08-26T20:26:56.2350910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2351266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2351594Z return mod(**inputs) 2025-08-26T20:26:56.2352019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1528, in forward 2025-08-26T20:26:56.2352474Z logits = self.lm_head(outputs[0]) 2025-08-26T20:26:56.2352609Z 2025-08-26T20:26:56.2352722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:26:56.2353089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:26:56.2353441Z return mod(**inputs) 2025-08-26T20:26:56.2353888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1534, in forward 2025-08-26T20:26:56.2354455Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:26:56.2354665Z 2025-08-26T20:27:04.0600282Z Compilation time (from dynamo_timed): 12.760721581 2025-08-26T20:27:04.0627616Z pass 2025-08-26T20:27:04.0628092Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:04.0629132Z TIMING: _recursive_pre_grad_passes:0.0067 _recursive_joint_graph_passes:0.30939 _recursive_post_grad_passes:0.06177 async_compile.wait:0.78999 code_gen:7.54212 inductor_compile:8.79578 backend_compile:11.09161 gc:0.00143 entire_frame_compile:12.76072 total_wall_time:12.76072 2025-08-26T20:27:04.0630367Z STATS: call_* op count: 252 | FakeTensorMode.__torch_dispatch__:9090 | FakeTensor.__torch_dispatch__:3104 | ProxyTorchDispatchMode.__torch_dispatch__:3279 2025-08-26T20:27:04.0630952Z Dynamo produced 1 graphs covering 252 ops with 0 graph breaks (0 unique) 2025-08-26T20:27:09.4020230Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:27:09.4021273Z from pkg_resources import resource_filename 2025-08-26T20:27:10.0221024Z 2025-08-26T20:27:11.1914773Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:27:11.1915313Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:27:11.1938078Z cpu eval BlenderbotSmallForConditionalGeneration 2025-08-26T20:27:11.4694896Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:11.5744150Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:11.6760551Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:23.9795324Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9795658Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9795940Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9796369Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9796623Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9796855Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9797103Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9797377Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9797654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9798088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9798479Z return mod(**inputs) 2025-08-26T20:27:23.9799639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9800204Z outputs = self.model( 2025-08-26T20:27:23.9800696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9801201Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9801700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9802197Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9802613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9803036Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9803530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9804046Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9804613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:23.9805185Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:23.9805437Z 2025-08-26T20:27:23.9805568Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9805983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9806422Z return mod(**inputs) 2025-08-26T20:27:23.9806908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9807409Z outputs = self.model( 2025-08-26T20:27:23.9807899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9808407Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9808899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9809390Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9809793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9810208Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9810708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9811206Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9811706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:23.9812191Z key_states = self.k_proj(current_states) 2025-08-26T20:27:23.9812343Z 2025-08-26T20:27:23.9812461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9812860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9813329Z return mod(**inputs) 2025-08-26T20:27:23.9813798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9814281Z outputs = self.model( 2025-08-26T20:27:23.9814739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9815272Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9815867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9816352Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9816744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9817161Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9817642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9818137Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9818629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:23.9819191Z value_states = self.v_proj(current_states) 2025-08-26T20:27:23.9819344Z 2025-08-26T20:27:23.9819431Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9819661Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9819888Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9820110Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9820379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9820768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9821125Z return mod(**inputs) 2025-08-26T20:27:23.9821586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9822065Z outputs = self.model( 2025-08-26T20:27:23.9822516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9823747Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9824214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9824695Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9825064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9825454Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9825938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9826434Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9826936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:23.9827452Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:23.9827933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:23.9828452Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:23.9828653Z 2025-08-26T20:27:23.9828774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9829163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9829515Z return mod(**inputs) 2025-08-26T20:27:23.9829954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9830394Z outputs = self.model( 2025-08-26T20:27:23.9830818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9831258Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9831689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9832172Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9832543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9832943Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9833420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9833914Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9834410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:23.9834918Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:23.9835418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:23.9835921Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:23.9836104Z 2025-08-26T20:27:23.9836221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9836633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9836983Z return mod(**inputs) 2025-08-26T20:27:23.9837426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9837881Z outputs = self.model( 2025-08-26T20:27:23.9838337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9838825Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9839378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9839866Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9840255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9840664Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9841162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9841648Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9842107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:23.9842559Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:23.9842714Z 2025-08-26T20:27:23.9842827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9843210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9843561Z return mod(**inputs) 2025-08-26T20:27:23.9844007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9844471Z outputs = self.model( 2025-08-26T20:27:23.9844907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9845365Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9845821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9846281Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9846652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9847038Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9847564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:23.9848084Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:23.9848273Z 2025-08-26T20:27:23.9848385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9848773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9849118Z return mod(**inputs) 2025-08-26T20:27:23.9849565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9850034Z outputs = self.model( 2025-08-26T20:27:23.9850477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9850946Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9851413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9851877Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9852287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9852678Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9853166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:23.9853722Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:23.9854161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:23.9854525Z return self.act(input) 2025-08-26T20:27:23.9854653Z 2025-08-26T20:27:23.9854765Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9855155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9855506Z return mod(**inputs) 2025-08-26T20:27:23.9855949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9856408Z outputs = self.model( 2025-08-26T20:27:23.9856863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9857342Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9857780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9858246Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9858613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9859000Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9859476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:23.9859956Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:23.9860103Z 2025-08-26T20:27:23.9860216Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9860602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9860946Z return mod(**inputs) 2025-08-26T20:27:23.9861401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9861866Z outputs = self.model( 2025-08-26T20:27:23.9862304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9862770Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9863266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9863738Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9864106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9864478Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9864925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9865391Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9865880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:23.9866440Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:23.9866667Z 2025-08-26T20:27:23.9866782Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9867171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9867541Z return mod(**inputs) 2025-08-26T20:27:23.9867988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9868439Z outputs = self.model( 2025-08-26T20:27:23.9868851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9869338Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9869798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9870262Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9870633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9871031Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9871510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9871986Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9872450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:23.9872898Z key_states = self.k_proj(current_states) 2025-08-26T20:27:23.9873054Z 2025-08-26T20:27:23.9873167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9873555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9873906Z return mod(**inputs) 2025-08-26T20:27:23.9874358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9874818Z outputs = self.model( 2025-08-26T20:27:23.9875267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9875745Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9876223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9876705Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9877090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9877492Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9878013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9878516Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9879023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:23.9879617Z value_states = self.v_proj(current_states) 2025-08-26T20:27:23.9879785Z 2025-08-26T20:27:23.9879877Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9880118Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9880351Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9880573Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9880841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9881238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9881579Z return mod(**inputs) 2025-08-26T20:27:23.9881996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9882434Z outputs = self.model( 2025-08-26T20:27:23.9882891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9883369Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9883841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9884307Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9884728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9885142Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9885644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9886160Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9886636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:23.9887107Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:23.9887564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:23.9888078Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:23.9888277Z 2025-08-26T20:27:23.9888399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9888778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9889124Z return mod(**inputs) 2025-08-26T20:27:23.9889583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9890063Z outputs = self.model( 2025-08-26T20:27:23.9890497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9890972Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9891436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9891906Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9892278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9892660Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9893131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9893655Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9894144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:23.9894638Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:23.9895107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:23.9895598Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:23.9895777Z 2025-08-26T20:27:23.9895888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9896403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9896767Z return mod(**inputs) 2025-08-26T20:27:23.9897213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9897691Z outputs = self.model( 2025-08-26T20:27:23.9898145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9898691Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9899156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9899618Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9899995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9900424Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9900910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9901398Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9901898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:23.9902387Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:23.9902537Z 2025-08-26T20:27:23.9902659Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9903054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9903404Z return mod(**inputs) 2025-08-26T20:27:23.9903858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9904340Z outputs = self.model( 2025-08-26T20:27:23.9904773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9905224Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9905666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9906115Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9906477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9906852Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9907308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:23.9907797Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:23.9907985Z 2025-08-26T20:27:23.9908095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9908483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9908837Z return mod(**inputs) 2025-08-26T20:27:23.9909345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9909815Z outputs = self.model( 2025-08-26T20:27:23.9910264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9910707Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9911140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9911575Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9911931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9912324Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9912801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:23.9913315Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:23.9913742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:23.9914109Z return self.act(input) 2025-08-26T20:27:23.9914233Z 2025-08-26T20:27:23.9914344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9914728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9915074Z return mod(**inputs) 2025-08-26T20:27:23.9915540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9916011Z outputs = self.model( 2025-08-26T20:27:23.9916471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9916954Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9917414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9917886Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9918261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9918659Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9919133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:23.9919684Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:23.9919844Z 2025-08-26T20:27:23.9919958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9920350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9920704Z return mod(**inputs) 2025-08-26T20:27:23.9921150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9921603Z outputs = self.model( 2025-08-26T20:27:23.9922030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9922517Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9922989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9923457Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9923837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9924234Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9924751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9925240Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9925721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:23.9926293Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:23.9926520Z 2025-08-26T20:27:23.9926632Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9927024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9927373Z return mod(**inputs) 2025-08-26T20:27:23.9927822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9928300Z outputs = self.model( 2025-08-26T20:27:23.9928747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9929255Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9929717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9930176Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9930555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9930963Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9931436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9931896Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9932352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:23.9932808Z key_states = self.k_proj(current_states) 2025-08-26T20:27:23.9932950Z 2025-08-26T20:27:23.9933056Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9933419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9933738Z return mod(**inputs) 2025-08-26T20:27:23.9934162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9934603Z outputs = self.model( 2025-08-26T20:27:23.9935029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9935475Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9935915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9936359Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9936713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9937078Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9937528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9937982Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9938442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:23.9938897Z value_states = self.v_proj(current_states) 2025-08-26T20:27:23.9939040Z 2025-08-26T20:27:23.9939131Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9939370Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9939588Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9939802Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9940043Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9940410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9940734Z return mod(**inputs) 2025-08-26T20:27:23.9941158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9941606Z outputs = self.model( 2025-08-26T20:27:23.9942033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9942475Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9942919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9943378Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9943737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9944083Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9944500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9944945Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9945397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:23.9945842Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:23.9946280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:23.9946750Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:23.9946942Z 2025-08-26T20:27:23.9947045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9947403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9947726Z return mod(**inputs) 2025-08-26T20:27:23.9948142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9948578Z outputs = self.model( 2025-08-26T20:27:23.9949009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9949456Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9949910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9950349Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9950681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9951037Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9951479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9951928Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9952369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:23.9952828Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:23.9953266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:23.9953753Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:23.9953915Z 2025-08-26T20:27:23.9954029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9954388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9954733Z return mod(**inputs) 2025-08-26T20:27:23.9955190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9955672Z outputs = self.model( 2025-08-26T20:27:23.9956133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9956609Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9957076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9957560Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9957938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9958354Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9958829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9959395Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9959908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:23.9960448Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:23.9960600Z 2025-08-26T20:27:23.9960716Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9961125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9961492Z return mod(**inputs) 2025-08-26T20:27:23.9961945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9962484Z outputs = self.model( 2025-08-26T20:27:23.9962938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9963422Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9963894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9964377Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9964751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9965140Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9965622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:23.9966152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:23.9966339Z 2025-08-26T20:27:23.9966457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9966843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9967186Z return mod(**inputs) 2025-08-26T20:27:23.9967648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9968118Z outputs = self.model( 2025-08-26T20:27:23.9968613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9969043Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9969546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9969982Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9970333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9970696Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9971129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:23.9971608Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:23.9971992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:23.9972329Z return self.act(input) 2025-08-26T20:27:23.9972437Z 2025-08-26T20:27:23.9972546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9972900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9973243Z return mod(**inputs) 2025-08-26T20:27:23.9973656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9974086Z outputs = self.model( 2025-08-26T20:27:23.9974493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9974929Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9975381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9975829Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9976184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9976551Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9977008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:23.9977448Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:23.9977585Z 2025-08-26T20:27:23.9977696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9978055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9978368Z return mod(**inputs) 2025-08-26T20:27:23.9978782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9979210Z outputs = self.model( 2025-08-26T20:27:23.9979624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9980063Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9980490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9980927Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9981276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9981643Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9982094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9982549Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9983011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:23.9983576Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:23.9983781Z 2025-08-26T20:27:23.9983891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9984243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9984567Z return mod(**inputs) 2025-08-26T20:27:23.9984981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9985411Z outputs = self.model( 2025-08-26T20:27:23.9985828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9986266Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9986709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9987153Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9987508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9987899Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9988341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9988803Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9989262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:23.9989731Z key_states = self.k_proj(current_states) 2025-08-26T20:27:23.9989868Z 2025-08-26T20:27:23.9989980Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9990336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9990668Z return mod(**inputs) 2025-08-26T20:27:23.9991088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9991527Z outputs = self.model( 2025-08-26T20:27:23.9991939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9992382Z encoder_outputs = self.encoder( 2025-08-26T20:27:23.9992816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:23.9993255Z layer_outputs = encoder_layer( 2025-08-26T20:27:23.9993607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:23.9993964Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:23.9994414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:23.9994877Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:23.9995337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:23.9995796Z value_states = self.v_proj(current_states) 2025-08-26T20:27:23.9995947Z 2025-08-26T20:27:23.9996038Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9996448Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9996682Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9996905Z cudagraph partition due to non gpu ops 2025-08-26T20:27:23.9997149Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:23.9997524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:23.9997857Z return mod(**inputs) 2025-08-26T20:27:23.9998369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:23.9998815Z outputs = self.model( 2025-08-26T20:27:23.9999308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:23.9999795Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0000263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0000740Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0001095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0001470Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0001924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0002421Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0002879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0003337Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0003794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0004274Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0004485Z 2025-08-26T20:27:24.0004596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0004957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0005278Z return mod(**inputs) 2025-08-26T20:27:24.0005709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0006155Z outputs = self.model( 2025-08-26T20:27:24.0006582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0007027Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0007460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0007904Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0008254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0008621Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0009070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0009517Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0009962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0010417Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0010852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0011299Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0011471Z 2025-08-26T20:27:24.0011578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0011941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0012270Z return mod(**inputs) 2025-08-26T20:27:24.0012735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0013178Z outputs = self.model( 2025-08-26T20:27:24.0013616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0014058Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0014498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0014945Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0015291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0015653Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0016094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0016555Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0017013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0017467Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0017611Z 2025-08-26T20:27:24.0017718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0018078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0018409Z return mod(**inputs) 2025-08-26T20:27:24.0018848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0019380Z outputs = self.model( 2025-08-26T20:27:24.0019876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0020325Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0020769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0021208Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0021566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0021934Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0022387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0022877Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0023055Z 2025-08-26T20:27:24.0023160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0023532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0023863Z return mod(**inputs) 2025-08-26T20:27:24.0024288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0024736Z outputs = self.model( 2025-08-26T20:27:24.0025153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0025600Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0026038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0026481Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0026831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0027249Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0027759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0028281Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0028696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0029082Z return self.act(input) 2025-08-26T20:27:24.0029200Z 2025-08-26T20:27:24.0029305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0030092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0030426Z return mod(**inputs) 2025-08-26T20:27:24.0030912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0031350Z outputs = self.model( 2025-08-26T20:27:24.0031802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0032299Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0032767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0033250Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0033631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0034033Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0034546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:24.0035055Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0035212Z 2025-08-26T20:27:24.0035339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0035744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0036116Z return mod(**inputs) 2025-08-26T20:27:24.0036592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0037090Z outputs = self.model( 2025-08-26T20:27:24.0037561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0038065Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0038565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0039061Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0039530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0039978Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0040496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0041010Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0041520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0042103Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0042334Z 2025-08-26T20:27:24.0042449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0042856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0043225Z return mod(**inputs) 2025-08-26T20:27:24.0043755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0044245Z outputs = self.model( 2025-08-26T20:27:24.0044712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0045210Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0045700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0046188Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0046574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0046989Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0047448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0047931Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0048423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0048892Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0049038Z 2025-08-26T20:27:24.0049143Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0049512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0049843Z return mod(**inputs) 2025-08-26T20:27:24.0050267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0050721Z outputs = self.model( 2025-08-26T20:27:24.0051146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0051597Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0052039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0052480Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0052827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0053196Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0053645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0054106Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0054563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0055013Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0055165Z 2025-08-26T20:27:24.0055249Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0055467Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0055681Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0055883Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0056119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0056558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0056987Z return mod(**inputs) 2025-08-26T20:27:24.0057407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0057855Z outputs = self.model( 2025-08-26T20:27:24.0058294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0058769Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0059274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0059744Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0060124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0060569Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0061024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0061492Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0061951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0062452Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0062938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0063479Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0063681Z 2025-08-26T20:27:24.0063800Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0064181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0064512Z return mod(**inputs) 2025-08-26T20:27:24.0064961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0065448Z outputs = self.model( 2025-08-26T20:27:24.0065893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0066361Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0066832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0067302Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0067676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0068053Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0068525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0069010Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0069501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0069996Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0070483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0070973Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0071150Z 2025-08-26T20:27:24.0071270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0071645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0071996Z return mod(**inputs) 2025-08-26T20:27:24.0072452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0072919Z outputs = self.model( 2025-08-26T20:27:24.0073376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0073851Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0074372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0074865Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0075244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0075647Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0076133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0076638Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0077154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0077658Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0077808Z 2025-08-26T20:27:24.0077924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0078331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0078712Z return mod(**inputs) 2025-08-26T20:27:24.0079181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0079804Z outputs = self.model( 2025-08-26T20:27:24.0080327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0080823Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0081333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0081772Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0082119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0082482Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0082938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0083437Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0083619Z 2025-08-26T20:27:24.0083732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0084103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0084437Z return mod(**inputs) 2025-08-26T20:27:24.0084872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0085309Z outputs = self.model( 2025-08-26T20:27:24.0085731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0086157Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0086592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0087026Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0087377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0087742Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0088175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0088663Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0089056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0089408Z return self.act(input) 2025-08-26T20:27:24.0089520Z 2025-08-26T20:27:24.0089677Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0090039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0090368Z return mod(**inputs) 2025-08-26T20:27:24.0090800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0091234Z outputs = self.model( 2025-08-26T20:27:24.0091654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0092100Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0092538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0092983Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0093340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0093727Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0094178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:24.0094632Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0094773Z 2025-08-26T20:27:24.0094885Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0095253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0095610Z return mod(**inputs) 2025-08-26T20:27:24.0096035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0096603Z outputs = self.model( 2025-08-26T20:27:24.0097045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0097495Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0097934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0098386Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0098762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0099165Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0099633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0100136Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0100631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0101156Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0101367Z 2025-08-26T20:27:24.0101482Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0101843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0102176Z return mod(**inputs) 2025-08-26T20:27:24.0102604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0103057Z outputs = self.model( 2025-08-26T20:27:24.0103487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0103929Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0104461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0104915Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0105281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0105658Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0106109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0106589Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0107058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0107513Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0107656Z 2025-08-26T20:27:24.0107769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0108147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0108512Z return mod(**inputs) 2025-08-26T20:27:24.0109012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0109494Z outputs = self.model( 2025-08-26T20:27:24.0109957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0110413Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0110923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0111403Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0111778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0112167Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0112651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0113154Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0113647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0114138Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0114290Z 2025-08-26T20:27:24.0114378Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0114611Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0114839Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0115064Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0115308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0115697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0116051Z return mod(**inputs) 2025-08-26T20:27:24.0116501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0116980Z outputs = self.model( 2025-08-26T20:27:24.0117434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0117908Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0118381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0118851Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0119274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0119693Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0120299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0120819Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0121320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0121900Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0122357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0122854Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0123045Z 2025-08-26T20:27:24.0123162Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0123533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0123862Z return mod(**inputs) 2025-08-26T20:27:24.0124291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0124771Z outputs = self.model( 2025-08-26T20:27:24.0125205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0125652Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0126088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0126556Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0126915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0127284Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0127725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0128187Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0128647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0129116Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0129565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0130021Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0130195Z 2025-08-26T20:27:24.0130300Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0130666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0130999Z return mod(**inputs) 2025-08-26T20:27:24.0131424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0131862Z outputs = self.model( 2025-08-26T20:27:24.0132292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0132747Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0133188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0133638Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0133984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0134354Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0134843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0135312Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0135779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0136226Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0136376Z 2025-08-26T20:27:24.0136483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0136850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0137180Z return mod(**inputs) 2025-08-26T20:27:24.0137595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0138034Z outputs = self.model( 2025-08-26T20:27:24.0138464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0138941Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0139378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0139809Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0140164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0140537Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0140996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0141474Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0141645Z 2025-08-26T20:27:24.0141749Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0142109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0142430Z return mod(**inputs) 2025-08-26T20:27:24.0142849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0143281Z outputs = self.model( 2025-08-26T20:27:24.0143699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0144143Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0144594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0145033Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0145379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0145751Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0146198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0146683Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0147082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0147448Z return self.act(input) 2025-08-26T20:27:24.0147576Z 2025-08-26T20:27:24.0147690Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0148088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0148437Z return mod(**inputs) 2025-08-26T20:27:24.0148884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0149380Z outputs = self.model( 2025-08-26T20:27:24.0149818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0150263Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0150697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0151130Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0151483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0151849Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0152317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:24.0152790Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0152937Z 2025-08-26T20:27:24.0153052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0153436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0153823Z return mod(**inputs) 2025-08-26T20:27:24.0154246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0154690Z outputs = self.model( 2025-08-26T20:27:24.0155105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0155572Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0156027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0156493Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0156873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0157258Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0157740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0158239Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0158730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0159386Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0159635Z 2025-08-26T20:27:24.0159752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0160163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0160523Z return mod(**inputs) 2025-08-26T20:27:24.0160973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0161418Z outputs = self.model( 2025-08-26T20:27:24.0161852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0162308Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0162770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0163255Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0163626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0164021Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0164597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0165092Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0165569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0166030Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0166185Z 2025-08-26T20:27:24.0166299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0166689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0167051Z return mod(**inputs) 2025-08-26T20:27:24.0167492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0167954Z outputs = self.model( 2025-08-26T20:27:24.0168512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0168971Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0169442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0169903Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0170286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0170653Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0171108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0171608Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0172094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0172573Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0172736Z 2025-08-26T20:27:24.0172825Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0173057Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0173282Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0173498Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0173749Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0174136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0174484Z return mod(**inputs) 2025-08-26T20:27:24.0174924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0175395Z outputs = self.model( 2025-08-26T20:27:24.0175850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0176318Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0176784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0177256Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0177630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0178029Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0178501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0181575Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0182101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0182645Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0183135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0183656Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0183855Z 2025-08-26T20:27:24.0183962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0184343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0184678Z return mod(**inputs) 2025-08-26T20:27:24.0185114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0185588Z outputs = self.model( 2025-08-26T20:27:24.0186014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0186461Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0186905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0187378Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0187736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0188107Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0188553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0189043Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0189540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0190033Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0190490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0190977Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0191160Z 2025-08-26T20:27:24.0191273Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0191665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0192018Z return mod(**inputs) 2025-08-26T20:27:24.0192471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0192937Z outputs = self.model( 2025-08-26T20:27:24.0193386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0193859Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0194332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0194798Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0195182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0195578Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0196058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0196677Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0197266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0197754Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0197910Z 2025-08-26T20:27:24.0198054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0198455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0198820Z return mod(**inputs) 2025-08-26T20:27:24.0199335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0199839Z outputs = self.model( 2025-08-26T20:27:24.0200305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0200789Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0201269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0201740Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0202127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0202572Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0203068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0203611Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0203812Z 2025-08-26T20:27:24.0203929Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0204333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0204769Z return mod(**inputs) 2025-08-26T20:27:24.0205235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0205705Z outputs = self.model( 2025-08-26T20:27:24.0206183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0206670Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0207146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0207624Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0207963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0208331Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0208786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0209276Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0209671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0210013Z return self.act(input) 2025-08-26T20:27:24.0210131Z 2025-08-26T20:27:24.0210239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0210604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0210940Z return mod(**inputs) 2025-08-26T20:27:24.0211341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0211771Z outputs = self.model( 2025-08-26T20:27:24.0212184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0212658Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0213087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0213548Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0213896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0214257Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0214689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:24.0215130Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0215266Z 2025-08-26T20:27:24.0215370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0215740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0216079Z return mod(**inputs) 2025-08-26T20:27:24.0216493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0216924Z outputs = self.model( 2025-08-26T20:27:24.0217357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0217814Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0218242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0218673Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0219024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0219406Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0219849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0220304Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0220763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0221278Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0221494Z 2025-08-26T20:27:24.0221601Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0221969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0222318Z return mod(**inputs) 2025-08-26T20:27:24.0222772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0223235Z outputs = self.model( 2025-08-26T20:27:24.0223688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0224156Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0224624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0225070Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0225434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0225818Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0226288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0226785Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0227304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0227807Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0227955Z 2025-08-26T20:27:24.0228083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0228477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0228844Z return mod(**inputs) 2025-08-26T20:27:24.0229300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0229785Z outputs = self.model( 2025-08-26T20:27:24.0230256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0230743Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0231211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0231690Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0232067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0232448Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0232964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0233456Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0233946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0234436Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0234613Z 2025-08-26T20:27:24.0234702Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0234938Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0235169Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0235390Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0235637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0236032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0236401Z return mod(**inputs) 2025-08-26T20:27:24.0236853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0237323Z outputs = self.model( 2025-08-26T20:27:24.0237777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0238248Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0238724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0239200Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0239668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0240083Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0240584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0241095Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0241608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0242105Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0242594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0243156Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0243362Z 2025-08-26T20:27:24.0243487Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0243913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0244277Z return mod(**inputs) 2025-08-26T20:27:24.0244762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0245275Z outputs = self.model( 2025-08-26T20:27:24.0245767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0246264Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0246744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0247233Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0247632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0248059Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0248571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0249094Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0249600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0250098Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0250575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0251084Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0251269Z 2025-08-26T20:27:24.0251381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0251770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0252125Z return mod(**inputs) 2025-08-26T20:27:24.0252575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0253034Z outputs = self.model( 2025-08-26T20:27:24.0253483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0253951Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0254415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0254894Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0255262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0255650Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0256117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-26T20:27:24.0256600Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:27:24.0257075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0257550Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0257701Z 2025-08-26T20:27:24.0257813Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0258200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0258546Z return mod(**inputs) 2025-08-26T20:27:24.0259015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0259505Z outputs = self.model( 2025-08-26T20:27:24.0259953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0260411Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0260879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0261346Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0261723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0262118Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0262602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0263124Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0263312Z 2025-08-26T20:27:24.0263426Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0263808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0264172Z return mod(**inputs) 2025-08-26T20:27:24.0264604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0265043Z outputs = self.model( 2025-08-26T20:27:24.0265476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0265966Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0266434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0266898Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0267267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0267658Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0268138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-26T20:27:24.0268663Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0269081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0269437Z return self.act(input) 2025-08-26T20:27:24.0269566Z 2025-08-26T20:27:24.0269679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0270069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0270418Z return mod(**inputs) 2025-08-26T20:27:24.0270867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0271331Z outputs = self.model( 2025-08-26T20:27:24.0271790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-26T20:27:24.0272231Z encoder_outputs = self.encoder( 2025-08-26T20:27:24.0272693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-26T20:27:24.0273158Z layer_outputs = encoder_layer( 2025-08-26T20:27:24.0273534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0273969Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0274459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-26T20:27:24.0274974Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0275129Z 2025-08-26T20:27:24.0275245Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0275648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0276007Z return mod(**inputs) 2025-08-26T20:27:24.0276471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0276952Z outputs = self.model( 2025-08-26T20:27:24.0277409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0277907Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0278391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0278883Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0279345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0279793Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0280287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0280811Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0281329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0281924Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0282162Z 2025-08-26T20:27:24.0282280Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0282685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0283053Z return mod(**inputs) 2025-08-26T20:27:24.0283526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0284013Z outputs = self.model( 2025-08-26T20:27:24.0284486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0284974Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0285453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0285941Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0286325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0286732Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0287222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0287746Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0288259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0288742Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0288895Z 2025-08-26T20:27:24.0289006Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0289402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0289755Z return mod(**inputs) 2025-08-26T20:27:24.0290237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0290706Z outputs = self.model( 2025-08-26T20:27:24.0291178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0291652Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0292115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0292574Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0292950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0293337Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0293811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0294311Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0294801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0295306Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0295466Z 2025-08-26T20:27:24.0295553Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0295787Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0296006Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0296346Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0296609Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0297003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0297417Z return mod(**inputs) 2025-08-26T20:27:24.0297863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0298330Z outputs = self.model( 2025-08-26T20:27:24.0298783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0299256Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0299718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0300195Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0300574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0300966Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0301446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0301942Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0302440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0302921Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0303373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0303862Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0304052Z 2025-08-26T20:27:24.0304157Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0304523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0304859Z return mod(**inputs) 2025-08-26T20:27:24.0305337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0305811Z outputs = self.model( 2025-08-26T20:27:24.0306285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0306762Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0307244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0307739Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0308093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0308456Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0308909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0309380Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0309851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0310318Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0310789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0311251Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0311425Z 2025-08-26T20:27:24.0311533Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0311896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0312243Z return mod(**inputs) 2025-08-26T20:27:24.0312688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0313157Z outputs = self.model( 2025-08-26T20:27:24.0313607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0314070Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0314529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0314998Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0315382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0315780Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0316268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0316775Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0317269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0317754Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0317907Z 2025-08-26T20:27:24.0318028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0318425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0318779Z return mod(**inputs) 2025-08-26T20:27:24.0319295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0319791Z outputs = self.model( 2025-08-26T20:27:24.0320253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0320762Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0321263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0321753Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0322146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0322552Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0323037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0323625Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0324149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0324728Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0324957Z 2025-08-26T20:27:24.0325083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0325479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0325845Z return mod(**inputs) 2025-08-26T20:27:24.0326295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0326739Z outputs = self.model( 2025-08-26T20:27:24.0327160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0327596Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0328039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0328518Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0328871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0329256Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0329727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0330237Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0330744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0331223Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0331369Z 2025-08-26T20:27:24.0331479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0331869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0332225Z return mod(**inputs) 2025-08-26T20:27:24.0332670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0333147Z outputs = self.model( 2025-08-26T20:27:24.0333589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0334063Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0334528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0334993Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0335349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0335724Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0336225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0336731Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0337255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0337739Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0337893Z 2025-08-26T20:27:24.0337992Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0338214Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0338427Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0338642Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0338884Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0339272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0339617Z return mod(**inputs) 2025-08-26T20:27:24.0340045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0340487Z outputs = self.model( 2025-08-26T20:27:24.0340916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0341384Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0341826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0342270Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0342633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0343059Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0343512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0343998Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0344492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0344982Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0345463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0345978Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0346177Z 2025-08-26T20:27:24.0346299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0346691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0347039Z return mod(**inputs) 2025-08-26T20:27:24.0347498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0347962Z outputs = self.model( 2025-08-26T20:27:24.0348416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0348887Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0349356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0349827Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0350202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0350590Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0351079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0351586Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0352111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0352610Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0353084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0353559Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0353737Z 2025-08-26T20:27:24.0353852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0354239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0354591Z return mod(**inputs) 2025-08-26T20:27:24.0355043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0355515Z outputs = self.model( 2025-08-26T20:27:24.0355960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0356467Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0356929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0357406Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0357772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0358161Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0358658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0359160Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0359944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0360441Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0360604Z 2025-08-26T20:27:24.0360722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0361108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0361443Z return mod(**inputs) 2025-08-26T20:27:24.0361860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0362302Z outputs = self.model( 2025-08-26T20:27:24.0362733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0363186Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0363634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0364075Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0364438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0364804Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0365248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0365738Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0365917Z 2025-08-26T20:27:24.0366021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0366404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0366739Z return mod(**inputs) 2025-08-26T20:27:24.0367204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0367663Z outputs = self.model( 2025-08-26T20:27:24.0368106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0368560Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0368999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0369446Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0369791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0370176Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0370648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0371166Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0371581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0371961Z return self.act(input) 2025-08-26T20:27:24.0372084Z 2025-08-26T20:27:24.0372196Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0372583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0372930Z return mod(**inputs) 2025-08-26T20:27:24.0373366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0373848Z outputs = self.model( 2025-08-26T20:27:24.0374302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0374768Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0375241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0375703Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0376078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0376469Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0376945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0377426Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0377574Z 2025-08-26T20:27:24.0377689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0378077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0378426Z return mod(**inputs) 2025-08-26T20:27:24.0378874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0379339Z outputs = self.model( 2025-08-26T20:27:24.0379778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0380250Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0380714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0381184Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0381577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0381970Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0382468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0382967Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0383463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0384011Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0384239Z 2025-08-26T20:27:24.0384353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0384739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0385091Z return mod(**inputs) 2025-08-26T20:27:24.0385544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0386020Z outputs = self.model( 2025-08-26T20:27:24.0386481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0386989Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0387456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0387935Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0388302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0388706Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0389207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0389711Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0390211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0390689Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0390847Z 2025-08-26T20:27:24.0390959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0391349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0391700Z return mod(**inputs) 2025-08-26T20:27:24.0392157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0392642Z outputs = self.model( 2025-08-26T20:27:24.0393108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0393596Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0394083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0394553Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0394939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0395345Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0395834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0396472Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0396977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0397537Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0397706Z 2025-08-26T20:27:24.0397797Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0398038Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0398293Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0398526Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0398786Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0399187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0399612Z return mod(**inputs) 2025-08-26T20:27:24.0400078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0400562Z outputs = self.model( 2025-08-26T20:27:24.0401036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0401534Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0402017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0402517Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0402945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0403347Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0403837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0404340Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0404854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0405391Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0405884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0406415Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0406623Z 2025-08-26T20:27:24.0406742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0407116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0407460Z return mod(**inputs) 2025-08-26T20:27:24.0407909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0408375Z outputs = self.model( 2025-08-26T20:27:24.0408815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0409291Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0409741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0410189Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0410544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0410905Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0411358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0411823Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0412290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0412759Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0413236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0413739Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0413919Z 2025-08-26T20:27:24.0414031Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0414417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0414757Z return mod(**inputs) 2025-08-26T20:27:24.0415203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0415642Z outputs = self.model( 2025-08-26T20:27:24.0416090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0416571Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0417032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0417507Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0417861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0418758Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0419205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0419670Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0420136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0420618Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0420759Z 2025-08-26T20:27:24.0420875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0421241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0421568Z return mod(**inputs) 2025-08-26T20:27:24.0422010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0422486Z outputs = self.model( 2025-08-26T20:27:24.0422905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0423349Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0423818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0424275Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0424643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0425030Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0425500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0426010Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0426509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0427054Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0427275Z 2025-08-26T20:27:24.0427394Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0427777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0428129Z return mod(**inputs) 2025-08-26T20:27:24.0428596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0429063Z outputs = self.model( 2025-08-26T20:27:24.0429529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0429998Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0430467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0430936Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0431312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0431696Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0432177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0432681Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0433188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0433681Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0433825Z 2025-08-26T20:27:24.0433938Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0434339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0434702Z return mod(**inputs) 2025-08-26T20:27:24.0435147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0435630Z outputs = self.model( 2025-08-26T20:27:24.0436082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0436566Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0437046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0437533Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0437919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0438313Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0438801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0439389Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0439923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0440419Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0440561Z 2025-08-26T20:27:24.0440642Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0440861Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0441075Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0441286Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0441515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0441876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0442202Z return mod(**inputs) 2025-08-26T20:27:24.0442620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0443049Z outputs = self.model( 2025-08-26T20:27:24.0443521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0443970Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0444435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0444880Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0445251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0445644Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0446121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0446627Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0447133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0447604Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0448044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0448518Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0448774Z 2025-08-26T20:27:24.0448893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0449280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0449624Z return mod(**inputs) 2025-08-26T20:27:24.0450049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0450491Z outputs = self.model( 2025-08-26T20:27:24.0450828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0450918Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0451250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0451346Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0451571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0451653Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0451971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0452081Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0452396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0452499Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0452812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0452931Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0452935Z 2025-08-26T20:27:24.0453047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0453268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0453340Z return mod(**inputs) 2025-08-26T20:27:24.0453675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0453750Z outputs = self.model( 2025-08-26T20:27:24.0454077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0454187Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0454514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0454626Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0454853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0454944Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0455258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0455365Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0455684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0455771Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0455777Z 2025-08-26T20:27:24.0455896Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0456112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0456185Z return mod(**inputs) 2025-08-26T20:27:24.0456523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0456615Z outputs = self.model( 2025-08-26T20:27:24.0456961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0457041Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0457382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0457473Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0457702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0457789Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0458100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0458233Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0458238Z 2025-08-26T20:27:24.0458342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0458545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0458618Z return mod(**inputs) 2025-08-26T20:27:24.0458934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0459011Z outputs = self.model( 2025-08-26T20:27:24.0459327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0459410Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0459743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0459820Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0460070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0460155Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0460491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0460622Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0460883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0460960Z return self.act(input) 2025-08-26T20:27:24.0460964Z 2025-08-26T20:27:24.0461069Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0461296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0461365Z return mod(**inputs) 2025-08-26T20:27:24.0461706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0461781Z outputs = self.model( 2025-08-26T20:27:24.0462113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0462199Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0462530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0462614Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0462855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0462955Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0463302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0463401Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0463405Z 2025-08-26T20:27:24.0463516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0463714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0463791Z return mod(**inputs) 2025-08-26T20:27:24.0464114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0464184Z outputs = self.model( 2025-08-26T20:27:24.0464502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0464575Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0464905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0464981Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0465217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0465309Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0465633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0465751Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0466077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0466249Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0466253Z 2025-08-26T20:27:24.0466365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0466577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0466666Z return mod(**inputs) 2025-08-26T20:27:24.0466975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0467051Z outputs = self.model( 2025-08-26T20:27:24.0467360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0467452Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0467766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0467855Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0468089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0468169Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0468490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0468596Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0468922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0469017Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0469021Z 2025-08-26T20:27:24.0469132Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0469350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0469422Z return mod(**inputs) 2025-08-26T20:27:24.0469749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0469846Z outputs = self.model( 2025-08-26T20:27:24.0470172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0470258Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0470597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0470698Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0470937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0471023Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0471357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0471465Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0471791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0471885Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0471889Z 2025-08-26T20:27:24.0471984Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0472072Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0472156Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0472246Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0472358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0472572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0472652Z return mod(**inputs) 2025-08-26T20:27:24.0472987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0473075Z outputs = self.model( 2025-08-26T20:27:24.0473410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0473497Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0473836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0473917Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0474207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0474294Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0474639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0474748Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0475072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0475184Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0475498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0475649Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0475654Z 2025-08-26T20:27:24.0475769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0475993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0476063Z return mod(**inputs) 2025-08-26T20:27:24.0476402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0476501Z outputs = self.model( 2025-08-26T20:27:24.0476825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0476912Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0477240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0477337Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0477591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0477676Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0478014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0495678Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0496336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0496485Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0496803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0496927Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0496939Z 2025-08-26T20:27:24.0497066Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0497286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0497370Z return mod(**inputs) 2025-08-26T20:27:24.0497702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0497789Z outputs = self.model( 2025-08-26T20:27:24.0498105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0498188Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0498506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0498585Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0498827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0499071Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0499440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0499560Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0499876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0499973Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0499977Z 2025-08-26T20:27:24.0500086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0500306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0500377Z return mod(**inputs) 2025-08-26T20:27:24.0500698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0500779Z outputs = self.model( 2025-08-26T20:27:24.0501095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0501182Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0501542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0501621Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0501859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0501942Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0502262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0502413Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0502727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0502885Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0502891Z 2025-08-26T20:27:24.0503001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0503217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0503286Z return mod(**inputs) 2025-08-26T20:27:24.0503610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0503682Z outputs = self.model( 2025-08-26T20:27:24.0504005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0504095Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0504436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0504534Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0504778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0504869Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0505194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0506184Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0506875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0507018Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0507160Z 2025-08-26T20:27:24.0507333Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0507700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0507800Z return mod(**inputs) 2025-08-26T20:27:24.0508308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0508391Z outputs = self.model( 2025-08-26T20:27:24.0508752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0508847Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0509278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0509363Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0509622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0509721Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0510066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0510218Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0510538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0510638Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0510642Z 2025-08-26T20:27:24.0510731Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0510836Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0510922Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0510999Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0511121Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0511451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0511537Z return mod(**inputs) 2025-08-26T20:27:24.0511955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0512036Z outputs = self.model( 2025-08-26T20:27:24.0512400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0512484Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0512830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0512922Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0513184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0513286Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0513655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0513790Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0514155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0514266Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0514613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0514771Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0514777Z 2025-08-26T20:27:24.0514923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0515165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0515265Z return mod(**inputs) 2025-08-26T20:27:24.0515624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0515702Z outputs = self.model( 2025-08-26T20:27:24.0516051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0516133Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0516534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0516622Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0516870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0516968Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0517308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0518181Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0518757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0519126Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0519597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0519774Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0519780Z 2025-08-26T20:27:24.0519912Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0520139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0520224Z return mod(**inputs) 2025-08-26T20:27:24.0520572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0520657Z outputs = self.model( 2025-08-26T20:27:24.0521025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0521106Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0521456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0521539Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0521795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0521898Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0522244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0522374Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0522713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0522812Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0522816Z 2025-08-26T20:27:24.0522931Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0523154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0523238Z return mod(**inputs) 2025-08-26T20:27:24.0523629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0523713Z outputs = self.model( 2025-08-26T20:27:24.0524079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0524175Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0524522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0524602Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0524856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0524942Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0525292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0525430Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0525434Z 2025-08-26T20:27:24.0525550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0525780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0525873Z return mod(**inputs) 2025-08-26T20:27:24.0526230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0526307Z outputs = self.model( 2025-08-26T20:27:24.0526675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0526755Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0527135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0527223Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0527470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0527566Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0527902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0528043Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0528292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0528377Z return self.act(input) 2025-08-26T20:27:24.0528381Z 2025-08-26T20:27:24.0528489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0528685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0528751Z return mod(**inputs) 2025-08-26T20:27:24.0529166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0529244Z outputs = self.model( 2025-08-26T20:27:24.0529558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0529634Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0529939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0530011Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0530263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0530361Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0530709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0530803Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0530807Z 2025-08-26T20:27:24.0530931Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0531135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0531210Z return mod(**inputs) 2025-08-26T20:27:24.0531530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0531606Z outputs = self.model( 2025-08-26T20:27:24.0531929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0532010Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0532333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0532405Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0532641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0532743Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0533066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0533171Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0533495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0533671Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0533692Z 2025-08-26T20:27:24.0533808Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0534028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0534097Z return mod(**inputs) 2025-08-26T20:27:24.0534441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0534519Z outputs = self.model( 2025-08-26T20:27:24.0534849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0534932Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0535277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0535358Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0535582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0535659Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0535967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0536069Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0536379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0536459Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0536462Z 2025-08-26T20:27:24.0536574Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0536774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0536843Z return mod(**inputs) 2025-08-26T20:27:24.0537198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0537273Z outputs = self.model( 2025-08-26T20:27:24.0537627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0537719Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0538045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0538117Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0538357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0538443Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0538752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0538862Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0539166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0539258Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0539280Z 2025-08-26T20:27:24.0539372Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0539453Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0539537Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0539618Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0539722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0539929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0540013Z return mod(**inputs) 2025-08-26T20:27:24.0540335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0540407Z outputs = self.model( 2025-08-26T20:27:24.0540722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0540804Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0541115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0541198Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0541428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0541515Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0541826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0541933Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0542254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0542357Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0542664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0542807Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0542812Z 2025-08-26T20:27:24.0542923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0543133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0543203Z return mod(**inputs) 2025-08-26T20:27:24.0543549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0543640Z outputs = self.model( 2025-08-26T20:27:24.0543998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0544078Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0544406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0544497Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0544723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0544809Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0545126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0545232Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0545535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0545641Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0545957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0546102Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0546106Z 2025-08-26T20:27:24.0546224Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0546442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0546517Z return mod(**inputs) 2025-08-26T20:27:24.0546861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0546961Z outputs = self.model( 2025-08-26T20:27:24.0547309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0547393Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0547734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0547813Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0548057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0548149Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0548482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0548601Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0548935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0549025Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0549037Z 2025-08-26T20:27:24.0549145Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0549358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0549436Z return mod(**inputs) 2025-08-26T20:27:24.0549771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0549851Z outputs = self.model( 2025-08-26T20:27:24.0550184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0550265Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0550618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0550699Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0550958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0551047Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0551375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0551500Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0551824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0552000Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0552004Z 2025-08-26T20:27:24.0552116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0552339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0552410Z return mod(**inputs) 2025-08-26T20:27:24.0552739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0552840Z outputs = self.model( 2025-08-26T20:27:24.0553166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0553250Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0553585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0553681Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0553935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0554023Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0554367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0554486Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0554824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0554913Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0554917Z 2025-08-26T20:27:24.0555028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0555252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0555324Z return mod(**inputs) 2025-08-26T20:27:24.0555671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0555743Z outputs = self.model( 2025-08-26T20:27:24.0556081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0556172Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0556507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0556592Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0556834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0556925Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0557278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0557397Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0557750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0557846Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0557851Z 2025-08-26T20:27:24.0557945Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0558029Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0558112Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0558202Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0558315Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0558532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0558604Z return mod(**inputs) 2025-08-26T20:27:24.0558941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0559024Z outputs = self.model( 2025-08-26T20:27:24.0559692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0559828Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0560182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0560269Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0560520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0560618Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0560990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0561109Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0561439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0561543Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0561871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0562021Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0562026Z 2025-08-26T20:27:24.0562139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0562367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0562442Z return mod(**inputs) 2025-08-26T20:27:24.0562791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0562870Z outputs = self.model( 2025-08-26T20:27:24.0563213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0563304Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0563648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0563734Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0564003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0564097Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0564436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0564577Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0564927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0565053Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0565381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0565508Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0565512Z 2025-08-26T20:27:24.0565641Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0565861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0565936Z return mod(**inputs) 2025-08-26T20:27:24.0566290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0566367Z outputs = self.model( 2025-08-26T20:27:24.0566718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0566798Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0567169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0567251Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0567499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0567596Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0567942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0568175Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0568520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0568619Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0568623Z 2025-08-26T20:27:24.0568746Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0568972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0569058Z return mod(**inputs) 2025-08-26T20:27:24.0569410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0569497Z outputs = self.model( 2025-08-26T20:27:24.0569845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0569933Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0570296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0570382Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0570636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0570722Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0571035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0571173Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0571176Z 2025-08-26T20:27:24.0571288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0571509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0571581Z return mod(**inputs) 2025-08-26T20:27:24.0571932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0572023Z outputs = self.model( 2025-08-26T20:27:24.0572333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0572418Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0572745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0572831Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0573068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0573155Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0573493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0573625Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0573864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0573960Z return self.act(input) 2025-08-26T20:27:24.0573963Z 2025-08-26T20:27:24.0574098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0574319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0574391Z return mod(**inputs) 2025-08-26T20:27:24.0574732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0574823Z outputs = self.model( 2025-08-26T20:27:24.0575165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0575243Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0575576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0575663Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0575907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0576000Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0576332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0576428Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0576435Z 2025-08-26T20:27:24.0576546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0576764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0576843Z return mod(**inputs) 2025-08-26T20:27:24.0577179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0577260Z outputs = self.model( 2025-08-26T20:27:24.0577600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0577679Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0578021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0578099Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0578348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0578435Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0578794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0578929Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0579260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0579435Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0579440Z 2025-08-26T20:27:24.0579554Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0579773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0579843Z return mod(**inputs) 2025-08-26T20:27:24.0580173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0580254Z outputs = self.model( 2025-08-26T20:27:24.0580581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0580667Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0581015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0581184Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0581428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0581511Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0581845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0581977Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0582350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0582446Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0582450Z 2025-08-26T20:27:24.0582563Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0582786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0582858Z return mod(**inputs) 2025-08-26T20:27:24.0583193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0583265Z outputs = self.model( 2025-08-26T20:27:24.0583596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0583680Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0584008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0584099Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0584338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0584431Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0584761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0584868Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0585202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0585297Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0585302Z 2025-08-26T20:27:24.0585418Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0585506Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0585595Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0585696Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0585809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0586032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0586104Z return mod(**inputs) 2025-08-26T20:27:24.0586455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0586529Z outputs = self.model( 2025-08-26T20:27:24.0586865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0586957Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0587297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0587381Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0587633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0587745Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0588080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0588187Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0588521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0588645Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0588968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0589114Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0589119Z 2025-08-26T20:27:24.0589230Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0589452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0589525Z return mod(**inputs) 2025-08-26T20:27:24.0589864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0589939Z outputs = self.model( 2025-08-26T20:27:24.0590269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0590358Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0590684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0590769Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0591012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0591106Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0591435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0591540Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0591916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0592029Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0592377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0592498Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0592502Z 2025-08-26T20:27:24.0592634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0592851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0592925Z return mod(**inputs) 2025-08-26T20:27:24.0593266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0593344Z outputs = self.model( 2025-08-26T20:27:24.0593695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0593780Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0594127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0594214Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0594464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0594557Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0594914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0595020Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0595366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0595460Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0595484Z 2025-08-26T20:27:24.0595606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0595827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0595911Z return mod(**inputs) 2025-08-26T20:27:24.0596555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0596679Z outputs = self.model( 2025-08-26T20:27:24.0597143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0597228Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0597581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0597659Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0597907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0598005Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0598341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0598469Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0598809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0598986Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0598990Z 2025-08-26T20:27:24.0599103Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0599386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0599488Z return mod(**inputs) 2025-08-26T20:27:24.0599947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0600042Z outputs = self.model( 2025-08-26T20:27:24.0600445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0600535Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0600886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0600965Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0601220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0601305Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0601640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0601759Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0602084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0602178Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0602183Z 2025-08-26T20:27:24.0602337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0602554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0602626Z return mod(**inputs) 2025-08-26T20:27:24.0602965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0603038Z outputs = self.model( 2025-08-26T20:27:24.0603370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0603490Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0603820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0603905Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0604175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0604256Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0604570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0604680Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0604994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0605085Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0605090Z 2025-08-26T20:27:24.0605181Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0605262Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0605341Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0605429Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0605535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0605772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0605840Z return mod(**inputs) 2025-08-26T20:27:24.0606152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0606233Z outputs = self.model( 2025-08-26T20:27:24.0606543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0606624Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0606965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0607057Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0607292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0607375Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0607689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0607799Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0608118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0608219Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0608511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0608656Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0608662Z 2025-08-26T20:27:24.0608766Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0608996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0609064Z return mod(**inputs) 2025-08-26T20:27:24.0609376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0609454Z outputs = self.model( 2025-08-26T20:27:24.0609764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0609868Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0610180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0610262Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0610488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0610569Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0610891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0610996Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0611339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0611440Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0611743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0611863Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0611866Z 2025-08-26T20:27:24.0611978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0612199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0612273Z return mod(**inputs) 2025-08-26T20:27:24.0612618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0612690Z outputs = self.model( 2025-08-26T20:27:24.0613034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0613122Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0613460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0613541Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0613791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0613874Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0614183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0614289Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0614601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0614687Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0614691Z 2025-08-26T20:27:24.0614801Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0615003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0615068Z return mod(**inputs) 2025-08-26T20:27:24.0615393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0615483Z outputs = self.model( 2025-08-26T20:27:24.0615806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0615879Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0616199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0616296Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0616524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0616611Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0616919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0617047Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0617052Z 2025-08-26T20:27:24.0617155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0617356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0617429Z return mod(**inputs) 2025-08-26T20:27:24.0617745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0617824Z outputs = self.model( 2025-08-26T20:27:24.0618168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0618246Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0618593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0618670Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0618914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0618997Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0619330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0619457Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0619693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0619775Z return self.act(input) 2025-08-26T20:27:24.0619797Z 2025-08-26T20:27:24.0619908Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0620145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0620217Z return mod(**inputs) 2025-08-26T20:27:24.0620543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0620628Z outputs = self.model( 2025-08-26T20:27:24.0620955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0621041Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0621372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0621459Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0621701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0621786Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0622123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0622232Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0622236Z 2025-08-26T20:27:24.0622351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0622562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0622633Z return mod(**inputs) 2025-08-26T20:27:24.0622971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0623064Z outputs = self.model( 2025-08-26T20:27:24.0623401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0623480Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0623815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0623892Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0624130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0624242Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0624567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0624683Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0625010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0625173Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0625177Z 2025-08-26T20:27:24.0625298Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0625514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0625591Z return mod(**inputs) 2025-08-26T20:27:24.0625917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0625996Z outputs = self.model( 2025-08-26T20:27:24.0626331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0626411Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0626768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0626846Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0627108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0627199Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0627526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0627638Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0627965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0628063Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0628066Z 2025-08-26T20:27:24.0628176Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0628397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0628467Z return mod(**inputs) 2025-08-26T20:27:24.0628803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0628909Z outputs = self.model( 2025-08-26T20:27:24.0629237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0629324Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0629663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0629759Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0630005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0630091Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0630425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0630531Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0630865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0630960Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0630964Z 2025-08-26T20:27:24.0631049Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0631142Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0631224Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0631314Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0631424Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0631638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0631715Z return mod(**inputs) 2025-08-26T20:27:24.0632050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0632132Z outputs = self.model( 2025-08-26T20:27:24.0632461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0632537Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0632870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0632948Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0633197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0633299Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0633632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0633753Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0634078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0634190Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0634501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0634652Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0634658Z 2025-08-26T20:27:24.0634770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0634989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0635068Z return mod(**inputs) 2025-08-26T20:27:24.0635404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0635488Z outputs = self.model( 2025-08-26T20:27:24.0635843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0635927Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0636254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0636330Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0636580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0636689Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0637037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0637147Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0637485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0637603Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0637930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0638054Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0638058Z 2025-08-26T20:27:24.0638167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0638394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0638469Z return mod(**inputs) 2025-08-26T20:27:24.0638813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0638896Z outputs = self.model( 2025-08-26T20:27:24.0639328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0639460Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0639811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0639896Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0640152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0640242Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0640634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0640763Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0641110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0641202Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0641208Z 2025-08-26T20:27:24.0641320Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0641546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0641617Z return mod(**inputs) 2025-08-26T20:27:24.0641963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0642041Z outputs = self.model( 2025-08-26T20:27:24.0642381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0642471Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0642808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0642919Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0643168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0643264Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0643611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0643747Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0644089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0644255Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0644261Z 2025-08-26T20:27:24.0644380Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0644597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0644677Z return mod(**inputs) 2025-08-26T20:27:24.0645029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0645102Z outputs = self.model( 2025-08-26T20:27:24.0645451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0645534Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0645880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0645958Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0646205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0646300Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0646638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0646765Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0647099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0647197Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0647201Z 2025-08-26T20:27:24.0647334Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0647555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0647634Z return mod(**inputs) 2025-08-26T20:27:24.0647992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0648074Z outputs = self.model( 2025-08-26T20:27:24.0648403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0648487Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0648804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0648880Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0649114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0649194Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0649514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0649625Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0649954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0650047Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0650051Z 2025-08-26T20:27:24.0650131Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0650219Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0650296Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0650390Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0650500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0650702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0650773Z return mod(**inputs) 2025-08-26T20:27:24.0651084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0651154Z outputs = self.model( 2025-08-26T20:27:24.0651473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0651547Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0651864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0651939Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0652173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0652253Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0652562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0652679Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0652992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0653098Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0653392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0653527Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0653541Z 2025-08-26T20:27:24.0653645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0653867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0653946Z return mod(**inputs) 2025-08-26T20:27:24.0654306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0654393Z outputs = self.model( 2025-08-26T20:27:24.0654733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0654812Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0655155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0655232Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0655488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0655569Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0655884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0656000Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0656324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0656432Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0656724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0656842Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0656867Z 2025-08-26T20:27:24.0656973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0657171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0657244Z return mod(**inputs) 2025-08-26T20:27:24.0657557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0657635Z outputs = self.model( 2025-08-26T20:27:24.0657964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0658043Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0658388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0658466Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0658715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0658803Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0659136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0659251Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0659580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0659679Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0659683Z 2025-08-26T20:27:24.0659793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0660009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0660079Z return mod(**inputs) 2025-08-26T20:27:24.0660457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0660557Z outputs = self.model( 2025-08-26T20:27:24.0660897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0660999Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0661338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0661424Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0661660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0661745Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0662081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0662211Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0662216Z 2025-08-26T20:27:24.0662331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0662548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0662618Z return mod(**inputs) 2025-08-26T20:27:24.0662962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0663055Z outputs = self.model( 2025-08-26T20:27:24.0663395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0663472Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0663820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0663927Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0664166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0664258Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0664585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0664720Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0664960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0665031Z return self.act(input) 2025-08-26T20:27:24.0665042Z 2025-08-26T20:27:24.0665147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0665345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0665419Z return mod(**inputs) 2025-08-26T20:27:24.0665731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0665806Z outputs = self.model( 2025-08-26T20:27:24.0666124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0666203Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0666537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0666612Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0666855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0666941Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0667267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0667380Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0667384Z 2025-08-26T20:27:24.0667497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0667731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0667805Z return mod(**inputs) 2025-08-26T20:27:24.0668143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0668214Z outputs = self.model( 2025-08-26T20:27:24.0668540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0668625Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0668953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0669039Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0669277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0669362Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0669700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0669829Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0670158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0670322Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0670343Z 2025-08-26T20:27:24.0670461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0670675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0670746Z return mod(**inputs) 2025-08-26T20:27:24.0671088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0671160Z outputs = self.model( 2025-08-26T20:27:24.0671505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0671583Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0671935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0672021Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0672264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0672359Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0672694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0672810Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0673145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0673233Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0673237Z 2025-08-26T20:27:24.0673356Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0673572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0673650Z return mod(**inputs) 2025-08-26T20:27:24.0673990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0674082Z outputs = self.model( 2025-08-26T20:27:24.0674421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0674515Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0674849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0674927Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0675170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0675255Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0675577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0675693Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0676018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0676118Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0676123Z 2025-08-26T20:27:24.0676211Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0676351Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0676443Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0676527Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0676646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0676863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0676936Z return mod(**inputs) 2025-08-26T20:27:24.0677284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0677376Z outputs = self.model( 2025-08-26T20:27:24.0677724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0677806Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0678161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0678239Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0678481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0678575Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0678908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0679021Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0679515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0679631Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0679965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0680115Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0680120Z 2025-08-26T20:27:24.0680241Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0680460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0680544Z return mod(**inputs) 2025-08-26T20:27:24.0680893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0680971Z outputs = self.model( 2025-08-26T20:27:24.0681329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0681410Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0681771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0681854Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0682101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0682195Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0682534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0682652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0682993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0683104Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0683424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0683564Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0683568Z 2025-08-26T20:27:24.0683688Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0683907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0683983Z return mod(**inputs) 2025-08-26T20:27:24.0684334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0684431Z outputs = self.model( 2025-08-26T20:27:24.0684767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0684845Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0685193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0685273Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0685523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0685609Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0685943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0686056Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0686392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0686490Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0686494Z 2025-08-26T20:27:24.0686605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0686833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0686906Z return mod(**inputs) 2025-08-26T20:27:24.0687244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0687326Z outputs = self.model( 2025-08-26T20:27:24.0687661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0687745Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0688106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0688185Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0688455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0688543Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0688890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0689009Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0689352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0689520Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0689526Z 2025-08-26T20:27:24.0689640Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0689870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0689944Z return mod(**inputs) 2025-08-26T20:27:24.0690289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0690396Z outputs = self.model( 2025-08-26T20:27:24.0690738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0690825Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0691178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0691265Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0691531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0691628Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0691965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0692086Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0692437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0692527Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0692530Z 2025-08-26T20:27:24.0692650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0692870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0692943Z return mod(**inputs) 2025-08-26T20:27:24.0693289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0693366Z outputs = self.model( 2025-08-26T20:27:24.0693727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0693811Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0694160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0694239Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0694483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0694577Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0694915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0695056Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0695401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0695512Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0695524Z 2025-08-26T20:27:24.0695610Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0695697Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0695788Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0695870Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0695979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0696355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0696463Z return mod(**inputs) 2025-08-26T20:27:24.0696890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0696970Z outputs = self.model( 2025-08-26T20:27:24.0697320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0697402Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0697746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0697910Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0698151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0698247Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0698580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0698730Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0699064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0699171Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0699486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0699632Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0699636Z 2025-08-26T20:27:24.0699753Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0699966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0700039Z return mod(**inputs) 2025-08-26T20:27:24.0700378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0700454Z outputs = self.model( 2025-08-26T20:27:24.0700789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0700867Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0701193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0701275Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0701513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0701603Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0701928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0702048Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0702408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0702514Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0702854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0702974Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0702978Z 2025-08-26T20:27:24.0703091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0703306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0703381Z return mod(**inputs) 2025-08-26T20:27:24.0703714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0703788Z outputs = self.model( 2025-08-26T20:27:24.0704128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0704204Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0704539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0704637Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0704879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0704971Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0705306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0705446Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0705775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0705872Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0705876Z 2025-08-26T20:27:24.0705987Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0706200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0706280Z return mod(**inputs) 2025-08-26T20:27:24.0706614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0706693Z outputs = self.model( 2025-08-26T20:27:24.0707025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0707105Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0707448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0707524Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0707773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0707858Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0708199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0708330Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0708334Z 2025-08-26T20:27:24.0708445Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0708666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0708738Z return mod(**inputs) 2025-08-26T20:27:24.0709094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0709169Z outputs = self.model( 2025-08-26T20:27:24.0709515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0709603Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0709927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0710011Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0710246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0710337Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0710657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0710784Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0711021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0711095Z return self.act(input) 2025-08-26T20:27:24.0711099Z 2025-08-26T20:27:24.0711237Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0711447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0711517Z return mod(**inputs) 2025-08-26T20:27:24.0711850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0711923Z outputs = self.model( 2025-08-26T20:27:24.0712253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0712350Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0712677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0712759Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0712983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0713073Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0713396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0713491Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0713494Z 2025-08-26T20:27:24.0713601Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0713814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0713892Z return mod(**inputs) 2025-08-26T20:27:24.0714224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0714304Z outputs = self.model( 2025-08-26T20:27:24.0714637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0714717Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0715055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0715131Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0715377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0715463Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0715825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0715938Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0716298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0716477Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0716481Z 2025-08-26T20:27:24.0716593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0716819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0716891Z return mod(**inputs) 2025-08-26T20:27:24.0717239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0717316Z outputs = self.model( 2025-08-26T20:27:24.0717655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0717741Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0718080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0718186Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0718433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0718520Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0718864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0718993Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0719418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0719513Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0719518Z 2025-08-26T20:27:24.0719637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0719852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0719925Z return mod(**inputs) 2025-08-26T20:27:24.0720269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0720344Z outputs = self.model( 2025-08-26T20:27:24.0720697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0720776Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0721101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0721187Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0721427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0721514Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0721825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0721936Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0722417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0722526Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0722533Z 2025-08-26T20:27:24.0722624Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0722726Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0722815Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0722897Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0723866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0724091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0724162Z return mod(**inputs) 2025-08-26T20:27:24.0724478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0724548Z outputs = self.model( 2025-08-26T20:27:24.0724859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0724943Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0725252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0725333Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0725560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0725646Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0725980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0726081Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0726400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0726499Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0726818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0726960Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0726964Z 2025-08-26T20:27:24.0727068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0727278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0727346Z return mod(**inputs) 2025-08-26T20:27:24.0727660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0727730Z outputs = self.model( 2025-08-26T20:27:24.0728049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0728123Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0728438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0728519Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0728746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0728833Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0729140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0729240Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0729557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0729654Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0729959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0730087Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0730091Z 2025-08-26T20:27:24.0730200Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0730412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0730479Z return mod(**inputs) 2025-08-26T20:27:24.0730790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0730857Z outputs = self.model( 2025-08-26T20:27:24.0731169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0731241Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0731545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0731625Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0731848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0731936Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0732249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-26T20:27:24.0732376Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:27:24.0732704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0732793Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0732797Z 2025-08-26T20:27:24.0732915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0733146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0733224Z return mod(**inputs) 2025-08-26T20:27:24.0733555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0733630Z outputs = self.model( 2025-08-26T20:27:24.0733964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0734044Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0734387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0734458Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0734690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0734771Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0735084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0735204Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0735519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-26T20:27:24.0735679Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:27:24.0735682Z 2025-08-26T20:27:24.0735784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0735989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0736055Z return mod(**inputs) 2025-08-26T20:27:24.0736369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0736446Z outputs = self.model( 2025-08-26T20:27:24.0736772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0736873Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0737213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0737290Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0737534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0737625Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0737937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0738047Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0738364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-26T20:27:24.0738445Z key_states = self.k_proj(current_states) 2025-08-26T20:27:24.0738448Z 2025-08-26T20:27:24.0738551Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0738773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0738838Z return mod(**inputs) 2025-08-26T20:27:24.0739153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0739224Z outputs = self.model( 2025-08-26T20:27:24.0739553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0739659Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0740003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0740081Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0740309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0740389Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0740724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0740840Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0741178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-26T20:27:24.0741273Z value_states = self.v_proj(current_states) 2025-08-26T20:27:24.0741277Z 2025-08-26T20:27:24.0741369Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0741457Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0741541Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0741631Z cudagraph partition due to non gpu ops 2025-08-26T20:27:24.0741742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0741962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0742034Z return mod(**inputs) 2025-08-26T20:27:24.0742373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0742456Z outputs = self.model( 2025-08-26T20:27:24.0742783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0742866Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0743204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0743286Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0743527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0743608Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0743926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0744035Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0744353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0744456Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0744780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:27:24.0744931Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:24.0744934Z 2025-08-26T20:27:24.0745046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0745265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0745353Z return mod(**inputs) 2025-08-26T20:27:24.0745710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0745778Z outputs = self.model( 2025-08-26T20:27:24.0746097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0746198Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0746526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0746610Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0746850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0746933Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0747274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0747388Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0747723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-26T20:27:24.0747826Z attn_output, attn_weights = attention_interface( 2025-08-26T20:27:24.0748149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:27:24.0748259Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:27:24.0748263Z 2025-08-26T20:27:24.0748366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0748575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0748644Z return mod(**inputs) 2025-08-26T20:27:24.0748979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0749052Z outputs = self.model( 2025-08-26T20:27:24.0749387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0749471Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0749816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0749901Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0750143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0750252Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0750579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-26T20:27:24.0750695Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:27:24.0751026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-26T20:27:24.0751113Z attn_output = self.out_proj(attn_output) 2025-08-26T20:27:24.0751116Z 2025-08-26T20:27:24.0751234Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0751446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0751515Z return mod(**inputs) 2025-08-26T20:27:24.0751854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0751925Z outputs = self.model( 2025-08-26T20:27:24.0752284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0752362Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0752699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0752774Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0753013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0753124Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0753454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0753590Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0753594Z 2025-08-26T20:27:24.0753701Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0753912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0753988Z return mod(**inputs) 2025-08-26T20:27:24.0754314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0754394Z outputs = self.model( 2025-08-26T20:27:24.0754731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0754816Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0755149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0755227Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0755470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0755558Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0755900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-26T20:27:24.0756029Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:27:24.0756271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:24.0756354Z return self.act(input) 2025-08-26T20:27:24.0756358Z 2025-08-26T20:27:24.0756483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0756704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0756775Z return mod(**inputs) 2025-08-26T20:27:24.0757131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-26T20:27:24.0757208Z outputs = self.model( 2025-08-26T20:27:24.0757533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-26T20:27:24.0757619Z decoder_outputs = self.decoder( 2025-08-26T20:27:24.0757947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-26T20:27:24.0758033Z layer_outputs = decoder_layer( 2025-08-26T20:27:24.0758270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:24.0758355Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:24.0758692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-26T20:27:24.0758780Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:27:24.0758805Z 2025-08-26T20:27:24.0758921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0759140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0759304Z return mod(**inputs) 2025-08-26T20:27:24.0759655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1393, in forward 2025-08-26T20:27:24.0759815Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-26T20:27:24.0759820Z 2025-08-26T20:27:24.0759944Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:24.0760162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:24.0760245Z return mod(**inputs) 2025-08-26T20:27:24.0760593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1398, in forward 2025-08-26T20:27:24.0760787Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:27:24.0760799Z 2025-08-26T20:27:33.6175826Z Compilation time (from dynamo_timed): 20.746391923 2025-08-26T20:27:33.6190110Z pass 2025-08-26T20:27:33.6190556Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:33.6191366Z TIMING: _recursive_pre_grad_passes:0.01122 _recursive_joint_graph_passes:0.57671 _recursive_post_grad_passes:0.11798 async_compile.wait:0.7426 code_gen:8.93005 inductor_compile:11.46776 backend_compile:16.8195 gc:0.00075 entire_frame_compile:20.74639 total_wall_time:20.74639 2025-08-26T20:27:33.6192291Z STATS: call_* op count: 652 | FakeTensorMode.__torch_dispatch__:22573 | FakeTensor.__torch_dispatch__:7513 | ProxyTorchDispatchMode.__torch_dispatch__:8304 2025-08-26T20:27:33.6192890Z Dynamo produced 1 graphs covering 652 ops with 0 graph breaks (0 unique) 2025-08-26T20:27:39.1821522Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:27:39.1822430Z from pkg_resources import resource_filename 2025-08-26T20:27:39.7713554Z 2025-08-26T20:27:41.2388997Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:27:41.2389333Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:27:41.2396731Z cpu eval CamemBert 2025-08-26T20:27:41.7823331Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:42.0278462Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:42.2708864Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:50.2987362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.2988088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.2988448Z return mod(**inputs) 2025-08-26T20:27:50.2988910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.2989329Z outputs = self.roberta( 2025-08-26T20:27:50.2989768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-26T20:27:50.2990201Z embedding_output = self.embeddings( 2025-08-26T20:27:50.2990654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-26T20:27:50.2991289Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:27:50.2992358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1590, in create_position_ids_from_input_ids 2025-08-26T20:27:50.2992847Z mask = input_ids.ne(padding_idx).int() 2025-08-26T20:27:50.2992998Z 2025-08-26T20:27:50.2993084Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2993300Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2993509Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2993780Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2993990Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2994216Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2994443Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2994658Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2994875Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2995099Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2995346Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2995578Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.2995859Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.2996498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.2996892Z return mod(**inputs) 2025-08-26T20:27:50.2997317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.2998368Z outputs = self.roberta( 2025-08-26T20:27:50.2999019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-26T20:27:50.2999743Z embedding_output = self.embeddings( 2025-08-26T20:27:50.3000234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-26T20:27:50.3000868Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:27:50.3001651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-08-26T20:27:50.3002352Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:27:50.3002636Z 2025-08-26T20:27:50.3002773Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3003211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3003589Z return mod(**inputs) 2025-08-26T20:27:50.3004205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3004734Z outputs = self.roberta( 2025-08-26T20:27:50.3005190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-26T20:27:50.3005658Z embedding_output = self.embeddings( 2025-08-26T20:27:50.3006117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-26T20:27:50.3006728Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:27:50.3007431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-08-26T20:27:50.3008085Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:27:50.3008355Z 2025-08-26T20:27:50.3008479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3008915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3009288Z return mod(**inputs) 2025-08-26T20:27:50.3009735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3010149Z outputs = self.roberta( 2025-08-26T20:27:50.3010603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3011032Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3011450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3011904Z layer_outputs = layer_module( 2025-08-26T20:27:50.3012266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3012757Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3013217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3013677Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3014107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3014523Z return func(*args, **kwargs) 2025-08-26T20:27:50.3014937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3015352Z self_outputs = self.self( 2025-08-26T20:27:50.3015728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3016110Z return func(*args, **kwargs) 2025-08-26T20:27:50.3016503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3017070Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3017377Z 2025-08-26T20:27:50.3017491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3017875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3018255Z return mod(**inputs) 2025-08-26T20:27:50.3018644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3019054Z outputs = self.roberta( 2025-08-26T20:27:50.3019470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3019889Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3020317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3020725Z layer_outputs = layer_module( 2025-08-26T20:27:50.3021102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3021589Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3022008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3022436Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3022817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3023206Z return func(*args, **kwargs) 2025-08-26T20:27:50.3023618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3024056Z self_outputs = self.self( 2025-08-26T20:27:50.3024460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3024890Z return func(*args, **kwargs) 2025-08-26T20:27:50.3025311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3025743Z self.key(current_states) 2025-08-26T20:27:50.3025864Z 2025-08-26T20:27:50.3025984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3026386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3026772Z return mod(**inputs) 2025-08-26T20:27:50.3027215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3027652Z outputs = self.roberta( 2025-08-26T20:27:50.3028072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3028509Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3028991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3029436Z layer_outputs = layer_module( 2025-08-26T20:27:50.3029812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3030213Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3030655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3031098Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3031509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3031905Z return func(*args, **kwargs) 2025-08-26T20:27:50.3032328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3032761Z self_outputs = self.self( 2025-08-26T20:27:50.3033147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3033550Z return func(*args, **kwargs) 2025-08-26T20:27:50.3033970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3034397Z self.value(current_states) 2025-08-26T20:27:50.3034526Z 2025-08-26T20:27:50.3034615Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3034894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3035284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3035631Z return mod(**inputs) 2025-08-26T20:27:50.3036073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3036506Z outputs = self.roberta( 2025-08-26T20:27:50.3036940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3037392Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3037834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3038277Z layer_outputs = layer_module( 2025-08-26T20:27:50.3038666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3039075Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3039617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3040095Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3040553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3040963Z return func(*args, **kwargs) 2025-08-26T20:27:50.3041399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3041840Z self_outputs = self.self( 2025-08-26T20:27:50.3042235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3042656Z return func(*args, **kwargs) 2025-08-26T20:27:50.3043078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3043576Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3043763Z 2025-08-26T20:27:50.3043880Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3044253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3044578Z return mod(**inputs) 2025-08-26T20:27:50.3044976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3045388Z outputs = self.roberta( 2025-08-26T20:27:50.3045784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3046210Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3046655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3047069Z layer_outputs = layer_module( 2025-08-26T20:27:50.3047429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3047796Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3048234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3048683Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3049073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3049473Z return func(*args, **kwargs) 2025-08-26T20:27:50.3049878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3050411Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3050876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3051309Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3051452Z 2025-08-26T20:27:50.3051566Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3051935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3052273Z return mod(**inputs) 2025-08-26T20:27:50.3052674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3053100Z outputs = self.roberta( 2025-08-26T20:27:50.3053508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3053927Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3054352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3054761Z layer_outputs = layer_module( 2025-08-26T20:27:50.3055117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3055513Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3055965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3056412Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3056837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3057268Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3057719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3058224Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3058682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3059102Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3059239Z 2025-08-26T20:27:50.3059350Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3059702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3060042Z return mod(**inputs) 2025-08-26T20:27:50.3060426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3060843Z outputs = self.roberta( 2025-08-26T20:27:50.3061253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3061669Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3062085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3062507Z layer_outputs = layer_module( 2025-08-26T20:27:50.3062867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3063238Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3063658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3064095Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3064499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3064906Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3065419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3065977Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3066489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3066995Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3067422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3067802Z return self.act(input) 2025-08-26T20:27:50.3067931Z 2025-08-26T20:27:50.3068044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3068439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3068777Z return mod(**inputs) 2025-08-26T20:27:50.3069171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3069586Z outputs = self.roberta( 2025-08-26T20:27:50.3069983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3070420Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3070832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3071238Z layer_outputs = layer_module( 2025-08-26T20:27:50.3071589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3071957Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3072400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3072838Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3073267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3073691Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3074158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3074693Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3075199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3075635Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3075793Z 2025-08-26T20:27:50.3075911Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3076303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3076665Z return mod(**inputs) 2025-08-26T20:27:50.3077094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3077533Z outputs = self.roberta( 2025-08-26T20:27:50.3077956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3078412Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3078856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3079388Z layer_outputs = layer_module( 2025-08-26T20:27:50.3079807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3080220Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3080708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3081168Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3081583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3081970Z return func(*args, **kwargs) 2025-08-26T20:27:50.3082412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3082863Z self_outputs = self.self( 2025-08-26T20:27:50.3083262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3083682Z return func(*args, **kwargs) 2025-08-26T20:27:50.3084127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3084740Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3085035Z 2025-08-26T20:27:50.3085161Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3085562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3085958Z return mod(**inputs) 2025-08-26T20:27:50.3086396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3086855Z outputs = self.roberta( 2025-08-26T20:27:50.3087292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3087755Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3088190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3088631Z layer_outputs = layer_module( 2025-08-26T20:27:50.3088996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3089369Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3089782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3090212Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3090609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3090996Z return func(*args, **kwargs) 2025-08-26T20:27:50.3091398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3091818Z self_outputs = self.self( 2025-08-26T20:27:50.3092193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3092576Z return func(*args, **kwargs) 2025-08-26T20:27:50.3092984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3093396Z self.key(current_states) 2025-08-26T20:27:50.3093526Z 2025-08-26T20:27:50.3093639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3094014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3094355Z return mod(**inputs) 2025-08-26T20:27:50.3094755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3095170Z outputs = self.roberta( 2025-08-26T20:27:50.3095571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3096012Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3096637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3097109Z layer_outputs = layer_module( 2025-08-26T20:27:50.3097479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3097877Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3098309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3098736Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3099124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3099525Z return func(*args, **kwargs) 2025-08-26T20:27:50.3099935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3100372Z self_outputs = self.self( 2025-08-26T20:27:50.3100763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3101147Z return func(*args, **kwargs) 2025-08-26T20:27:50.3101593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3102044Z self.value(current_states) 2025-08-26T20:27:50.3102180Z 2025-08-26T20:27:50.3102271Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3102513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3102883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3103268Z return mod(**inputs) 2025-08-26T20:27:50.3103664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3104076Z outputs = self.roberta( 2025-08-26T20:27:50.3104464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3104878Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3105287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3105693Z layer_outputs = layer_module( 2025-08-26T20:27:50.3106038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3106412Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3106818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3107233Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3107610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3107972Z return func(*args, **kwargs) 2025-08-26T20:27:50.3108363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3108771Z self_outputs = self.self( 2025-08-26T20:27:50.3109134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3109509Z return func(*args, **kwargs) 2025-08-26T20:27:50.3109899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3110377Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3110571Z 2025-08-26T20:27:50.3110710Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3111083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3111412Z return mod(**inputs) 2025-08-26T20:27:50.3111830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3112243Z outputs = self.roberta( 2025-08-26T20:27:50.3112646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3113065Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3113501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3113955Z layer_outputs = layer_module( 2025-08-26T20:27:50.3114338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3114738Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3115186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3115632Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3116064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3116474Z return func(*args, **kwargs) 2025-08-26T20:27:50.3116898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3117383Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3117885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3118359Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3118510Z 2025-08-26T20:27:50.3118630Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3119027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3119450Z return mod(**inputs) 2025-08-26T20:27:50.3119881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3120335Z outputs = self.roberta( 2025-08-26T20:27:50.3120762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3121188Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3121580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3121984Z layer_outputs = layer_module( 2025-08-26T20:27:50.3122334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3122694Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3123096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3123520Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3123929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3124330Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3124777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3125316Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3126420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3126870Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3127013Z 2025-08-26T20:27:50.3127128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3127521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3127850Z return mod(**inputs) 2025-08-26T20:27:50.3128239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3128647Z outputs = self.roberta( 2025-08-26T20:27:50.3129041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3129452Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3129856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3130269Z layer_outputs = layer_module( 2025-08-26T20:27:50.3130626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3130996Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3131462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3131909Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3132317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3132791Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3133229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3133732Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3134196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3134646Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3135038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3135386Z return self.act(input) 2025-08-26T20:27:50.3135500Z 2025-08-26T20:27:50.3135605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3135981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3136309Z return mod(**inputs) 2025-08-26T20:27:50.3136700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3137103Z outputs = self.roberta( 2025-08-26T20:27:50.3137496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3137908Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3138314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3138733Z layer_outputs = layer_module( 2025-08-26T20:27:50.3139079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3139443Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3139861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3140280Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3140685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3141086Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3141536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3142042Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3142496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3142909Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3143046Z 2025-08-26T20:27:50.3143148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3143504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3143823Z return mod(**inputs) 2025-08-26T20:27:50.3144206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3144612Z outputs = self.roberta( 2025-08-26T20:27:50.3145016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3145416Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3145811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3146239Z layer_outputs = layer_module( 2025-08-26T20:27:50.3146585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3146954Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3147384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3147851Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3148250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3148611Z return func(*args, **kwargs) 2025-08-26T20:27:50.3149007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3149417Z self_outputs = self.self( 2025-08-26T20:27:50.3149788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3150160Z return func(*args, **kwargs) 2025-08-26T20:27:50.3150561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3151121Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3151392Z 2025-08-26T20:27:50.3151505Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3151882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3152208Z return mod(**inputs) 2025-08-26T20:27:50.3152602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3153026Z outputs = self.roberta( 2025-08-26T20:27:50.3153446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3153882Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3154305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3154738Z layer_outputs = layer_module( 2025-08-26T20:27:50.3155111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3155506Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3155961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3156411Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3156855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3157272Z return func(*args, **kwargs) 2025-08-26T20:27:50.3157717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3158170Z self_outputs = self.self( 2025-08-26T20:27:50.3158582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3159000Z return func(*args, **kwargs) 2025-08-26T20:27:50.3159521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3159986Z self.key(current_states) 2025-08-26T20:27:50.3160115Z 2025-08-26T20:27:50.3160233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3160636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3161002Z return mod(**inputs) 2025-08-26T20:27:50.3161489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3161912Z outputs = self.roberta( 2025-08-26T20:27:50.3162331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3162767Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3163196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3163650Z layer_outputs = layer_module( 2025-08-26T20:27:50.3164017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3164414Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3164857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3165304Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3165711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3166110Z return func(*args, **kwargs) 2025-08-26T20:27:50.3166531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3166972Z self_outputs = self.self( 2025-08-26T20:27:50.3167359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3167750Z return func(*args, **kwargs) 2025-08-26T20:27:50.3168172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3168607Z self.value(current_states) 2025-08-26T20:27:50.3168734Z 2025-08-26T20:27:50.3168832Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3169090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3169480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3169842Z return mod(**inputs) 2025-08-26T20:27:50.3170252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3170688Z outputs = self.roberta( 2025-08-26T20:27:50.3171119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3171553Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3171981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3172403Z layer_outputs = layer_module( 2025-08-26T20:27:50.3172757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3173118Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3173537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3173956Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3174344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3174722Z return func(*args, **kwargs) 2025-08-26T20:27:50.3175117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3175530Z self_outputs = self.self( 2025-08-26T20:27:50.3175899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3176297Z return func(*args, **kwargs) 2025-08-26T20:27:50.3176691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3177188Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3177383Z 2025-08-26T20:27:50.3177491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3177857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3178205Z return mod(**inputs) 2025-08-26T20:27:50.3178591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3179005Z outputs = self.roberta( 2025-08-26T20:27:50.3179404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3179845Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3180273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3180678Z layer_outputs = layer_module( 2025-08-26T20:27:50.3181029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3181394Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3181805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3182223Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3182611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3182985Z return func(*args, **kwargs) 2025-08-26T20:27:50.3183385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3183855Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3184313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3184737Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3184885Z 2025-08-26T20:27:50.3184990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3185356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3185686Z return mod(**inputs) 2025-08-26T20:27:50.3186095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3186512Z outputs = self.roberta( 2025-08-26T20:27:50.3186933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3187344Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3187742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3188153Z layer_outputs = layer_module( 2025-08-26T20:27:50.3188509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3188880Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3189300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3189727Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3190157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3190559Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3191010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3191485Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3191925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3192337Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3192502Z 2025-08-26T20:27:50.3192606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3192965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3193285Z return mod(**inputs) 2025-08-26T20:27:50.3193658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3194059Z outputs = self.roberta( 2025-08-26T20:27:50.3194453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3194854Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3195249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3195655Z layer_outputs = layer_module( 2025-08-26T20:27:50.3196014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3196526Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3196952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3197368Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3197786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3198221Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3198688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3199262Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3199756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3200236Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3200695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3201069Z return self.act(input) 2025-08-26T20:27:50.3201192Z 2025-08-26T20:27:50.3201348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3201734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3202058Z return mod(**inputs) 2025-08-26T20:27:50.3202440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3202841Z outputs = self.roberta( 2025-08-26T20:27:50.3203215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3203621Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3204019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3204418Z layer_outputs = layer_module( 2025-08-26T20:27:50.3204761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3205119Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3205526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3205971Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3206374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3206755Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3207189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3207695Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3208141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3208552Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3208687Z 2025-08-26T20:27:50.3208791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3209150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3209473Z return mod(**inputs) 2025-08-26T20:27:50.3209852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3210245Z outputs = self.roberta( 2025-08-26T20:27:50.3210618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3211028Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3211439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3211855Z layer_outputs = layer_module( 2025-08-26T20:27:50.3212208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3212563Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3212967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3213374Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3213756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3214118Z return func(*args, **kwargs) 2025-08-26T20:27:50.3214510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3214928Z self_outputs = self.self( 2025-08-26T20:27:50.3215288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3215715Z return func(*args, **kwargs) 2025-08-26T20:27:50.3216106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3216662Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3216947Z 2025-08-26T20:27:50.3217050Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3217408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3217735Z return mod(**inputs) 2025-08-26T20:27:50.3218114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3218514Z outputs = self.roberta( 2025-08-26T20:27:50.3218892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3219295Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3219683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3220106Z layer_outputs = layer_module( 2025-08-26T20:27:50.3220450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3220808Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3221215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3221638Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3222019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3222386Z return func(*args, **kwargs) 2025-08-26T20:27:50.3222771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3223170Z self_outputs = self.self( 2025-08-26T20:27:50.3223519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3223882Z return func(*args, **kwargs) 2025-08-26T20:27:50.3224274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3224681Z self.key(current_states) 2025-08-26T20:27:50.3224796Z 2025-08-26T20:27:50.3224904Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3225271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3225601Z return mod(**inputs) 2025-08-26T20:27:50.3225992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3226395Z outputs = self.roberta( 2025-08-26T20:27:50.3226781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3227183Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3227577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3227974Z layer_outputs = layer_module( 2025-08-26T20:27:50.3228316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3228667Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3229093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3229501Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3229897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3230261Z return func(*args, **kwargs) 2025-08-26T20:27:50.3230666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3231070Z self_outputs = self.self( 2025-08-26T20:27:50.3231436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3231805Z return func(*args, **kwargs) 2025-08-26T20:27:50.3232195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3232610Z self.value(current_states) 2025-08-26T20:27:50.3232736Z 2025-08-26T20:27:50.3232823Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3233072Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3233438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3233796Z return mod(**inputs) 2025-08-26T20:27:50.3234188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3234599Z outputs = self.roberta( 2025-08-26T20:27:50.3234993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3235430Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3235894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3236333Z layer_outputs = layer_module( 2025-08-26T20:27:50.3236707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3237105Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3237537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3237992Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3238400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3238776Z return func(*args, **kwargs) 2025-08-26T20:27:50.3239168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3239690Z self_outputs = self.self( 2025-08-26T20:27:50.3240099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3240511Z return func(*args, **kwargs) 2025-08-26T20:27:50.3240939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3241429Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3241642Z 2025-08-26T20:27:50.3241747Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3242115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3242450Z return mod(**inputs) 2025-08-26T20:27:50.3242845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3243251Z outputs = self.roberta( 2025-08-26T20:27:50.3243683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3244106Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3244606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3245042Z layer_outputs = layer_module( 2025-08-26T20:27:50.3245421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3245791Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3246214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3246635Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3247016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3247393Z return func(*args, **kwargs) 2025-08-26T20:27:50.3247796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3248268Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3248739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3249184Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3249332Z 2025-08-26T20:27:50.3249439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3249811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3250144Z return mod(**inputs) 2025-08-26T20:27:50.3250531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3250963Z outputs = self.roberta( 2025-08-26T20:27:50.3251353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3251766Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3252175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3252577Z layer_outputs = layer_module( 2025-08-26T20:27:50.3253180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3253545Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3253961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3254380Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3254788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3255192Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3255639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3256163Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3256648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3257091Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3257246Z 2025-08-26T20:27:50.3257359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3257749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3258080Z return mod(**inputs) 2025-08-26T20:27:50.3258490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3258949Z outputs = self.roberta( 2025-08-26T20:27:50.3259367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3259827Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3260261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3260693Z layer_outputs = layer_module( 2025-08-26T20:27:50.3261070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3261456Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3261896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3262340Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3262770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3263195Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3263665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3264213Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3264707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3265181Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3265590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3265956Z return self.act(input) 2025-08-26T20:27:50.3266100Z 2025-08-26T20:27:50.3266221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3266610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3266963Z return mod(**inputs) 2025-08-26T20:27:50.3267380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3267821Z outputs = self.roberta( 2025-08-26T20:27:50.3268239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3268680Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3269096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3269506Z layer_outputs = layer_module( 2025-08-26T20:27:50.3269862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3270232Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3270665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3271117Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3271554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3271977Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3272436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3272967Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3273460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3273910Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3274058Z 2025-08-26T20:27:50.3274222Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3274601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3274976Z return mod(**inputs) 2025-08-26T20:27:50.3275411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3275867Z outputs = self.roberta( 2025-08-26T20:27:50.3276296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3276751Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3277201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3277640Z layer_outputs = layer_module( 2025-08-26T20:27:50.3278021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3278431Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3278885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3279430Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3279902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3280318Z return func(*args, **kwargs) 2025-08-26T20:27:50.3280751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3281183Z self_outputs = self.self( 2025-08-26T20:27:50.3281553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3281979Z return func(*args, **kwargs) 2025-08-26T20:27:50.3282417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3282958Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3283232Z 2025-08-26T20:27:50.3283340Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3283717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3284065Z return mod(**inputs) 2025-08-26T20:27:50.3284474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3284874Z outputs = self.roberta( 2025-08-26T20:27:50.3285282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3285717Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3286208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3286640Z layer_outputs = layer_module( 2025-08-26T20:27:50.3287016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3287406Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3287846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3288300Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3288702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3289105Z return func(*args, **kwargs) 2025-08-26T20:27:50.3289555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3289990Z self_outputs = self.self( 2025-08-26T20:27:50.3290391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3290784Z return func(*args, **kwargs) 2025-08-26T20:27:50.3291210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3291641Z self.key(current_states) 2025-08-26T20:27:50.3291762Z 2025-08-26T20:27:50.3291880Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3292266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3292613Z return mod(**inputs) 2025-08-26T20:27:50.3293029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3293462Z outputs = self.roberta( 2025-08-26T20:27:50.3293878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3294305Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3294738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3295197Z layer_outputs = layer_module( 2025-08-26T20:27:50.3295569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3295959Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3296574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3297069Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3297466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3297850Z return func(*args, **kwargs) 2025-08-26T20:27:50.3298248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3298662Z self_outputs = self.self( 2025-08-26T20:27:50.3299037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3299419Z return func(*args, **kwargs) 2025-08-26T20:27:50.3299821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3300231Z self.value(current_states) 2025-08-26T20:27:50.3300364Z 2025-08-26T20:27:50.3300450Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3300702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3301078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3301433Z return mod(**inputs) 2025-08-26T20:27:50.3302679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3303522Z outputs = self.roberta( 2025-08-26T20:27:50.3304186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3305111Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3305578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3306032Z layer_outputs = layer_module( 2025-08-26T20:27:50.3306431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3306911Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3307619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3308135Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3308661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3309088Z return func(*args, **kwargs) 2025-08-26T20:27:50.3309526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3309967Z self_outputs = self.self( 2025-08-26T20:27:50.3310374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3310779Z return func(*args, **kwargs) 2025-08-26T20:27:50.3311218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3311733Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3311945Z 2025-08-26T20:27:50.3312073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3312482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3312905Z return mod(**inputs) 2025-08-26T20:27:50.3313333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3313777Z outputs = self.roberta( 2025-08-26T20:27:50.3314249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3314878Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3315471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3315967Z layer_outputs = layer_module( 2025-08-26T20:27:50.3316357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3316776Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3317242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3317717Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3318151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3318566Z return func(*args, **kwargs) 2025-08-26T20:27:50.3319013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3319623Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3320146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3320609Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3320768Z 2025-08-26T20:27:50.3320891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3321300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3321680Z return mod(**inputs) 2025-08-26T20:27:50.3322108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3322550Z outputs = self.roberta( 2025-08-26T20:27:50.3322995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3323449Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3323952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3324416Z layer_outputs = layer_module( 2025-08-26T20:27:50.3324817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3325222Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3325682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3326146Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3326598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3327034Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3327535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3328063Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3328503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3328908Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3329041Z 2025-08-26T20:27:50.3329167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3329518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3329832Z return mod(**inputs) 2025-08-26T20:27:50.3330205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3330594Z outputs = self.roberta( 2025-08-26T20:27:50.3330981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3331448Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3331860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3332268Z layer_outputs = layer_module( 2025-08-26T20:27:50.3332617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3332988Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3333405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3333833Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3334252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3334641Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3335079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3335569Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3336019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3336458Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3336834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3337177Z return self.act(input) 2025-08-26T20:27:50.3337291Z 2025-08-26T20:27:50.3337407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3337777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3338107Z return mod(**inputs) 2025-08-26T20:27:50.3338533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3338936Z outputs = self.roberta( 2025-08-26T20:27:50.3339336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3339742Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3340133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3340545Z layer_outputs = layer_module( 2025-08-26T20:27:50.3340899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3341275Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3341690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3342120Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3342518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3342905Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3343337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3343845Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3344359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3344798Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3344939Z 2025-08-26T20:27:50.3345061Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3345431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3345778Z return mod(**inputs) 2025-08-26T20:27:50.3346173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3346594Z outputs = self.roberta( 2025-08-26T20:27:50.3346983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3347391Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3347785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3348187Z layer_outputs = layer_module( 2025-08-26T20:27:50.3348543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3348923Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3349328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3349745Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3350130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3350503Z return func(*args, **kwargs) 2025-08-26T20:27:50.3350894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3351293Z self_outputs = self.self( 2025-08-26T20:27:50.3351652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3352026Z return func(*args, **kwargs) 2025-08-26T20:27:50.3352416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3352961Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3353241Z 2025-08-26T20:27:50.3353346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3353733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3354064Z return mod(**inputs) 2025-08-26T20:27:50.3354457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3354868Z outputs = self.roberta( 2025-08-26T20:27:50.3355259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3355703Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3356147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3356586Z layer_outputs = layer_module( 2025-08-26T20:27:50.3356956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3357343Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3357795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3358262Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3358674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3359077Z return func(*args, **kwargs) 2025-08-26T20:27:50.3359886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3360358Z self_outputs = self.self( 2025-08-26T20:27:50.3360793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3361187Z return func(*args, **kwargs) 2025-08-26T20:27:50.3361596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3361993Z self.key(current_states) 2025-08-26T20:27:50.3362116Z 2025-08-26T20:27:50.3362236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3362629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3362978Z return mod(**inputs) 2025-08-26T20:27:50.3363393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3363823Z outputs = self.roberta( 2025-08-26T20:27:50.3364236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3364674Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3365099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3365536Z layer_outputs = layer_module( 2025-08-26T20:27:50.3365908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3366298Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3366733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3367175Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3367583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3367979Z return func(*args, **kwargs) 2025-08-26T20:27:50.3368406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3368847Z self_outputs = self.self( 2025-08-26T20:27:50.3369237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3369655Z return func(*args, **kwargs) 2025-08-26T20:27:50.3370077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3370518Z self.value(current_states) 2025-08-26T20:27:50.3370645Z 2025-08-26T20:27:50.3370734Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3370997Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3371388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3371737Z return mod(**inputs) 2025-08-26T20:27:50.3372143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3372582Z outputs = self.roberta( 2025-08-26T20:27:50.3372998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3373438Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3373870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3374319Z layer_outputs = layer_module( 2025-08-26T20:27:50.3374695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3375080Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3375528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3376001Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3376426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3376834Z return func(*args, **kwargs) 2025-08-26T20:27:50.3377273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3377724Z self_outputs = self.self( 2025-08-26T20:27:50.3378106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3378510Z return func(*args, **kwargs) 2025-08-26T20:27:50.3378932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3379456Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3379657Z 2025-08-26T20:27:50.3379777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3380158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3380505Z return mod(**inputs) 2025-08-26T20:27:50.3380924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3381367Z outputs = self.roberta( 2025-08-26T20:27:50.3381779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3382232Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3382675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3383113Z layer_outputs = layer_module( 2025-08-26T20:27:50.3383486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3383863Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3384334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3384792Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3385221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3385630Z return func(*args, **kwargs) 2025-08-26T20:27:50.3386042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3386549Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3387050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3387510Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3387660Z 2025-08-26T20:27:50.3387778Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3388162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3388522Z return mod(**inputs) 2025-08-26T20:27:50.3388938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3389415Z outputs = self.roberta( 2025-08-26T20:27:50.3389829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3390259Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3390696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3391137Z layer_outputs = layer_module( 2025-08-26T20:27:50.3391531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3391926Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3392378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3392840Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3393278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3393703Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3394188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3394714Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3395205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3395658Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3395811Z 2025-08-26T20:27:50.3395924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3396617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3397045Z return mod(**inputs) 2025-08-26T20:27:50.3397470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3397923Z outputs = self.roberta( 2025-08-26T20:27:50.3398347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3398807Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3399301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3399800Z layer_outputs = layer_module( 2025-08-26T20:27:50.3400244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3400641Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3401185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3401655Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3402105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3402535Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3403018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3403555Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3404061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3404550Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3404966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3405348Z return self.act(input) 2025-08-26T20:27:50.3405514Z 2025-08-26T20:27:50.3405632Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3406034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3406390Z return mod(**inputs) 2025-08-26T20:27:50.3406812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3407264Z outputs = self.roberta( 2025-08-26T20:27:50.3407687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3408174Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3408613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3409064Z layer_outputs = layer_module( 2025-08-26T20:27:50.3409458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3409845Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3410282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3410695Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3411101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3411504Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3411952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3412459Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3412923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3413348Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3413495Z 2025-08-26T20:27:50.3413603Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3413969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3414039Z return mod(**inputs) 2025-08-26T20:27:50.3414328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3414401Z outputs = self.roberta( 2025-08-26T20:27:50.3414693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3414779Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3415075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3415158Z layer_outputs = layer_module( 2025-08-26T20:27:50.3415386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3415468Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3415755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3415840Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3416097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3416175Z return func(*args, **kwargs) 2025-08-26T20:27:50.3416455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3416528Z self_outputs = self.self( 2025-08-26T20:27:50.3416778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3416878Z return func(*args, **kwargs) 2025-08-26T20:27:50.3417161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3417383Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3417386Z 2025-08-26T20:27:50.3417492Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3417718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3417795Z return mod(**inputs) 2025-08-26T20:27:50.3418082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3418164Z outputs = self.roberta( 2025-08-26T20:27:50.3418445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3418529Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3418804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3418881Z layer_outputs = layer_module( 2025-08-26T20:27:50.3419114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3419200Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3419489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3419574Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3419823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3419906Z return func(*args, **kwargs) 2025-08-26T20:27:50.3420189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3420272Z self_outputs = self.self( 2025-08-26T20:27:50.3420519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3420593Z return func(*args, **kwargs) 2025-08-26T20:27:50.3420881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3420960Z self.key(current_states) 2025-08-26T20:27:50.3420963Z 2025-08-26T20:27:50.3421096Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3421301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3421392Z return mod(**inputs) 2025-08-26T20:27:50.3421672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3421745Z outputs = self.roberta( 2025-08-26T20:27:50.3422035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3422127Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3422415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3422489Z layer_outputs = layer_module( 2025-08-26T20:27:50.3422716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3422802Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3423075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3423164Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3423428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3423507Z return func(*args, **kwargs) 2025-08-26T20:27:50.3423784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3423855Z self_outputs = self.self( 2025-08-26T20:27:50.3424106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3424189Z return func(*args, **kwargs) 2025-08-26T20:27:50.3424464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3424535Z self.value(current_states) 2025-08-26T20:27:50.3424540Z 2025-08-26T20:27:50.3424620Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3424732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3424923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3424993Z return mod(**inputs) 2025-08-26T20:27:50.3425258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3425323Z outputs = self.roberta( 2025-08-26T20:27:50.3425591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3425664Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3425933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3426001Z layer_outputs = layer_module( 2025-08-26T20:27:50.3426222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3426298Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3426561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3426650Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3426881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3426954Z return func(*args, **kwargs) 2025-08-26T20:27:50.3427237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3427307Z self_outputs = self.self( 2025-08-26T20:27:50.3427549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3427644Z return func(*args, **kwargs) 2025-08-26T20:27:50.3427916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3428046Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3428050Z 2025-08-26T20:27:50.3428158Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3428350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3428418Z return mod(**inputs) 2025-08-26T20:27:50.3428696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3428768Z outputs = self.roberta( 2025-08-26T20:27:50.3429042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3429116Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3429444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3429545Z layer_outputs = layer_module( 2025-08-26T20:27:50.3429761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3429846Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3430117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3430215Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3430461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3430528Z return func(*args, **kwargs) 2025-08-26T20:27:50.3430804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3430932Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3431211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3431298Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3431301Z 2025-08-26T20:27:50.3431407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3431615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3431683Z return mod(**inputs) 2025-08-26T20:27:50.3431974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3432043Z outputs = self.roberta( 2025-08-26T20:27:50.3432314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3432393Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3432663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3432741Z layer_outputs = layer_module( 2025-08-26T20:27:50.3432955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3433038Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3433312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3433400Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3433687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3433787Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3434105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3434230Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3434505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3434596Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3434600Z 2025-08-26T20:27:50.3434704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3434916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3434983Z return mod(**inputs) 2025-08-26T20:27:50.3435271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3435341Z outputs = self.roberta( 2025-08-26T20:27:50.3435618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3435720Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3435997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3436075Z layer_outputs = layer_module( 2025-08-26T20:27:50.3436301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3436379Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3436682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3436766Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3437042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3437302Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3437651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3437783Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3438090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3438220Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3438449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3438535Z return self.act(input) 2025-08-26T20:27:50.3438540Z 2025-08-26T20:27:50.3438652Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3438865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3438945Z return mod(**inputs) 2025-08-26T20:27:50.3439292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3439384Z outputs = self.roberta( 2025-08-26T20:27:50.3439695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3439776Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3440092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3440174Z layer_outputs = layer_module( 2025-08-26T20:27:50.3440449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3440540Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3440872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3440959Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3441214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3441299Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3441636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3441793Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3442095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3442186Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3442197Z 2025-08-26T20:27:50.3442312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3442528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3442634Z return mod(**inputs) 2025-08-26T20:27:50.3442943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3443026Z outputs = self.roberta( 2025-08-26T20:27:50.3443335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3443416Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3443746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3443824Z layer_outputs = layer_module( 2025-08-26T20:27:50.3444075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3444166Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3444465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3444567Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3444837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3444922Z return func(*args, **kwargs) 2025-08-26T20:27:50.3445222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3445312Z self_outputs = self.self( 2025-08-26T20:27:50.3445581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3445660Z return func(*args, **kwargs) 2025-08-26T20:27:50.3445972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3446209Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3446213Z 2025-08-26T20:27:50.3446334Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3446555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3446627Z return mod(**inputs) 2025-08-26T20:27:50.3446944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3447023Z outputs = self.roberta( 2025-08-26T20:27:50.3447358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3447440Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3447766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3447850Z layer_outputs = layer_module( 2025-08-26T20:27:50.3448097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3448198Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3448465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3448554Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3448794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3448864Z return func(*args, **kwargs) 2025-08-26T20:27:50.3449145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3449218Z self_outputs = self.self( 2025-08-26T20:27:50.3449469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3449561Z return func(*args, **kwargs) 2025-08-26T20:27:50.3449838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3449918Z self.key(current_states) 2025-08-26T20:27:50.3449922Z 2025-08-26T20:27:50.3450027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3450234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3450320Z return mod(**inputs) 2025-08-26T20:27:50.3450607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3450678Z outputs = self.roberta( 2025-08-26T20:27:50.3450955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3451039Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3451311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3451390Z layer_outputs = layer_module( 2025-08-26T20:27:50.3451611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3451692Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3451976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3452065Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3452327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3452403Z return func(*args, **kwargs) 2025-08-26T20:27:50.3452695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3452780Z self_outputs = self.self( 2025-08-26T20:27:50.3453023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3453102Z return func(*args, **kwargs) 2025-08-26T20:27:50.3453380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3453469Z self.value(current_states) 2025-08-26T20:27:50.3453473Z 2025-08-26T20:27:50.3453591Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3453707Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3453929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3454027Z return mod(**inputs) 2025-08-26T20:27:50.3454319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3454393Z outputs = self.roberta( 2025-08-26T20:27:50.3454674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3454755Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3455029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3455112Z layer_outputs = layer_module( 2025-08-26T20:27:50.3455338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3455428Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3455708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3455813Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3456063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3456133Z return func(*args, **kwargs) 2025-08-26T20:27:50.3456418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3456499Z self_outputs = self.self( 2025-08-26T20:27:50.3456736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3456831Z return func(*args, **kwargs) 2025-08-26T20:27:50.3457104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3457248Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3457252Z 2025-08-26T20:27:50.3457355Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3457557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3457623Z return mod(**inputs) 2025-08-26T20:27:50.3457895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3457970Z outputs = self.roberta( 2025-08-26T20:27:50.3458241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3458321Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3458588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3458658Z layer_outputs = layer_module( 2025-08-26T20:27:50.3458883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3458962Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3459240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3459319Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3459557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3459631Z return func(*args, **kwargs) 2025-08-26T20:27:50.3459922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3460058Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3460342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3460434Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3460439Z 2025-08-26T20:27:50.3460542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3460739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3460813Z return mod(**inputs) 2025-08-26T20:27:50.3461083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3461159Z outputs = self.roberta( 2025-08-26T20:27:50.3461430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3461504Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3461783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3461854Z layer_outputs = layer_module( 2025-08-26T20:27:50.3462100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3462198Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3462473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3462558Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3462816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3462939Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3463241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3463368Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3463644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3463725Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3463729Z 2025-08-26T20:27:50.3463841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3464037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3464110Z return mod(**inputs) 2025-08-26T20:27:50.3464383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3464460Z outputs = self.roberta( 2025-08-26T20:27:50.3464727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3464799Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3465076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3465148Z layer_outputs = layer_module( 2025-08-26T20:27:50.3465371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3465461Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3465730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3465821Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3466077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3466178Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3466503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3466631Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3466899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3467011Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3467231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3467301Z return self.act(input) 2025-08-26T20:27:50.3467305Z 2025-08-26T20:27:50.3467415Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3467614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3467681Z return mod(**inputs) 2025-08-26T20:27:50.3467959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3468029Z outputs = self.roberta( 2025-08-26T20:27:50.3468306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3468399Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3468675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3468753Z layer_outputs = layer_module( 2025-08-26T20:27:50.3468980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3469090Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3469368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3469458Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3469722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3469799Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3470115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3470252Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3470540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3470623Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3470628Z 2025-08-26T20:27:50.3470739Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3470941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3471008Z return mod(**inputs) 2025-08-26T20:27:50.3471296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3471365Z outputs = self.roberta( 2025-08-26T20:27:50.3471645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3471716Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3471989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3472069Z layer_outputs = layer_module( 2025-08-26T20:27:50.3472289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3472419Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3472721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3472830Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3473110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3473187Z return func(*args, **kwargs) 2025-08-26T20:27:50.3473490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3473566Z self_outputs = self.self( 2025-08-26T20:27:50.3473840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3473916Z return func(*args, **kwargs) 2025-08-26T20:27:50.3474218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3474451Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3474455Z 2025-08-26T20:27:50.3474568Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3474790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3474882Z return mod(**inputs) 2025-08-26T20:27:50.3475180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3475259Z outputs = self.roberta( 2025-08-26T20:27:50.3475562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3475680Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3475974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3476057Z layer_outputs = layer_module( 2025-08-26T20:27:50.3476296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3476380Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3476683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3476770Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3477045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3477121Z return func(*args, **kwargs) 2025-08-26T20:27:50.3477443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3477529Z self_outputs = self.self( 2025-08-26T20:27:50.3477793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3477875Z return func(*args, **kwargs) 2025-08-26T20:27:50.3478178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3478256Z self.key(current_states) 2025-08-26T20:27:50.3478268Z 2025-08-26T20:27:50.3478383Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3478596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3478674Z return mod(**inputs) 2025-08-26T20:27:50.3478972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3479059Z outputs = self.roberta( 2025-08-26T20:27:50.3479463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3479549Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3479870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3479949Z layer_outputs = layer_module( 2025-08-26T20:27:50.3480195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3480278Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3480581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3480681Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3480940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3481025Z return func(*args, **kwargs) 2025-08-26T20:27:50.3481321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3481398Z self_outputs = self.self( 2025-08-26T20:27:50.3481665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3481761Z return func(*args, **kwargs) 2025-08-26T20:27:50.3482066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3482141Z self.value(current_states) 2025-08-26T20:27:50.3482144Z 2025-08-26T20:27:50.3482235Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3482339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3482561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3482639Z return mod(**inputs) 2025-08-26T20:27:50.3482939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3483021Z outputs = self.roberta( 2025-08-26T20:27:50.3483314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3483394Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3483696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3483773Z layer_outputs = layer_module( 2025-08-26T20:27:50.3484015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3484099Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3484401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3484489Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3484749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3484831Z return func(*args, **kwargs) 2025-08-26T20:27:50.3485127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3485207Z self_outputs = self.self( 2025-08-26T20:27:50.3485463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3485538Z return func(*args, **kwargs) 2025-08-26T20:27:50.3485835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3485973Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3486001Z 2025-08-26T20:27:50.3486114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3486325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3486416Z return mod(**inputs) 2025-08-26T20:27:50.3486721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3486798Z outputs = self.roberta( 2025-08-26T20:27:50.3487094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3487172Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3487475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3487554Z layer_outputs = layer_module( 2025-08-26T20:27:50.3487793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3487887Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3488187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3488304Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3488561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3488634Z return func(*args, **kwargs) 2025-08-26T20:27:50.3488932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3489070Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3489393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3489484Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3489488Z 2025-08-26T20:27:50.3489603Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3489817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3489888Z return mod(**inputs) 2025-08-26T20:27:50.3490191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3490267Z outputs = self.roberta( 2025-08-26T20:27:50.3490561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3490638Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3490928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3491016Z layer_outputs = layer_module( 2025-08-26T20:27:50.3491252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3491343Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3491633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3491734Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3492014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3492095Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3492429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3492560Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3492876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3492965Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3492969Z 2025-08-26T20:27:50.3493099Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3493323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3493394Z return mod(**inputs) 2025-08-26T20:27:50.3493696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3493771Z outputs = self.roberta( 2025-08-26T20:27:50.3494066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3494147Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3494437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3494522Z layer_outputs = layer_module( 2025-08-26T20:27:50.3494760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3494851Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3495180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3495269Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3495558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3495640Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3495975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3496126Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3496778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3496935Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3497170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3497261Z return self.act(input) 2025-08-26T20:27:50.3497265Z 2025-08-26T20:27:50.3497376Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3497595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3497667Z return mod(**inputs) 2025-08-26T20:27:50.3497957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3498044Z outputs = self.roberta( 2025-08-26T20:27:50.3498337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3498426Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3498715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3498794Z layer_outputs = layer_module( 2025-08-26T20:27:50.3499035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3499119Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3499421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3499510Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3499850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3499934Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3500297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3500453Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3500746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3500842Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3500846Z 2025-08-26T20:27:50.3500956Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3501171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3501253Z return mod(**inputs) 2025-08-26T20:27:50.3501549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3501631Z outputs = self.roberta( 2025-08-26T20:27:50.3501926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3502013Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3502351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3502428Z layer_outputs = layer_module( 2025-08-26T20:27:50.3502672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3502758Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3503065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3503183Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3503447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3503532Z return func(*args, **kwargs) 2025-08-26T20:27:50.3503823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3503906Z self_outputs = self.self( 2025-08-26T20:27:50.3504164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3504238Z return func(*args, **kwargs) 2025-08-26T20:27:50.3504546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3504770Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3504775Z 2025-08-26T20:27:50.3504901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3505096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3505172Z return mod(**inputs) 2025-08-26T20:27:50.3505445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3505515Z outputs = self.roberta( 2025-08-26T20:27:50.3505797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3505869Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3506154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3506226Z layer_outputs = layer_module( 2025-08-26T20:27:50.3506449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3506554Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3506831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3506940Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3507185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3507264Z return func(*args, **kwargs) 2025-08-26T20:27:50.3507540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3507611Z self_outputs = self.self( 2025-08-26T20:27:50.3507860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3507931Z return func(*args, **kwargs) 2025-08-26T20:27:50.3508218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3508289Z self.key(current_states) 2025-08-26T20:27:50.3508292Z 2025-08-26T20:27:50.3508398Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3508642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3508731Z return mod(**inputs) 2025-08-26T20:27:50.3509017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3509087Z outputs = self.roberta( 2025-08-26T20:27:50.3509367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3509439Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3509759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3509840Z layer_outputs = layer_module( 2025-08-26T20:27:50.3510062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3510148Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3510430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3510521Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3510767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3510835Z return func(*args, **kwargs) 2025-08-26T20:27:50.3511112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3511186Z self_outputs = self.self( 2025-08-26T20:27:50.3511433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3511511Z return func(*args, **kwargs) 2025-08-26T20:27:50.3511785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3511869Z self.value(current_states) 2025-08-26T20:27:50.3511872Z 2025-08-26T20:27:50.3511957Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3512071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3512276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3512344Z return mod(**inputs) 2025-08-26T20:27:50.3512629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3512700Z outputs = self.roberta( 2025-08-26T20:27:50.3513008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3513094Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3513383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3513466Z layer_outputs = layer_module( 2025-08-26T20:27:50.3513683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3513766Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3514033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3514113Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3514359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3514429Z return func(*args, **kwargs) 2025-08-26T20:27:50.3514710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3514782Z self_outputs = self.self( 2025-08-26T20:27:50.3515034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3515123Z return func(*args, **kwargs) 2025-08-26T20:27:50.3515405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3515552Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3515555Z 2025-08-26T20:27:50.3515660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3515888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3515956Z return mod(**inputs) 2025-08-26T20:27:50.3516238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3516315Z outputs = self.roberta( 2025-08-26T20:27:50.3516592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3516673Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3516949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3517029Z layer_outputs = layer_module( 2025-08-26T20:27:50.3517256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3517338Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3517626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3517711Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3517976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3518050Z return func(*args, **kwargs) 2025-08-26T20:27:50.3518342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3518488Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3518778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3518874Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3518878Z 2025-08-26T20:27:50.3518996Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3519297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3519380Z return mod(**inputs) 2025-08-26T20:27:50.3519696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3519781Z outputs = self.roberta( 2025-08-26T20:27:50.3520079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3520167Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3520476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3520556Z layer_outputs = layer_module( 2025-08-26T20:27:50.3520810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3520900Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3521206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3521308Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3521572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3521680Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3521997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3522127Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3522394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3522517Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3522521Z 2025-08-26T20:27:50.3522635Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3522854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3522933Z return mod(**inputs) 2025-08-26T20:27:50.3523240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3523323Z outputs = self.roberta( 2025-08-26T20:27:50.3523624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3523703Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3524021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3524099Z layer_outputs = layer_module( 2025-08-26T20:27:50.3524350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3524438Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3524745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3524838Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3525125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3525217Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3525548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3525687Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3525997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3526123Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3526391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3526471Z return self.act(input) 2025-08-26T20:27:50.3526507Z 2025-08-26T20:27:50.3526629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3526851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3526930Z return mod(**inputs) 2025-08-26T20:27:50.3527234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3527309Z outputs = self.roberta( 2025-08-26T20:27:50.3527618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3527699Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3528006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3528085Z layer_outputs = layer_module( 2025-08-26T20:27:50.3528329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3528423Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3528746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3528846Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3529135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3529224Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3529585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3529733Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3530044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3530133Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3530138Z 2025-08-26T20:27:50.3530261Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3530478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3530550Z return mod(**inputs) 2025-08-26T20:27:50.3530860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3530934Z outputs = self.roberta( 2025-08-26T20:27:50.3531242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3531317Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3531599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3531672Z layer_outputs = layer_module( 2025-08-26T20:27:50.3531899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3531986Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3532257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3532346Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3532590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3532663Z return func(*args, **kwargs) 2025-08-26T20:27:50.3532963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3533035Z self_outputs = self.self( 2025-08-26T20:27:50.3533307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3533379Z return func(*args, **kwargs) 2025-08-26T20:27:50.3533653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3533872Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3533875Z 2025-08-26T20:27:50.3533981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3534189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3534258Z return mod(**inputs) 2025-08-26T20:27:50.3534547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3534617Z outputs = self.roberta( 2025-08-26T20:27:50.3534893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3534975Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3535270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3535352Z layer_outputs = layer_module( 2025-08-26T20:27:50.3535573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3535652Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3535935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3536039Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3536289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3536360Z return func(*args, **kwargs) 2025-08-26T20:27:50.3536642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3536714Z self_outputs = self.self( 2025-08-26T20:27:50.3536955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3537034Z return func(*args, **kwargs) 2025-08-26T20:27:50.3537311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3537392Z self.key(current_states) 2025-08-26T20:27:50.3537395Z 2025-08-26T20:27:50.3537500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3537700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3537775Z return mod(**inputs) 2025-08-26T20:27:50.3538055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3538130Z outputs = self.roberta( 2025-08-26T20:27:50.3538404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3538484Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3538758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3538829Z layer_outputs = layer_module( 2025-08-26T20:27:50.3539060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3539156Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3539435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3539550Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3539790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3539869Z return func(*args, **kwargs) 2025-08-26T20:27:50.3540137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3540212Z self_outputs = self.self( 2025-08-26T20:27:50.3540451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3540520Z return func(*args, **kwargs) 2025-08-26T20:27:50.3540796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3540869Z self.value(current_states) 2025-08-26T20:27:50.3540872Z 2025-08-26T20:27:50.3540961Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3541065Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3541266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3541352Z return mod(**inputs) 2025-08-26T20:27:50.3541630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3541706Z outputs = self.roberta( 2025-08-26T20:27:50.3541978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3542890Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3543172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3543247Z layer_outputs = layer_module( 2025-08-26T20:27:50.3543484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3543565Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3543854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3543938Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3544192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3544276Z return func(*args, **kwargs) 2025-08-26T20:27:50.3544572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3544659Z self_outputs = self.self( 2025-08-26T20:27:50.3544920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3545002Z return func(*args, **kwargs) 2025-08-26T20:27:50.3545299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3545443Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3545447Z 2025-08-26T20:27:50.3545565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3545779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3545866Z return mod(**inputs) 2025-08-26T20:27:50.3546147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3546217Z outputs = self.roberta( 2025-08-26T20:27:50.3546528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3546603Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3546899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3546975Z layer_outputs = layer_module( 2025-08-26T20:27:50.3547207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3547287Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3547564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3547654Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3547908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3547984Z return func(*args, **kwargs) 2025-08-26T20:27:50.3548253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3548380Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3548658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3548764Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3548767Z 2025-08-26T20:27:50.3548877Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3549075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3549150Z return mod(**inputs) 2025-08-26T20:27:50.3549456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3549529Z outputs = self.roberta( 2025-08-26T20:27:50.3549810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3549886Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3550167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3550240Z layer_outputs = layer_module( 2025-08-26T20:27:50.3550464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3550551Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3550826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3550921Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3551185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3551263Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3551580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3551704Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3551984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3552067Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3552070Z 2025-08-26T20:27:50.3552181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3552382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3552450Z return mod(**inputs) 2025-08-26T20:27:50.3552777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3552852Z outputs = self.roberta( 2025-08-26T20:27:50.3553169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3553247Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3553542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3553624Z layer_outputs = layer_module( 2025-08-26T20:27:50.3553863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3553957Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3554261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3554361Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3554664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3554746Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3555084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3555235Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3555533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3555654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3555883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3555990Z return self.act(input) 2025-08-26T20:27:50.3555994Z 2025-08-26T20:27:50.3556107Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3556329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3556400Z return mod(**inputs) 2025-08-26T20:27:50.3556709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3556783Z outputs = self.roberta( 2025-08-26T20:27:50.3557081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3557166Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3557460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3557545Z layer_outputs = layer_module( 2025-08-26T20:27:50.3557784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3557871Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3558177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3558266Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3558563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3558647Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3558997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3559145Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3559517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3559662Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3559667Z 2025-08-26T20:27:50.3559784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3560034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3560112Z return mod(**inputs) 2025-08-26T20:27:50.3560421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3560505Z outputs = self.roberta( 2025-08-26T20:27:50.3560811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3560901Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3561223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3561310Z layer_outputs = layer_module( 2025-08-26T20:27:50.3561553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3561638Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3561947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3562061Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3562328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3562403Z return func(*args, **kwargs) 2025-08-26T20:27:50.3562758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3562841Z self_outputs = self.self( 2025-08-26T20:27:50.3563130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3563213Z return func(*args, **kwargs) 2025-08-26T20:27:50.3563502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-26T20:27:50.3563723Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:27:50.3563735Z 2025-08-26T20:27:50.3563847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3564060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3564140Z return mod(**inputs) 2025-08-26T20:27:50.3564433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3564513Z outputs = self.roberta( 2025-08-26T20:27:50.3564809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3564886Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3565190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3565267Z layer_outputs = layer_module( 2025-08-26T20:27:50.3565514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3565596Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3565889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3565982Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3566239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3566321Z return func(*args, **kwargs) 2025-08-26T20:27:50.3566634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3566718Z self_outputs = self.self( 2025-08-26T20:27:50.3567003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3567077Z return func(*args, **kwargs) 2025-08-26T20:27:50.3567452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-26T20:27:50.3567530Z self.key(current_states) 2025-08-26T20:27:50.3567534Z 2025-08-26T20:27:50.3567645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3567843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3567913Z return mod(**inputs) 2025-08-26T20:27:50.3568200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3568268Z outputs = self.roberta( 2025-08-26T20:27:50.3568554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3568628Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3568925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3569006Z layer_outputs = layer_module( 2025-08-26T20:27:50.3569235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3569326Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3569617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3569729Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3569968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3570039Z return func(*args, **kwargs) 2025-08-26T20:27:50.3570317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3570391Z self_outputs = self.self( 2025-08-26T20:27:50.3570637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3570710Z return func(*args, **kwargs) 2025-08-26T20:27:50.3570988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-26T20:27:50.3571071Z self.value(current_states) 2025-08-26T20:27:50.3571076Z 2025-08-26T20:27:50.3571162Z cudagraph partition due to non gpu ops 2025-08-26T20:27:50.3571277Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3571481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3571558Z return mod(**inputs) 2025-08-26T20:27:50.3571840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3571915Z outputs = self.roberta( 2025-08-26T20:27:50.3572201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3572277Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3572560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3572637Z layer_outputs = layer_module( 2025-08-26T20:27:50.3572876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3572991Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3573292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3573406Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3573678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3573755Z return func(*args, **kwargs) 2025-08-26T20:27:50.3574068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-26T20:27:50.3574140Z self_outputs = self.self( 2025-08-26T20:27:50.3574393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3574464Z return func(*args, **kwargs) 2025-08-26T20:27:50.3574768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-26T20:27:50.3574911Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:27:50.3574915Z 2025-08-26T20:27:50.3575027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3575246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3575337Z return mod(**inputs) 2025-08-26T20:27:50.3575648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3575723Z outputs = self.roberta( 2025-08-26T20:27:50.3576022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3576130Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3576404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3576483Z layer_outputs = layer_module( 2025-08-26T20:27:50.3576706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3576791Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3577066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-26T20:27:50.3577148Z self_attention_outputs = self.attention( 2025-08-26T20:27:50.3577400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:27:50.3577470Z return func(*args, **kwargs) 2025-08-26T20:27:50.3577752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-26T20:27:50.3577886Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:27:50.3578162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-26T20:27:50.3578255Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3578258Z 2025-08-26T20:27:50.3578368Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3578592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3578662Z return mod(**inputs) 2025-08-26T20:27:50.3578954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3579035Z outputs = self.roberta( 2025-08-26T20:27:50.3579331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3579419Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3579735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3579821Z layer_outputs = layer_module( 2025-08-26T20:27:50.3580077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3580164Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3580472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3580562Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3580847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3580928Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3581255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3581393Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3581683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-26T20:27:50.3581777Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3581800Z 2025-08-26T20:27:50.3581917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3582134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3582204Z return mod(**inputs) 2025-08-26T20:27:50.3582496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3582575Z outputs = self.roberta( 2025-08-26T20:27:50.3582888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3582975Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3583269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3583344Z layer_outputs = layer_module( 2025-08-26T20:27:50.3583593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3583676Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3583974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3584063Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3584350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3584435Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3584761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-26T20:27:50.3584896Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:27:50.3585187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-26T20:27:50.3585316Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:27:50.3585544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:27:50.3585620Z return self.act(input) 2025-08-26T20:27:50.3585632Z 2025-08-26T20:27:50.3585744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3585955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3586035Z return mod(**inputs) 2025-08-26T20:27:50.3586348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-26T20:27:50.3586429Z outputs = self.roberta( 2025-08-26T20:27:50.3586741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-26T20:27:50.3586821Z encoder_outputs = self.encoder( 2025-08-26T20:27:50.3587119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-26T20:27:50.3587194Z layer_outputs = layer_module( 2025-08-26T20:27:50.3587435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:27:50.3587519Z return super().__call__(*args, **kwargs) 2025-08-26T20:27:50.3587808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-26T20:27:50.3587906Z layer_output = apply_chunking_to_forward( 2025-08-26T20:27:50.3588183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:27:50.3588272Z return forward_fn(*input_tensors) 2025-08-26T20:27:50.3588597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-26T20:27:50.3588770Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:27:50.3589060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-26T20:27:50.3589147Z hidden_states = self.dense(hidden_states) 2025-08-26T20:27:50.3589151Z 2025-08-26T20:27:50.3589267Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3589501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3589584Z return mod(**inputs) 2025-08-26T20:27:50.3589886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-08-26T20:27:50.3589999Z prediction_scores = self.lm_head(sequence_output) 2025-08-26T20:27:50.3590305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 756, in forward 2025-08-26T20:27:50.3590388Z x = self.dense(features) 2025-08-26T20:27:50.3590391Z 2025-08-26T20:27:50.3590511Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3590726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3590808Z return mod(**inputs) 2025-08-26T20:27:50.3591111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-08-26T20:27:50.3591225Z prediction_scores = self.lm_head(sequence_output) 2025-08-26T20:27:50.3591534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 761, in forward 2025-08-26T20:27:50.3591612Z x = self.decoder(x) 2025-08-26T20:27:50.3591616Z 2025-08-26T20:27:50.3591736Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:27:50.3591951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:27:50.3592028Z return mod(**inputs) 2025-08-26T20:27:50.3592334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1059, in forward 2025-08-26T20:27:50.3592541Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:27:50.3592547Z 2025-08-26T20:27:59.0353484Z Compilation time (from dynamo_timed): 15.398851805 2025-08-26T20:27:59.0430730Z pass 2025-08-26T20:27:59.0431756Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:27:59.0432760Z TIMING: _recursive_pre_grad_passes:0.00769 _recursive_joint_graph_passes:0.386 _recursive_post_grad_passes:0.07591 async_compile.wait:0.75162 code_gen:7.92999 inductor_compile:9.2397 backend_compile:12.57259 gc:0.00026 entire_frame_compile:15.39885 total_wall_time:15.39885 2025-08-26T20:27:59.0436534Z STATS: call_* op count: 297 | FakeTensorMode.__torch_dispatch__:12430 | FakeTensor.__torch_dispatch__:4399 | ProxyTorchDispatchMode.__torch_dispatch__:4530 2025-08-26T20:27:59.0437174Z Dynamo produced 1 graphs covering 297 ops with 0 graph breaks (0 unique) 2025-08-26T20:28:04.4228570Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:28:04.4229702Z from pkg_resources import resource_filename 2025-08-26T20:28:05.0180244Z 2025-08-26T20:28:14.2620769Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:28:14.2621384Z loading model: 0it [00:09, ?it/s] 2025-08-26T20:28:14.2645374Z cpu eval DebertaV2ForMaskedLM 2025-08-26T20:28:14.4001355Z Compilation time (from dynamo_timed): 0 2025-08-26T20:28:14.4006707Z pass_due_to_skip 2025-08-26T20:28:14.4012108Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:28:14.4012627Z TIMING: total_wall_time:0 2025-08-26T20:28:14.4013020Z STATS: call_* op count: 0 2025-08-26T20:28:14.4013307Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-08-26T20:28:19.1997549Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:28:19.1998901Z from pkg_resources import resource_filename 2025-08-26T20:28:19.7791440Z 2025-08-26T20:28:27.3119449Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:28:27.3119923Z loading model: 0it [00:07, ?it/s] 2025-08-26T20:28:27.3143257Z cpu eval DebertaV2ForQuestionAnswering 2025-08-26T20:28:30.6873799Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:28:32.1382937Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:28:33.4594739Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:28:49.0640680Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0641526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0642225Z return mod(**inputs) 2025-08-26T20:28:49.0643029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0644284Z outputs = self.deberta( 2025-08-26T20:28:49.0648225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0648927Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0649681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0650197Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0650643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0651073Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0651888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0652380Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0653645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0654324Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0654791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0655382Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0655667Z 2025-08-26T20:28:49.0655792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0656216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0656595Z return mod(**inputs) 2025-08-26T20:28:49.0657027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0657506Z outputs = self.deberta( 2025-08-26T20:28:49.0657948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0658547Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0659001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0659467Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0659906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0660322Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0660864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0661352Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0661838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0662297Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0662747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.0663352Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0663619Z 2025-08-26T20:28:49.0663747Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0664142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0664502Z return mod(**inputs) 2025-08-26T20:28:49.0664937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0665389Z outputs = self.deberta( 2025-08-26T20:28:49.0665813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0666271Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0666716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0667187Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0667592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0667985Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0668479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0669010Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0669511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0669991Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0670438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0671007Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0671634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0672222Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0672444Z 2025-08-26T20:28:49.0672571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0672968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0673379Z return mod(**inputs) 2025-08-26T20:28:49.0673817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0674279Z outputs = self.deberta( 2025-08-26T20:28:49.0674742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0675195Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0675650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0676125Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0676535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0677001Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0677458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0677943Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0678421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0678886Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0679707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0680340Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0680649Z 2025-08-26T20:28:49.0680767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0681181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0681559Z return mod(**inputs) 2025-08-26T20:28:49.0681981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0682431Z outputs = self.deberta( 2025-08-26T20:28:49.0682857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0683312Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0683761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0684224Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0684635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0685041Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0685535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0686008Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0686493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0686947Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0687396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0687998Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0688303Z 2025-08-26T20:28:49.0688432Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0688829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0689211Z return mod(**inputs) 2025-08-26T20:28:49.0689639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0690107Z outputs = self.deberta( 2025-08-26T20:28:49.0690530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0691003Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0691448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0691913Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0692309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0692692Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0693151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0693608Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0694065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0694502Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0694935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0695498Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0695777Z 2025-08-26T20:28:49.0695890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0696537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0696907Z return mod(**inputs) 2025-08-26T20:28:49.0697320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0697755Z outputs = self.deberta( 2025-08-26T20:28:49.0698172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0698618Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0699041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0699491Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0699887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0700285Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0700722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0701252Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0701710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0702188Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0702636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0703222Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0703836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0704389Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0704606Z 2025-08-26T20:28:49.0704738Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0705137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0705494Z return mod(**inputs) 2025-08-26T20:28:49.0705914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0706403Z outputs = self.deberta( 2025-08-26T20:28:49.0706818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0707254Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0707680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0708121Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0708565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0708961Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0709399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0709844Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0710298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0710739Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0711171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.0711603Z context_layer = torch.bmm( 2025-08-26T20:28:49.0711729Z 2025-08-26T20:28:49.0711843Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0712240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0712608Z return mod(**inputs) 2025-08-26T20:28:49.0713033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0713480Z outputs = self.deberta( 2025-08-26T20:28:49.0713903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0714337Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0714778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0715232Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0715641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0716033Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0716517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0716984Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0717473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0717924Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0718370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.0718950Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.0719310Z 2025-08-26T20:28:49.0719436Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0719848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0720215Z return mod(**inputs) 2025-08-26T20:28:49.0720647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0721104Z outputs = self.deberta( 2025-08-26T20:28:49.0721537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0722020Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0722465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0722927Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0723338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0723744Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0724223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0724715Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0725224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.0725750Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.0726248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.0726711Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0726866Z 2025-08-26T20:28:49.0726983Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0727383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0727748Z return mod(**inputs) 2025-08-26T20:28:49.0728184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0728635Z outputs = self.deberta( 2025-08-26T20:28:49.0729067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0729539Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0729987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0730454Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0730852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0731267Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0731710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.0732220Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.0732784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.0733244Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0733426Z 2025-08-26T20:28:49.0733542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0733941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0734319Z return mod(**inputs) 2025-08-26T20:28:49.0734742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0735183Z outputs = self.deberta( 2025-08-26T20:28:49.0735612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0736063Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0736519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0737002Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0737409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0737847Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0738309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.0738835Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.0739338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.0739831Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.0740283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.0740672Z return self.act(input) 2025-08-26T20:28:49.0740794Z 2025-08-26T20:28:49.0740917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0741305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0741675Z return mod(**inputs) 2025-08-26T20:28:49.0742090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0742524Z outputs = self.deberta( 2025-08-26T20:28:49.0742941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0743387Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0743823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0744287Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0744688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0745078Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0745520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.0746025Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.0746518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.0746977Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0747125Z 2025-08-26T20:28:49.0747238Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0747630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0748007Z return mod(**inputs) 2025-08-26T20:28:49.0748418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0748878Z outputs = self.deberta( 2025-08-26T20:28:49.0749284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0749715Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0750151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0750595Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0750981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0751378Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0751832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0752306Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0752784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0753249Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0753715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0754313Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0754589Z 2025-08-26T20:28:49.0754719Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0755132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0755516Z return mod(**inputs) 2025-08-26T20:28:49.0755946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0756397Z outputs = self.deberta( 2025-08-26T20:28:49.0756829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0757280Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0757718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0758176Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0758585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0758990Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0759655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0760156Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0760639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0761091Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0761549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.0762126Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0762387Z 2025-08-26T20:28:49.0762503Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0762903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0763268Z return mod(**inputs) 2025-08-26T20:28:49.0763727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0764178Z outputs = self.deberta( 2025-08-26T20:28:49.0764631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0765083Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0765524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0765995Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0766390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0766800Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0767275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0767734Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0768180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0768621Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0769061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0769643Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0770233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0770776Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0770998Z 2025-08-26T20:28:49.0771111Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0771505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0771850Z return mod(**inputs) 2025-08-26T20:28:49.0772265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0772704Z outputs = self.deberta( 2025-08-26T20:28:49.0773113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0773562Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0773996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0774487Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0774889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0775291Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0775751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0776221Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0776685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0777129Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0777567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0778156Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0778441Z 2025-08-26T20:28:49.0778560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0778956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0779320Z return mod(**inputs) 2025-08-26T20:28:49.0779742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0780241Z outputs = self.deberta( 2025-08-26T20:28:49.0780657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0781091Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0781508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0781957Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0782367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0782773Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0783210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0783671Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0784143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0784623Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0785072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0785647Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0785936Z 2025-08-26T20:28:49.0786049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0786460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0786812Z return mod(**inputs) 2025-08-26T20:28:49.0787237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0787682Z outputs = self.deberta( 2025-08-26T20:28:49.0788115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0788568Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0789017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0789485Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0789883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0790285Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0790737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0791204Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0791672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0792125Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0792571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0793161Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0793432Z 2025-08-26T20:28:49.0793555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0793946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0794306Z return mod(**inputs) 2025-08-26T20:28:49.0794748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0795199Z outputs = self.deberta( 2025-08-26T20:28:49.0795655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0796103Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0796777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0797253Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0797661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0798081Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0798540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0799026Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0799601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0800077Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0800540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0801187Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0801806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0802370Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0802610Z 2025-08-26T20:28:49.0802737Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0803140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0803495Z return mod(**inputs) 2025-08-26T20:28:49.0803922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0804374Z outputs = self.deberta( 2025-08-26T20:28:49.0804803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0805244Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0805663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0806087Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0806464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0806833Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0807250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0807713Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0808165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0808609Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0809054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.0809489Z context_layer = torch.bmm( 2025-08-26T20:28:49.0809624Z 2025-08-26T20:28:49.0809739Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0810130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0810479Z return mod(**inputs) 2025-08-26T20:28:49.0810914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0811354Z outputs = self.deberta( 2025-08-26T20:28:49.0811803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0812251Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0812691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0813145Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0813546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0813949Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0814392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0814856Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0815305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0815748Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0816210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.0816772Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.0817033Z 2025-08-26T20:28:49.0817150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0817532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0817904Z return mod(**inputs) 2025-08-26T20:28:49.0818324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0818765Z outputs = self.deberta( 2025-08-26T20:28:49.0819175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0819614Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0820047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0820494Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0820888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0821274Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0821717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0822176Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0822635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.0823118Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.0823595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.0824039Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0824196Z 2025-08-26T20:28:49.0824308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0824696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0825049Z return mod(**inputs) 2025-08-26T20:28:49.0825459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0825919Z outputs = self.deberta( 2025-08-26T20:28:49.0826337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0826786Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0827209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0827665Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0828061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0828456Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0828896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.0829378Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.0829876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.0830324Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0830475Z 2025-08-26T20:28:49.0830597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0831001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0831344Z return mod(**inputs) 2025-08-26T20:28:49.0831759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0832192Z outputs = self.deberta( 2025-08-26T20:28:49.0832607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0833054Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0833494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0833945Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0834344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0834747Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0835197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.0835707Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.0836212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.0836704Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.0837118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.0837477Z return self.act(input) 2025-08-26T20:28:49.0837603Z 2025-08-26T20:28:49.0837715Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0838104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0838453Z return mod(**inputs) 2025-08-26T20:28:49.0838859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0839386Z outputs = self.deberta( 2025-08-26T20:28:49.0839811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0840247Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0840676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0841123Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0841500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0841882Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0842297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.0842764Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.0843219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.0843636Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0843786Z 2025-08-26T20:28:49.0843894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0844262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0844594Z return mod(**inputs) 2025-08-26T20:28:49.0844973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0845383Z outputs = self.deberta( 2025-08-26T20:28:49.0845770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0846208Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0846606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0847049Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0847447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0847857Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0848301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0848754Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0849217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0849657Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0850103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0850646Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0850906Z 2025-08-26T20:28:49.0851018Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0851408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0851765Z return mod(**inputs) 2025-08-26T20:28:49.0852181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0852623Z outputs = self.deberta( 2025-08-26T20:28:49.0853031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0853466Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0853894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0854341Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0854733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0855126Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0855562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0856095Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0856569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0857003Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0857442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.0857990Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0858242Z 2025-08-26T20:28:49.0858361Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0858750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0859096Z return mod(**inputs) 2025-08-26T20:28:49.0859517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0859959Z outputs = self.deberta( 2025-08-26T20:28:49.0860372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0860803Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0861251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0861693Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0862088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0862472Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0862896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0863373Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0863824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0864271Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0864708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0865259Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0865854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0866396Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0866602Z 2025-08-26T20:28:49.0866716Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0867109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0867451Z return mod(**inputs) 2025-08-26T20:28:49.0867868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0868305Z outputs = self.deberta( 2025-08-26T20:28:49.0868727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0869160Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0869580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0870027Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0870420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0870807Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0871273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0871741Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0872194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0872633Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0873077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0873670Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0873953Z 2025-08-26T20:28:49.0874066Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0874459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0874812Z return mod(**inputs) 2025-08-26T20:28:49.0875227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0875683Z outputs = self.deberta( 2025-08-26T20:28:49.0876105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0876573Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0877014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0877485Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0877881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0878300Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0878756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0879295Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0879791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0880236Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0880686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0881278Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0881566Z 2025-08-26T20:28:49.0881693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0882104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0882457Z return mod(**inputs) 2025-08-26T20:28:49.0882886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0883344Z outputs = self.deberta( 2025-08-26T20:28:49.0883775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0884217Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0884659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0885124Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0885534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0885939Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0886412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0886889Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0887374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0887836Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0888289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0888862Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0889148Z 2025-08-26T20:28:49.0889255Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0889624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0889957Z return mod(**inputs) 2025-08-26T20:28:49.0890353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0890758Z outputs = self.deberta( 2025-08-26T20:28:49.0891160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0891614Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0892050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0892499Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0892895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0893286Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0893750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0894223Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0894672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0895126Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0895565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0896134Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0896969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0897516Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0897733Z 2025-08-26T20:28:49.0897848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0898244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0898596Z return mod(**inputs) 2025-08-26T20:28:49.0899006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0899417Z outputs = self.deberta( 2025-08-26T20:28:49.0899810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0900227Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0900652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0901104Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0901498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0901949Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0902390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0902866Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0903299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0903741Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0904178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.0904612Z context_layer = torch.bmm( 2025-08-26T20:28:49.0904738Z 2025-08-26T20:28:49.0905692Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0906081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0906435Z return mod(**inputs) 2025-08-26T20:28:49.0906850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0907289Z outputs = self.deberta( 2025-08-26T20:28:49.0907701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0908161Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0908597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0909046Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0909446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0909877Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0910311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0910758Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0911211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0911653Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0912092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.0912662Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.0912929Z 2025-08-26T20:28:49.0913041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0913431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0913782Z return mod(**inputs) 2025-08-26T20:28:49.0914199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0914633Z outputs = self.deberta( 2025-08-26T20:28:49.0915045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0915484Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0915920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0916368Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0916770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0917165Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0917604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0918098Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0918559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.0919047Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.0919596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.0920044Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0920196Z 2025-08-26T20:28:49.0920316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0920695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0921041Z return mod(**inputs) 2025-08-26T20:28:49.0921461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0921896Z outputs = self.deberta( 2025-08-26T20:28:49.0922304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0922742Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0923184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0923661Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0924055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0924439Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0924876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.0925371Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.0925842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.0926257Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0926401Z 2025-08-26T20:28:49.0926508Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0926877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0927204Z return mod(**inputs) 2025-08-26T20:28:49.0927603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0928053Z outputs = self.deberta( 2025-08-26T20:28:49.0928474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0928917Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0929351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0929796Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0930160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0930531Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0930949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.0931412Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.0931874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.0932322Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.0932741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.0933096Z return self.act(input) 2025-08-26T20:28:49.0933214Z 2025-08-26T20:28:49.0933334Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0933733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0934085Z return mod(**inputs) 2025-08-26T20:28:49.0934508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0934958Z outputs = self.deberta( 2025-08-26T20:28:49.0935374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0935779Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0936186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0936609Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0936979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0937348Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0937752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.0938265Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.0938770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.0939213Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.0939361Z 2025-08-26T20:28:49.0939478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0939876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0940199Z return mod(**inputs) 2025-08-26T20:28:49.0940578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0940994Z outputs = self.deberta( 2025-08-26T20:28:49.0941399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0941836Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0942265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0942723Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0943119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0943503Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0943944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0944401Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0944858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0945298Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0945731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0946289Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0946555Z 2025-08-26T20:28:49.0946667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0947057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0947404Z return mod(**inputs) 2025-08-26T20:28:49.0947839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0948280Z outputs = self.deberta( 2025-08-26T20:28:49.0948716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0949152Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0949572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0950022Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0950419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0950809Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0951249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0951698Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0952151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0952588Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0953044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.0953598Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0953847Z 2025-08-26T20:28:49.0953958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0954347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0954714Z return mod(**inputs) 2025-08-26T20:28:49.0955142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0955569Z outputs = self.deberta( 2025-08-26T20:28:49.0955972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0956410Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0956845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0957307Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0957705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0958105Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0958560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0959028Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0959586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0960048Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0960541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.0961101Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.0961697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0962247Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0962455Z 2025-08-26T20:28:49.0962561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0962953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0963292Z return mod(**inputs) 2025-08-26T20:28:49.0963701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0964112Z outputs = self.deberta( 2025-08-26T20:28:49.0964504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0964925Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0965356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0965805Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0966196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0966592Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0967023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0967457Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0967885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0968352Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0968789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0969344Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0969615Z 2025-08-26T20:28:49.0969730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0970160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0970511Z return mod(**inputs) 2025-08-26T20:28:49.0970902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0971316Z outputs = self.deberta( 2025-08-26T20:28:49.0971706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0972117Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0972515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0972934Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0973305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0973676Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0974083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0974512Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0974936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0975352Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0975784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.0976349Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.0976623Z 2025-08-26T20:28:49.0976729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0977097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0977437Z return mod(**inputs) 2025-08-26T20:28:49.0977870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0978303Z outputs = self.deberta( 2025-08-26T20:28:49.0978740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0979178Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0979608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0980056Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0980450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0980853Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0981305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0981774Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0982235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0982693Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0983149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0983705Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0983966Z 2025-08-26T20:28:49.0984088Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0984467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0984844Z return mod(**inputs) 2025-08-26T20:28:49.0985257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0985690Z outputs = self.deberta( 2025-08-26T20:28:49.0986105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0986537Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0986968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0987414Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0987807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0988200Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0988635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0989091Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0989542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0989974Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0990405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.0990967Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.0991569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.0992110Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.0992307Z 2025-08-26T20:28:49.0992427Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.0992833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.0993176Z return mod(**inputs) 2025-08-26T20:28:49.0993609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.0994046Z outputs = self.deberta( 2025-08-26T20:28:49.0994458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.0994881Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.0995308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.0995750Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.0996147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.0996743Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.0997178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.0997649Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.0998103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.0998598Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.0999036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.0999524Z context_layer = torch.bmm( 2025-08-26T20:28:49.0999662Z 2025-08-26T20:28:49.0999778Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1000206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1000557Z return mod(**inputs) 2025-08-26T20:28:49.1000968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1001405Z outputs = self.deberta( 2025-08-26T20:28:49.1001822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1002271Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1002702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1003147Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1003544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1003938Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1004379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1004836Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1005296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1005744Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1006199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1006798Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1007065Z 2025-08-26T20:28:49.1007181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1007543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1007880Z return mod(**inputs) 2025-08-26T20:28:49.1008309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1008729Z outputs = self.deberta( 2025-08-26T20:28:49.1009162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1009613Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1010047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1010506Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1010905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1011295Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1011746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1012207Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1012663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1013153Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1013648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1014120Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1014269Z 2025-08-26T20:28:49.1014378Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1014750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1015089Z return mod(**inputs) 2025-08-26T20:28:49.1015487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1015896Z outputs = self.deberta( 2025-08-26T20:28:49.1016287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1016692Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1017105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1017552Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1017949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1018336Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1018768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1019232Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1019685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1020101Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1020242Z 2025-08-26T20:28:49.1020356Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1020725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1021045Z return mod(**inputs) 2025-08-26T20:28:49.1021431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1021840Z outputs = self.deberta( 2025-08-26T20:28:49.1022255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1022682Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1023103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1023524Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1023943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1024335Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1024776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1025273Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1025770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1026240Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1026655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1027014Z return self.act(input) 2025-08-26T20:28:49.1027140Z 2025-08-26T20:28:49.1027253Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1027643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1028021Z return mod(**inputs) 2025-08-26T20:28:49.1028426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1028862Z outputs = self.deberta( 2025-08-26T20:28:49.1029274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1029710Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1030161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1030601Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1030994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1031380Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1031815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1032316Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1032802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1033241Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1033395Z 2025-08-26T20:28:49.1033505Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1033889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1034236Z return mod(**inputs) 2025-08-26T20:28:49.1034640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1035071Z outputs = self.deberta( 2025-08-26T20:28:49.1035463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1035897Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1036312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1036757Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1037148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1037536Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1038001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1038450Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1038938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1039670Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1040124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1040727Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1040975Z 2025-08-26T20:28:49.1041082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1041456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1041801Z return mod(**inputs) 2025-08-26T20:28:49.1042227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1042669Z outputs = self.deberta( 2025-08-26T20:28:49.1043089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1043551Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1043989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1044450Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1044839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1045238Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1045700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1046151Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1046612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1047040Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1047476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1048022Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1048279Z 2025-08-26T20:28:49.1048399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1048786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1049127Z return mod(**inputs) 2025-08-26T20:28:49.1049542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1049973Z outputs = self.deberta( 2025-08-26T20:28:49.1050386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1050821Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1051238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1051680Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1052075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1052463Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1052890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1053366Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1053822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1054273Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1054712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1055268Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1055864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1056404Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1056604Z 2025-08-26T20:28:49.1056724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1057115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1057459Z return mod(**inputs) 2025-08-26T20:28:49.1057875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1058314Z outputs = self.deberta( 2025-08-26T20:28:49.1058751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1059195Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1059627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1060083Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1060479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1060883Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1061329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1061775Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1062229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1062665Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1063107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1063691Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1063983Z 2025-08-26T20:28:49.1064095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1064485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1064835Z return mod(**inputs) 2025-08-26T20:28:49.1065243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1065678Z outputs = self.deberta( 2025-08-26T20:28:49.1066105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1066550Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1066992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1067459Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1067857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1068254Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1068722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1069187Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1069666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1070110Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1070560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1071154Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1071443Z 2025-08-26T20:28:49.1071565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1071962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1072310Z return mod(**inputs) 2025-08-26T20:28:49.1072735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1073183Z outputs = self.deberta( 2025-08-26T20:28:49.1073608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1074067Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1074508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1074969Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1075374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1075792Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1076234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1076700Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1077164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1077616Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1078064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1078634Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1078914Z 2025-08-26T20:28:49.1079032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1079512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1079883Z return mod(**inputs) 2025-08-26T20:28:49.1080314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1080765Z outputs = self.deberta( 2025-08-26T20:28:49.1081197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1081650Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1082091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1082549Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1082965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1083377Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1083815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1084314Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1084762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1085230Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1085682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1086258Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1086853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1087388Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1087594Z 2025-08-26T20:28:49.1087708Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1088100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1088451Z return mod(**inputs) 2025-08-26T20:28:49.1088868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1089318Z outputs = self.deberta( 2025-08-26T20:28:49.1089733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1090168Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1090597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1091048Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1091457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1091840Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1092277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1092729Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1093169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1093611Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1094039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1094462Z context_layer = torch.bmm( 2025-08-26T20:28:49.1094587Z 2025-08-26T20:28:49.1094704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1094920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1094991Z return mod(**inputs) 2025-08-26T20:28:49.1095294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1095370Z outputs = self.deberta( 2025-08-26T20:28:49.1095669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1095750Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1096038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1096137Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1096553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1096657Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1097003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1097115Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1097427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1097512Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1097810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1098016Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1098020Z 2025-08-26T20:28:49.1098138Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1098350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1098431Z return mod(**inputs) 2025-08-26T20:28:49.1098729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1098802Z outputs = self.deberta( 2025-08-26T20:28:49.1099103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1099214Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1099511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1099606Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1099845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1099938Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1100255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1100361Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1100659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1100793Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1101096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1101188Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1101192Z 2025-08-26T20:28:49.1101311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1101531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1101611Z return mod(**inputs) 2025-08-26T20:28:49.1101915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1101991Z outputs = self.deberta( 2025-08-26T20:28:49.1102296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1102377Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1102678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1102772Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1103014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1103106Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1103402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1103543Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1103858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1103955Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1104981Z 2025-08-26T20:28:49.1105108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1105323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1105404Z return mod(**inputs) 2025-08-26T20:28:49.1105700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1105780Z outputs = self.deberta( 2025-08-26T20:28:49.1106067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1106149Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1106447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1106540Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1106787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1106899Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1107193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1107322Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1107609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1107737Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1107987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1108070Z return self.act(input) 2025-08-26T20:28:49.1108074Z 2025-08-26T20:28:49.1108183Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1108399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1108479Z return mod(**inputs) 2025-08-26T20:28:49.1108770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1108851Z outputs = self.deberta( 2025-08-26T20:28:49.1109142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1109226Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1109517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1109609Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1109855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1109942Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1110244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1110389Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1110676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1110774Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1110778Z 2025-08-26T20:28:49.1110887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1111107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1111199Z return mod(**inputs) 2025-08-26T20:28:49.1111500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1111589Z outputs = self.deberta( 2025-08-26T20:28:49.1111878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1111968Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1112256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1112353Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1112591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1112678Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1112975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1113076Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1113374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1113480Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1113772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1113979Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1113983Z 2025-08-26T20:28:49.1114094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1114311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1114407Z return mod(**inputs) 2025-08-26T20:28:49.1114707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1114780Z outputs = self.deberta( 2025-08-26T20:28:49.1115069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1115156Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1115445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1115543Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1115782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1115873Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1116167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1116268Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1116567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1116652Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1116961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1117160Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1117164Z 2025-08-26T20:28:49.1117276Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1117503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1117578Z return mod(**inputs) 2025-08-26T20:28:49.1117906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1117984Z outputs = self.deberta( 2025-08-26T20:28:49.1118303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1118387Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1118689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1118789Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1119028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1119123Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1119503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1119615Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1119922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1120007Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1120313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1120545Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1120896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1121044Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1121068Z 2025-08-26T20:28:49.1121184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1121415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1121488Z return mod(**inputs) 2025-08-26T20:28:49.1121800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1121877Z outputs = self.deberta( 2025-08-26T20:28:49.1122179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1122272Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1122582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1122684Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1122933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1123030Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1123329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1123434Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1123738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1123826Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1124130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1124366Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1124371Z 2025-08-26T20:28:49.1124491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1124712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1124804Z return mod(**inputs) 2025-08-26T20:28:49.1125115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1125207Z outputs = self.deberta( 2025-08-26T20:28:49.1125514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1125597Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1125893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1125995Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1126238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1126332Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1126631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1126732Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1127035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1127144Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1127447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1127678Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1127682Z 2025-08-26T20:28:49.1127802Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1128031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1128098Z return mod(**inputs) 2025-08-26T20:28:49.1128387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1128456Z outputs = self.deberta( 2025-08-26T20:28:49.1128736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1128811Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1129088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1129187Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1129407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1129492Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1129760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1129857Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1130122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1130199Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1130484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1130679Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1130682Z 2025-08-26T20:28:49.1130792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1130993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1131067Z return mod(**inputs) 2025-08-26T20:28:49.1131364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1131436Z outputs = self.deberta( 2025-08-26T20:28:49.1131731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1131807Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1132089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1132177Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1132403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1132494Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1132773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1132878Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1133179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1133268Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1133569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1133792Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1134135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1134276Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1134295Z 2025-08-26T20:28:49.1134414Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1134649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1134718Z return mod(**inputs) 2025-08-26T20:28:49.1135031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1135107Z outputs = self.deberta( 2025-08-26T20:28:49.1135409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1135487Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1135790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1135882Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1136121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1136219Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1136523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1136631Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1136939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1137025Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1137341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1137422Z context_layer = torch.bmm( 2025-08-26T20:28:49.1137427Z 2025-08-26T20:28:49.1137548Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1137765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1137843Z return mod(**inputs) 2025-08-26T20:28:49.1138165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1138253Z outputs = self.deberta( 2025-08-26T20:28:49.1138533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1138608Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1138891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1138977Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1139201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1139292Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1139584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1139689Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1139987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1140101Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1140400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1140600Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1140604Z 2025-08-26T20:28:49.1140722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1140937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1141032Z return mod(**inputs) 2025-08-26T20:28:49.1141327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1141399Z outputs = self.deberta( 2025-08-26T20:28:49.1141695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1141773Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1142075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1142161Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1142390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1142471Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1142744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1142844Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1143117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1143243Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1143533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1143622Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1143626Z 2025-08-26T20:28:49.1143744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1143963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1144036Z return mod(**inputs) 2025-08-26T20:28:49.1144314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1144411Z outputs = self.deberta( 2025-08-26T20:28:49.1144682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1144771Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1145051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1145137Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1145366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1145446Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1145725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1145858Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1146131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1146221Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1146226Z 2025-08-26T20:28:49.1146329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1146572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1146638Z return mod(**inputs) 2025-08-26T20:28:49.1146918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1146995Z outputs = self.deberta( 2025-08-26T20:28:49.1147264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1147361Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1147634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1147720Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1147951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1148032Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1148309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1148431Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1148703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1148827Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1149044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1149125Z return self.act(input) 2025-08-26T20:28:49.1149129Z 2025-08-26T20:28:49.1149233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1149443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1149511Z return mod(**inputs) 2025-08-26T20:28:49.1149789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1149866Z outputs = self.deberta( 2025-08-26T20:28:49.1150144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1150227Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1150501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1150602Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1150836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1150932Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1151216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1151353Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1151630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1151714Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1151717Z 2025-08-26T20:28:49.1151820Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1152027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1152094Z return mod(**inputs) 2025-08-26T20:28:49.1152372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1152443Z outputs = self.deberta( 2025-08-26T20:28:49.1152710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1152808Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1153079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1153171Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1153393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1153496Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1153778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1153874Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1154151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1154233Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1154513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1154705Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1154709Z 2025-08-26T20:28:49.1154817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1155019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1155087Z return mod(**inputs) 2025-08-26T20:28:49.1155369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1155438Z outputs = self.deberta( 2025-08-26T20:28:49.1155717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1155791Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1156066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1156165Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1156401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1156492Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1156798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1156900Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1157212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1157298Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1157594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1157787Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1157791Z 2025-08-26T20:28:49.1157906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1158119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1158190Z return mod(**inputs) 2025-08-26T20:28:49.1158493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1158567Z outputs = self.deberta( 2025-08-26T20:28:49.1158860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1158939Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1159317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1159427Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1159668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1159762Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1160060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1160202Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1160505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1160588Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1160884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1161090Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1161438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1161583Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1161589Z 2025-08-26T20:28:49.1161712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1161931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1162002Z return mod(**inputs) 2025-08-26T20:28:49.1162314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1162389Z outputs = self.deberta( 2025-08-26T20:28:49.1162693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1162772Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1163095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1163195Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1163440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1163534Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1163851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1163979Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1164286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1164373Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1164677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1164909Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1164913Z 2025-08-26T20:28:49.1165034Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1165256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1165329Z return mod(**inputs) 2025-08-26T20:28:49.1165667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1165744Z outputs = self.deberta( 2025-08-26T20:28:49.1166047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1166146Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1166457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1166551Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1166804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1166919Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1167213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1167321Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1167621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1167708Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1168012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1168242Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1168246Z 2025-08-26T20:28:49.1168366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1168591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1168667Z return mod(**inputs) 2025-08-26T20:28:49.1168958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1169032Z outputs = self.deberta( 2025-08-26T20:28:49.1169328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1169403Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1169681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1169769Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1169991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1170081Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1170367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1170473Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1170765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1170852Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1171127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1171322Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1171326Z 2025-08-26T20:28:49.1171439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1171638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1171712Z return mod(**inputs) 2025-08-26T20:28:49.1171991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1172060Z outputs = self.deberta( 2025-08-26T20:28:49.1172343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1172442Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1172721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1172810Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1173041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1173120Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1173409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1173510Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1173786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1173871Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1174144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1174334Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1174650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1174783Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1174789Z 2025-08-26T20:28:49.1174902Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1175101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1175173Z return mod(**inputs) 2025-08-26T20:28:49.1175454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1175524Z outputs = self.deberta( 2025-08-26T20:28:49.1175801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1175875Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1176155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1176240Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1176460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1176566Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1176839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1176953Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1177231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1177316Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1177593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1177667Z context_layer = torch.bmm( 2025-08-26T20:28:49.1177671Z 2025-08-26T20:28:49.1177784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1177990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1178062Z return mod(**inputs) 2025-08-26T20:28:49.1178342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1178413Z outputs = self.deberta( 2025-08-26T20:28:49.1178697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1178789Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1179068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1179153Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1179383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1179479Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1179753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1179854Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1180129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1180215Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1180489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1180678Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1180688Z 2025-08-26T20:28:49.1180790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1180989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1181064Z return mod(**inputs) 2025-08-26T20:28:49.1181346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1181423Z outputs = self.deberta( 2025-08-26T20:28:49.1181700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1181774Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1182051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1182137Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1182367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1182448Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1182723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1182843Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1183135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1183264Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1183551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1183642Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1183645Z 2025-08-26T20:28:49.1183747Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1183945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1184018Z return mod(**inputs) 2025-08-26T20:28:49.1184291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1184368Z outputs = self.deberta( 2025-08-26T20:28:49.1184634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1184705Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1184982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1185083Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1185308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1185386Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1185659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1185795Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1186060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1186150Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1186155Z 2025-08-26T20:28:49.1186256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1186457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1186523Z return mod(**inputs) 2025-08-26T20:28:49.1186792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1186868Z outputs = self.deberta( 2025-08-26T20:28:49.1187135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1187215Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1187482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1187572Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1187792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1187872Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1188142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1188259Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1188538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1188654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1188894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1188985Z return self.act(input) 2025-08-26T20:28:49.1188988Z 2025-08-26T20:28:49.1189090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1189304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1189373Z return mod(**inputs) 2025-08-26T20:28:49.1189641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1189715Z outputs = self.deberta( 2025-08-26T20:28:49.1189981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1190063Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1190332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1190425Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1190645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1190725Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1190997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1191147Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1191416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1191496Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1191500Z 2025-08-26T20:28:49.1191608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1191839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1191906Z return mod(**inputs) 2025-08-26T20:28:49.1192184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1192253Z outputs = self.deberta( 2025-08-26T20:28:49.1192526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1192600Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1192863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1192955Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1193173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1193257Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1193532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1193629Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1193910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1193990Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1194270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1194462Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1194465Z 2025-08-26T20:28:49.1194578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1194777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1194845Z return mod(**inputs) 2025-08-26T20:28:49.1195154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1195226Z outputs = self.deberta( 2025-08-26T20:28:49.1195522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1195599Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1195871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1195963Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1196362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1196462Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1196739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1196842Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1197127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1197211Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1197557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1197753Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1197757Z 2025-08-26T20:28:49.1197876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1198090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1198200Z return mod(**inputs) 2025-08-26T20:28:49.1198502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1198577Z outputs = self.deberta( 2025-08-26T20:28:49.1198877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1198955Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1199293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1199394Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1199634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1199728Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1200022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1200136Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1200433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1200519Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1200833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1201035Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1201368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1201509Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1201514Z 2025-08-26T20:28:49.1201634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1201880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1201952Z return mod(**inputs) 2025-08-26T20:28:49.1202283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1202359Z outputs = self.deberta( 2025-08-26T20:28:49.1202656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1202733Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1203040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1203132Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1203373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1203468Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1203755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1203861Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1204148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1204249Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1204549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1204777Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1204781Z 2025-08-26T20:28:49.1204899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1205132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1205209Z return mod(**inputs) 2025-08-26T20:28:49.1205500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1205574Z outputs = self.deberta( 2025-08-26T20:28:49.1205871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1205951Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1206249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1206341Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1206583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1206676Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1206964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1207070Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1207362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1207452Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1207740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1207962Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1207966Z 2025-08-26T20:28:49.1208083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1208296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1208374Z return mod(**inputs) 2025-08-26T20:28:49.1208699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1208774Z outputs = self.deberta( 2025-08-26T20:28:49.1209087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1209167Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1209460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1209553Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1209796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1209882Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1210173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1210278Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1210566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1210674Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1210963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1211167Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1211177Z 2025-08-26T20:28:49.1211288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1211498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1211602Z return mod(**inputs) 2025-08-26T20:28:49.1211894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1211973Z outputs = self.deberta( 2025-08-26T20:28:49.1212260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1212339Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1212634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1212725Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1212965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1213048Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1213345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1213453Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1213751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1213840Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1214137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1214348Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1214686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1214828Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1214834Z 2025-08-26T20:28:49.1214953Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1215194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1215272Z return mod(**inputs) 2025-08-26T20:28:49.1215593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1215669Z outputs = self.deberta( 2025-08-26T20:28:49.1215964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1216042Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1216342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1216432Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1216680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1216767Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1217054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1217162Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1217457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1217568Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1217874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1217951Z context_layer = torch.bmm( 2025-08-26T20:28:49.1217963Z 2025-08-26T20:28:49.1218076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1218308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1218387Z return mod(**inputs) 2025-08-26T20:28:49.1218687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1218769Z outputs = self.deberta( 2025-08-26T20:28:49.1219063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1219142Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1219439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1219530Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1219775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1219866Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1220165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1220271Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1220567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1220656Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1220951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1221159Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1221163Z 2025-08-26T20:28:49.1221274Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1221494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1221573Z return mod(**inputs) 2025-08-26T20:28:49.1221892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1221974Z outputs = self.deberta( 2025-08-26T20:28:49.1222288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1222369Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1222684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1222774Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1223024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1223108Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1223410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1223510Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1223804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1223932Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1224251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1224343Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1224346Z 2025-08-26T20:28:49.1224449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1224647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1224718Z return mod(**inputs) 2025-08-26T20:28:49.1225015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1225095Z outputs = self.deberta( 2025-08-26T20:28:49.1225371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1225453Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1225724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1225814Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1226046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1226125Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1226402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1226527Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1226799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1226897Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1226900Z 2025-08-26T20:28:49.1227007Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1227218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1227285Z return mod(**inputs) 2025-08-26T20:28:49.1227570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1227639Z outputs = self.deberta( 2025-08-26T20:28:49.1227916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1227997Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1228285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1228382Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1228627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1228711Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1228988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1229107Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1229404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1229525Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1229766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1229841Z return self.act(input) 2025-08-26T20:28:49.1229844Z 2025-08-26T20:28:49.1229953Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1230176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1230264Z return mod(**inputs) 2025-08-26T20:28:49.1230565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1230638Z outputs = self.deberta( 2025-08-26T20:28:49.1230941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1231028Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1231335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1231429Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1231654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1231736Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1232012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1232148Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1232440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1232527Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1232532Z 2025-08-26T20:28:49.1232652Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1232866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1232939Z return mod(**inputs) 2025-08-26T20:28:49.1233239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1233315Z outputs = self.deberta( 2025-08-26T20:28:49.1233610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1233690Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1233979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1234077Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1234319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1234412Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1234726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1234837Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1235141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1235227Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1235523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1235724Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1235728Z 2025-08-26T20:28:49.1235843Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1236054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1236132Z return mod(**inputs) 2025-08-26T20:28:49.1236430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1236505Z outputs = self.deberta( 2025-08-26T20:28:49.1236803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1236911Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1237214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1237305Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1237545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1237635Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1237943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1238050Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1238335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1238417Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1238711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1238904Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1238907Z 2025-08-26T20:28:49.1239025Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1239298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1239387Z return mod(**inputs) 2025-08-26T20:28:49.1239682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1239767Z outputs = self.deberta( 2025-08-26T20:28:49.1240080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1240159Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1240467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1240564Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1240813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1240907Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1241192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1241330Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1241620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1241737Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1242011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1242203Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1242533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1242677Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1242683Z 2025-08-26T20:28:49.1242802Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1243015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1243093Z return mod(**inputs) 2025-08-26T20:28:49.1243385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1243458Z outputs = self.deberta( 2025-08-26T20:28:49.1243786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1243857Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1244135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1244220Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1244442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1244546Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1244818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1244920Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1245194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1245277Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1245548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1245764Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1245768Z 2025-08-26T20:28:49.1245882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1246083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1246157Z return mod(**inputs) 2025-08-26T20:28:49.1246433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1246501Z outputs = self.deberta( 2025-08-26T20:28:49.1246779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1246854Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1247130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1247217Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1247448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1247530Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1247820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1247924Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1248227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1248315Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1248587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1248796Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1248806Z 2025-08-26T20:28:49.1248918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1249119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1249192Z return mod(**inputs) 2025-08-26T20:28:49.1249470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1249549Z outputs = self.deberta( 2025-08-26T20:28:49.1249821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1249915Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1250194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1250282Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1250515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1250612Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1250883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1250988Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1251260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1251347Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1251619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1251820Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1251823Z 2025-08-26T20:28:49.1251927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1252126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1252203Z return mod(**inputs) 2025-08-26T20:28:49.1252479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1252555Z outputs = self.deberta( 2025-08-26T20:28:49.1252828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1252903Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1253183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1253269Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1253497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1253576Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1253853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1253969Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1254259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1254347Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1254623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1254820Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1255136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1255274Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1255287Z 2025-08-26T20:28:49.1255392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1255599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1255676Z return mod(**inputs) 2025-08-26T20:28:49.1255971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1256071Z outputs = self.deberta( 2025-08-26T20:28:49.1256371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1256459Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1256746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1256833Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1257079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1257161Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1257435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1257536Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1257809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1257895Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1258166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1258245Z context_layer = torch.bmm( 2025-08-26T20:28:49.1258249Z 2025-08-26T20:28:49.1258352Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1258551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1258628Z return mod(**inputs) 2025-08-26T20:28:49.1258904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1258982Z outputs = self.deberta( 2025-08-26T20:28:49.1259257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1259332Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1259632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1259723Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1259965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1260050Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1260363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1260458Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1260760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1260852Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1261153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1261360Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1261364Z 2025-08-26T20:28:49.1261474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1261694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1261773Z return mod(**inputs) 2025-08-26T20:28:49.1262080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1262161Z outputs = self.deberta( 2025-08-26T20:28:49.1262464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1262573Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1262876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1262967Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1263212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1263296Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1263622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1263721Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1264017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1264147Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1264435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1264533Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1264537Z 2025-08-26T20:28:49.1264646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1264864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1264935Z return mod(**inputs) 2025-08-26T20:28:49.1265230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1265311Z outputs = self.deberta( 2025-08-26T20:28:49.1265598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1265683Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1265972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1266063Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1266309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1266393Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1266689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1266820Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1267134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1267243Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1267248Z 2025-08-26T20:28:49.1267359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1267584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1267653Z return mod(**inputs) 2025-08-26T20:28:49.1267960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1268033Z outputs = self.deberta( 2025-08-26T20:28:49.1268332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1268419Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1268708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1268806Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1269044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1269159Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1269454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1269582Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1269872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1270012Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1270244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1270320Z return self.act(input) 2025-08-26T20:28:49.1270324Z 2025-08-26T20:28:49.1270434Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1270652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1270724Z return mod(**inputs) 2025-08-26T20:28:49.1271020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1271093Z outputs = self.deberta( 2025-08-26T20:28:49.1271397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1271475Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1271763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1271864Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1272100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1272192Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1272478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1272621Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1272917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1273006Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1273010Z 2025-08-26T20:28:49.1273126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1273336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1273433Z return mod(**inputs) 2025-08-26T20:28:49.1273726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1273817Z outputs = self.deberta( 2025-08-26T20:28:49.1274117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1274197Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1274493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1274585Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1274821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1274918Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1275207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1275313Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1275603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1275708Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1276004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1276205Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1276209Z 2025-08-26T20:28:49.1276330Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1276542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1276640Z return mod(**inputs) 2025-08-26T20:28:49.1276937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1277011Z outputs = self.deberta( 2025-08-26T20:28:49.1277308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1277386Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1277683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1277774Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1278011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1278103Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1278395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1278502Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1278790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1278879Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1279168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1279426Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1279432Z 2025-08-26T20:28:49.1279553Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1279768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1279850Z return mod(**inputs) 2025-08-26T20:28:49.1280181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1280268Z outputs = self.deberta( 2025-08-26T20:28:49.1280593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1280674Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1280976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1281069Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1281317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1281402Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1281690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1281801Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1282088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1282182Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1282468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1282697Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1283028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1283170Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1283193Z 2025-08-26T20:28:49.1283315Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1283533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1283611Z return mod(**inputs) 2025-08-26T20:28:49.1283906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1283981Z outputs = self.deberta( 2025-08-26T20:28:49.1284282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1284359Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1284656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1284748Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1284991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1285081Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1285369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1285476Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1285763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1285855Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1286148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1286375Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1286386Z 2025-08-26T20:28:49.1286500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1286743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1286823Z return mod(**inputs) 2025-08-26T20:28:49.1287138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1287221Z outputs = self.deberta( 2025-08-26T20:28:49.1287509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1287588Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1287887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1287980Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1288223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1288311Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1288600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1288707Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1288999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1289110Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1289398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1289627Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1289632Z 2025-08-26T20:28:49.1289742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1289974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1290054Z return mod(**inputs) 2025-08-26T20:28:49.1290353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1290437Z outputs = self.deberta( 2025-08-26T20:28:49.1290726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1290806Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1291105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1291198Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1291441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1291528Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1291829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1291927Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1292219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1292310Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1292595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1292811Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1292815Z 2025-08-26T20:28:49.1292928Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1293150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1293220Z return mod(**inputs) 2025-08-26T20:28:49.1293535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1293618Z outputs = self.deberta( 2025-08-26T20:28:49.1293922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1294009Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1294293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1294384Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1294629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1294713Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1295009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1295108Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1295398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1295488Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1295812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1296034Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1296622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1296778Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1296836Z 2025-08-26T20:28:49.1296949Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1297177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1297256Z return mod(**inputs) 2025-08-26T20:28:49.1297555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1297641Z outputs = self.deberta( 2025-08-26T20:28:49.1297944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1298030Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1298330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1298422Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1298669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1298757Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1299064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1299164Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1299461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1299552Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1299850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1299943Z context_layer = torch.bmm( 2025-08-26T20:28:49.1299946Z 2025-08-26T20:28:49.1300052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1300264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1300362Z return mod(**inputs) 2025-08-26T20:28:49.1300664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1300776Z outputs = self.deberta( 2025-08-26T20:28:49.1301140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1301222Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1301495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1301581Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1301814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1301896Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1302180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1302271Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1302543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1302664Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1302936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1303134Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1303138Z 2025-08-26T20:28:49.1303240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1303447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1303536Z return mod(**inputs) 2025-08-26T20:28:49.1303821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1303898Z outputs = self.deberta( 2025-08-26T20:28:49.1304176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1304257Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1304536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1304624Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1304857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1304938Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1305224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1305320Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1305606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1305725Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1306004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1306097Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1306101Z 2025-08-26T20:28:49.1306205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1306420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1306489Z return mod(**inputs) 2025-08-26T20:28:49.1306795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1306875Z outputs = self.deberta( 2025-08-26T20:28:49.1307167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1307252Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1307526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1307619Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1307846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1307927Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1308207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1308332Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1308610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1308695Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1308698Z 2025-08-26T20:28:49.1308812Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1309031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1309097Z return mod(**inputs) 2025-08-26T20:28:49.1309384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1309453Z outputs = self.deberta( 2025-08-26T20:28:49.1309731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1309832Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1310105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1310200Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1310423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1310510Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1310782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1310900Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1311177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1311293Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1311516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1311585Z return self.act(input) 2025-08-26T20:28:49.1311589Z 2025-08-26T20:28:49.1311699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1311900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1311967Z return mod(**inputs) 2025-08-26T20:28:49.1312257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1312329Z outputs = self.deberta( 2025-08-26T20:28:49.1312623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1312702Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1312993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1313112Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1313350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1313460Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1313748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1313898Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1314185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1314273Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1314277Z 2025-08-26T20:28:49.1314394Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1314606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1314687Z return mod(**inputs) 2025-08-26T20:28:49.1314980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1315054Z outputs = self.deberta( 2025-08-26T20:28:49.1315348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1315446Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1315747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1315838Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1316087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1316192Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1316484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1316597Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1316896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1316993Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1317290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1317496Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1317500Z 2025-08-26T20:28:49.1317622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1317841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1317920Z return mod(**inputs) 2025-08-26T20:28:49.1318226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1318314Z outputs = self.deberta( 2025-08-26T20:28:49.1318616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1318699Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1319000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1319095Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1319411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1319501Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1319819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1319929Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1320256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1320358Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1320670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1320880Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1320884Z 2025-08-26T20:28:49.1321000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1321226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1321310Z return mod(**inputs) 2025-08-26T20:28:49.1321625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1321708Z outputs = self.deberta( 2025-08-26T20:28:49.1322012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1322108Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1322412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1322505Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1322751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1322835Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1323138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1323262Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1323558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1323649Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1323948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1324156Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1324498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1324642Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1324654Z 2025-08-26T20:28:49.1324766Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1324980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1325057Z return mod(**inputs) 2025-08-26T20:28:49.1325362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1325444Z outputs = self.deberta( 2025-08-26T20:28:49.1325745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1325822Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1326135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1326227Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1326471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1326574Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1326872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1326997Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1327299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1327392Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1327684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1327918Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1327922Z 2025-08-26T20:28:49.1328034Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1328253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1328324Z return mod(**inputs) 2025-08-26T20:28:49.1328597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1328672Z outputs = self.deberta( 2025-08-26T20:28:49.1328938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1329025Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1329297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1329382Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1329614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1329706Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1329974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1330064Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1330326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1330407Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1330668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1330878Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1330882Z 2025-08-26T20:28:49.1330984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1331188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1331255Z return mod(**inputs) 2025-08-26T20:28:49.1331533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1331612Z outputs = self.deberta( 2025-08-26T20:28:49.1331881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1331964Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1332234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1332320Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1332552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1332633Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1332927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1333021Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1333312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1333399Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1333665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1333866Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1333870Z 2025-08-26T20:28:49.1333973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1334176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1334243Z return mod(**inputs) 2025-08-26T20:28:49.1334513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1334589Z outputs = self.deberta( 2025-08-26T20:28:49.1334854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1334961Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1335225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1335311Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1335535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1335613Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1335905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1335997Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1336279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1336353Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1336611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1336801Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1337095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1337233Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1337238Z 2025-08-26T20:28:49.1337338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1337535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1337596Z return mod(**inputs) 2025-08-26T20:28:49.1337863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1337940Z outputs = self.deberta( 2025-08-26T20:28:49.1338199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1338280Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1338538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1338621Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1338841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1338932Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1339201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1339304Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1339573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1339650Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1339910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1339987Z context_layer = torch.bmm( 2025-08-26T20:28:49.1339991Z 2025-08-26T20:28:49.1340091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1340288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1340353Z return mod(**inputs) 2025-08-26T20:28:49.1340619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1340696Z outputs = self.deberta( 2025-08-26T20:28:49.1340955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1341052Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1341318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1341401Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1341626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1341720Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1341993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1342082Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1342362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1342441Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1342709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1342915Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1342919Z 2025-08-26T20:28:49.1343019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1343222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1343289Z return mod(**inputs) 2025-08-26T20:28:49.1343563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1343630Z outputs = self.deberta( 2025-08-26T20:28:49.1343894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1343971Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1344235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1344325Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1344543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1344621Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1344891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1344998Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1345284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1345400Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1345667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1345756Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1345759Z 2025-08-26T20:28:49.1345861Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1346063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1346130Z return mod(**inputs) 2025-08-26T20:28:49.1346408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1346475Z outputs = self.deberta( 2025-08-26T20:28:49.1346741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1346819Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1347099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1347192Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1347412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1347490Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1347760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1347895Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1348168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1348251Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1348255Z 2025-08-26T20:28:49.1348362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1348558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1348623Z return mod(**inputs) 2025-08-26T20:28:49.1348898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1348965Z outputs = self.deberta( 2025-08-26T20:28:49.1349234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1349307Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1349570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1349662Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1349880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1349967Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1350232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1350357Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1350624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1350740Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1350994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1351067Z return self.act(input) 2025-08-26T20:28:49.1351070Z 2025-08-26T20:28:49.1351180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1351403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1351472Z return mod(**inputs) 2025-08-26T20:28:49.1351767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1351834Z outputs = self.deberta( 2025-08-26T20:28:49.1352107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1352182Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1352459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1352544Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1352761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1352847Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1353118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1353278Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1353552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1353636Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1353647Z 2025-08-26T20:28:49.1353750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1353970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1354045Z return mod(**inputs) 2025-08-26T20:28:49.1354319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1354397Z outputs = self.deberta( 2025-08-26T20:28:49.1354669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1354744Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1355025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1355112Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1355343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1355428Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1355719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1355829Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1356124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1356214Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1356504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1356721Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1356725Z 2025-08-26T20:28:49.1356828Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1357028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1357101Z return mod(**inputs) 2025-08-26T20:28:49.1357643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1357724Z outputs = self.deberta( 2025-08-26T20:28:49.1358023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1358101Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1358382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1358469Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1358699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1358779Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1359061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1359162Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1359523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1359619Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1359934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1360136Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1360140Z 2025-08-26T20:28:49.1360251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1360468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1360561Z return mod(**inputs) 2025-08-26T20:28:49.1360845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1360922Z outputs = self.deberta( 2025-08-26T20:28:49.1361207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1361288Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1361551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1361635Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1361861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1361940Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1362216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1362309Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1362583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1362672Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1362945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1363142Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1363457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1363600Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1363606Z 2025-08-26T20:28:49.1363711Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1363928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1364005Z return mod(**inputs) 2025-08-26T20:28:49.1364302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1364381Z outputs = self.deberta( 2025-08-26T20:28:49.1364663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1364737Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1365025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1365111Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1365350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1365432Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1365710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1365805Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1366079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1366183Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1366452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1366674Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1366678Z 2025-08-26T20:28:49.1366799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1367008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1367075Z return mod(**inputs) 2025-08-26T20:28:49.1367358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1367434Z outputs = self.deberta( 2025-08-26T20:28:49.1367708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1367790Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1368061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1368147Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1368378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1368460Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1368742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1368834Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1369106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1369191Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1369463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1369682Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1369686Z 2025-08-26T20:28:49.1369790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1370000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1370084Z return mod(**inputs) 2025-08-26T20:28:49.1370364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1370457Z outputs = self.deberta( 2025-08-26T20:28:49.1370730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1370811Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1371086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1371172Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1371404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1371487Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1371765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1371856Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1372134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1372230Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1372498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1372700Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1372704Z 2025-08-26T20:28:49.1372807Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1373016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1373102Z return mod(**inputs) 2025-08-26T20:28:49.1373396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1373469Z outputs = self.deberta( 2025-08-26T20:28:49.1373748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1373830Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1374110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1374203Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1374431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1374511Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1374796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1374888Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1375172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1375249Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1375527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1375728Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1376053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1376194Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1376199Z 2025-08-26T20:28:49.1376303Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1376533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1376602Z return mod(**inputs) 2025-08-26T20:28:49.1376906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1376985Z outputs = self.deberta( 2025-08-26T20:28:49.1377256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1377337Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1377609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1377701Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1377925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1378010Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1378287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1378385Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1378680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1378760Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1379033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1379112Z context_layer = torch.bmm( 2025-08-26T20:28:49.1379116Z 2025-08-26T20:28:49.1379220Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1379446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1379513Z return mod(**inputs) 2025-08-26T20:28:49.1379798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1379868Z outputs = self.deberta( 2025-08-26T20:28:49.1380142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1380225Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1380501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1380595Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1380819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1380901Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1381190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1381282Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1381557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1381633Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1381899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1382095Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1382099Z 2025-08-26T20:28:49.1382201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1382406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1382474Z return mod(**inputs) 2025-08-26T20:28:49.1382780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1382850Z outputs = self.deberta( 2025-08-26T20:28:49.1383133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1383217Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1383484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1383580Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1383802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1383884Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1384162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1384258Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1384539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1384659Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1384959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1385047Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1385050Z 2025-08-26T20:28:49.1385156Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1385370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1385465Z return mod(**inputs) 2025-08-26T20:28:49.1385744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1385813Z outputs = self.deberta( 2025-08-26T20:28:49.1386076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1386153Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1386415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1386507Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1386726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1386811Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1387076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1387196Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1387468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1387552Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1387555Z 2025-08-26T20:28:49.1387663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1387864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1387932Z return mod(**inputs) 2025-08-26T20:28:49.1388214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1388284Z outputs = self.deberta( 2025-08-26T20:28:49.1388567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1388642Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1388926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1389010Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1389242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1389331Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1389596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1389723Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1389999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1390114Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1390339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1390410Z return self.act(input) 2025-08-26T20:28:49.1390414Z 2025-08-26T20:28:49.1390523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1390728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1390821Z return mod(**inputs) 2025-08-26T20:28:49.1391115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1391188Z outputs = self.deberta( 2025-08-26T20:28:49.1391486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1391559Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1391854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1391942Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1392168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1392256Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1392531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1392673Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1392944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1393034Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1393037Z 2025-08-26T20:28:49.1393140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1393343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1393416Z return mod(**inputs) 2025-08-26T20:28:49.1393695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1393774Z outputs = self.deberta( 2025-08-26T20:28:49.1394044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1394120Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1394400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1394486Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1394716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1394799Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1395094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1395191Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1395479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1395569Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1395841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1396042Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1396046Z 2025-08-26T20:28:49.1396150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1396538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1396616Z return mod(**inputs) 2025-08-26T20:28:49.1396899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1396983Z outputs = self.deberta( 2025-08-26T20:28:49.1397274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1397411Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1397701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1397792Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1398039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1398151Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1398450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1398549Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1398840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1398930Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1399272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1399484Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1399489Z 2025-08-26T20:28:49.1399597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1399816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1399889Z return mod(**inputs) 2025-08-26T20:28:49.1400184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1400265Z outputs = self.deberta( 2025-08-26T20:28:49.1400568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1400658Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1400947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1401033Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1401266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1401345Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1401629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1401754Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1402040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1402143Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1402419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1402619Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1402935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1403078Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1403083Z 2025-08-26T20:28:49.1403188Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1403397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1403464Z return mod(**inputs) 2025-08-26T20:28:49.1403744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1403841Z outputs = self.deberta( 2025-08-26T20:28:49.1404121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1404200Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1404478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1404567Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1404826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1404907Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1405185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1405277Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1405551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1405635Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1405907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1406127Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1406131Z 2025-08-26T20:28:49.1406237Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1406447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1406511Z return mod(**inputs) 2025-08-26T20:28:49.1406787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1406863Z outputs = self.deberta( 2025-08-26T20:28:49.1407136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1407214Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1407489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1407574Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1407803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1407883Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1408178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1408270Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1408565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1408647Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1408918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1409137Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1409141Z 2025-08-26T20:28:49.1409244Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1409455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1409522Z return mod(**inputs) 2025-08-26T20:28:49.1409809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1409883Z outputs = self.deberta( 2025-08-26T20:28:49.1410156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1410254Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1410527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1410622Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1410845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1410951Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1411229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1411319Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1411596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1411674Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1411948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1412153Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1412156Z 2025-08-26T20:28:49.1412271Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1412475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1412541Z return mod(**inputs) 2025-08-26T20:28:49.1412826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1412894Z outputs = self.deberta( 2025-08-26T20:28:49.1413168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1413249Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1413521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1413615Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1413843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1413925Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1414209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1414317Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1414612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1414692Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1414970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1415165Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1415477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1415619Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1415624Z 2025-08-26T20:28:49.1415730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1415940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1416015Z return mod(**inputs) 2025-08-26T20:28:49.1416302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1416390Z outputs = self.deberta( 2025-08-26T20:28:49.1416673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1416752Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1417022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1417114Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1417352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1417433Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1417711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1417803Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1418080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1418154Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1418439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1418513Z context_layer = torch.bmm( 2025-08-26T20:28:49.1418517Z 2025-08-26T20:28:49.1418622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1418835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1418913Z return mod(**inputs) 2025-08-26T20:28:49.1419195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1419263Z outputs = self.deberta( 2025-08-26T20:28:49.1419533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1419616Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1419892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1419985Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1420222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1420308Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1420595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1420687Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1420971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1421050Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1421317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1421508Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1421512Z 2025-08-26T20:28:49.1421615Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1421823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1421889Z return mod(**inputs) 2025-08-26T20:28:49.1422174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1422242Z outputs = self.deberta( 2025-08-26T20:28:49.1422523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1422615Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1422889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1422983Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1423206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1423293Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1423587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1423676Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1423947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1424063Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1424344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1424428Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1424432Z 2025-08-26T20:28:49.1424539Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1424752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1424818Z return mod(**inputs) 2025-08-26T20:28:49.1425093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1425162Z outputs = self.deberta( 2025-08-26T20:28:49.1425433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1425503Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1425774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1425868Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1426083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1426167Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1426479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1426610Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1426901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1427003Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1427007Z 2025-08-26T20:28:49.1427119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1427321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1427394Z return mod(**inputs) 2025-08-26T20:28:49.1427673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1427744Z outputs = self.deberta( 2025-08-26T20:28:49.1428050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1428130Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1428437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1428524Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1428748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1428860Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1429136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1429265Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1429550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1429690Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1429909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1429981Z return self.act(input) 2025-08-26T20:28:49.1429984Z 2025-08-26T20:28:49.1430094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1430296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1430369Z return mod(**inputs) 2025-08-26T20:28:49.1430650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1430720Z outputs = self.deberta( 2025-08-26T20:28:49.1431005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1431077Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1431362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1431449Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1431681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1431762Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1432037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1432180Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1432459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1432551Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1432554Z 2025-08-26T20:28:49.1432658Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1432860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1432948Z return mod(**inputs) 2025-08-26T20:28:49.1433225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1433315Z outputs = self.deberta( 2025-08-26T20:28:49.1433592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1433674Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1433946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1434033Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1434265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1434347Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1434626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1434720Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1434992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1435096Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1435373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1435576Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1435580Z 2025-08-26T20:28:49.1435689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1435930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1436000Z return mod(**inputs) 2025-08-26T20:28:49.1436304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1436385Z outputs = self.deberta( 2025-08-26T20:28:49.1436674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1436765Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1437056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1437149Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1437394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1437480Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1437780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1437879Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1438175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1438258Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1438549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1438749Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1438753Z 2025-08-26T20:28:49.1438863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1439081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1439152Z return mod(**inputs) 2025-08-26T20:28:49.1439538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1439625Z outputs = self.deberta( 2025-08-26T20:28:49.1439933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1440025Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1440327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1440431Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1440688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1440768Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1441059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1441157Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1441445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1441527Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1441806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1442027Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1442345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1442491Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1442511Z 2025-08-26T20:28:49.1442618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1442827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1442896Z return mod(**inputs) 2025-08-26T20:28:49.1443175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1443253Z outputs = self.deberta( 2025-08-26T20:28:49.1443524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1443603Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1443876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1443965Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1444199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1444283Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1444563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1444659Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1444937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1445018Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1445288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1445513Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1445517Z 2025-08-26T20:28:49.1445623Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1445861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1445934Z return mod(**inputs) 2025-08-26T20:28:49.1446258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1446334Z outputs = self.deberta( 2025-08-26T20:28:49.1446626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1446711Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1446999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1447099Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1447337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1447425Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1447724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1447822Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1448123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1448224Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1448515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1448746Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1448750Z 2025-08-26T20:28:49.1448860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1449117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1449189Z return mod(**inputs) 2025-08-26T20:28:49.1449494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1449569Z outputs = self.deberta( 2025-08-26T20:28:49.1449858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1449947Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1450234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1450333Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1450570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1450662Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1450950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1451050Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1451345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1451429Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1451724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1451929Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1451933Z 2025-08-26T20:28:49.1452041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1452261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1452334Z return mod(**inputs) 2025-08-26T20:28:49.1452654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1452728Z outputs = self.deberta( 2025-08-26T20:28:49.1453048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1453130Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1453422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1453519Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1453760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1453853Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1454145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1454245Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1454542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1454624Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1454939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1455142Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1455478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1455620Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1455644Z 2025-08-26T20:28:49.1455754Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1455979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1456049Z return mod(**inputs) 2025-08-26T20:28:49.1456350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1456425Z outputs = self.deberta( 2025-08-26T20:28:49.1456711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1456796Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1457081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1457179Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1457419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1457511Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1457797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1457898Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1458200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1458281Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1458590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1458670Z context_layer = torch.bmm( 2025-08-26T20:28:49.1458674Z 2025-08-26T20:28:49.1458785Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1459005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1459090Z return mod(**inputs) 2025-08-26T20:28:49.1459397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1459488Z outputs = self.deberta( 2025-08-26T20:28:49.1459787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1459866Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1460158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1460252Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1460476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1460564Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1460839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1460932Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1461212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1461312Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1461597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1461789Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1461793Z 2025-08-26T20:28:49.1461909Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1462121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1462208Z return mod(**inputs) 2025-08-26T20:28:49.1462511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1462584Z outputs = self.deberta( 2025-08-26T20:28:49.1462882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1462962Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1463250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1463351Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1463588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1463682Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1463971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1464078Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1464381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1464506Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1464803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1464893Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1464897Z 2025-08-26T20:28:49.1465014Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1465239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1465311Z return mod(**inputs) 2025-08-26T20:28:49.1465638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1465714Z outputs = self.deberta( 2025-08-26T20:28:49.1466029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1466108Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1466417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1466509Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1466746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1466840Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1467138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1467276Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1467577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1467668Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1467672Z 2025-08-26T20:28:49.1467788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1468020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1468098Z return mod(**inputs) 2025-08-26T20:28:49.1468399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1468479Z outputs = self.deberta( 2025-08-26T20:28:49.1468777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1468872Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1469177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1469271Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1469519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1469604Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1469902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1470041Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1470340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1470472Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1470700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1470775Z return self.act(input) 2025-08-26T20:28:49.1470787Z 2025-08-26T20:28:49.1470897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1471110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1471189Z return mod(**inputs) 2025-08-26T20:28:49.1471492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1471573Z outputs = self.deberta( 2025-08-26T20:28:49.1471871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1471949Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1472255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1472364Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1472609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1472708Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1473011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1473163Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1473463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1473558Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1473562Z 2025-08-26T20:28:49.1473670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1473890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1473961Z return mod(**inputs) 2025-08-26T20:28:49.1474259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1474343Z outputs = self.deberta( 2025-08-26T20:28:49.1474640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1474744Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1475039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1475131Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1475382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1475486Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1475797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1475902Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1476207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1476297Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1476597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1476813Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1476817Z 2025-08-26T20:28:49.1476929Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1477156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1477227Z return mod(**inputs) 2025-08-26T20:28:49.1477543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1477625Z outputs = self.deberta( 2025-08-26T20:28:49.1477928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1478019Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1478318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1478420Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1478665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1478754Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1479087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1479249Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1479601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1479689Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1479993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1480201Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1480205Z 2025-08-26T20:28:49.1480331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1480554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1480626Z return mod(**inputs) 2025-08-26T20:28:49.1480929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1481004Z outputs = self.deberta( 2025-08-26T20:28:49.1481292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1481404Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1481696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1481801Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1482049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1482136Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1482453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1482552Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1482846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1482930Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1483224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1483425Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1483754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1483907Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1483913Z 2025-08-26T20:28:49.1484022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1484245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1484315Z return mod(**inputs) 2025-08-26T20:28:49.1484621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1484697Z outputs = self.deberta( 2025-08-26T20:28:49.1484984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1485071Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1485363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1485464Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1485701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1485802Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1486100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1486216Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1486505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1486584Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1486853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1487062Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1487065Z 2025-08-26T20:28:49.1487171Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1487382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1487449Z return mod(**inputs) 2025-08-26T20:28:49.1487745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1487814Z outputs = self.deberta( 2025-08-26T20:28:49.1488095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1488174Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1488437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1488527Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1488744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1488842Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1489108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1489200Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1489476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1489553Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1489824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1490029Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1490033Z 2025-08-26T20:28:49.1490133Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1490338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1490404Z return mod(**inputs) 2025-08-26T20:28:49.1490681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1490751Z outputs = self.deberta( 2025-08-26T20:28:49.1491023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1491096Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1491360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1491453Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1491671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1491756Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1492038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1492131Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1492418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1492496Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1492768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1492961Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1492965Z 2025-08-26T20:28:49.1493074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1493272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1493340Z return mod(**inputs) 2025-08-26T20:28:49.1493621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1493690Z outputs = self.deberta( 2025-08-26T20:28:49.1493964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1494051Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1494320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1494412Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1494634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1494719Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1495000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1495101Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1495368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1495444Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1495724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1495916Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1496408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1496548Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1496554Z 2025-08-26T20:28:49.1496669Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1496876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1496946Z return mod(**inputs) 2025-08-26T20:28:49.1497238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1497322Z outputs = self.deberta( 2025-08-26T20:28:49.1497597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1497671Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1497939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1498034Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1498256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1498386Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1498677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1498770Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1499042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1499119Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1499398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1499470Z context_layer = torch.bmm( 2025-08-26T20:28:49.1499473Z 2025-08-26T20:28:49.1499583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1499788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1499854Z return mod(**inputs) 2025-08-26T20:28:49.1500140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1500210Z outputs = self.deberta( 2025-08-26T20:28:49.1500492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1500589Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1500854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1500947Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1501163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1501282Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1501548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1501646Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1501914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1501992Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1502268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1502453Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1502457Z 2025-08-26T20:28:49.1502567Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1502767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1502834Z return mod(**inputs) 2025-08-26T20:28:49.1503122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1503193Z outputs = self.deberta( 2025-08-26T20:28:49.1503477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1503551Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1503829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1503921Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1504145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1504233Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1504523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1504626Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1504913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1505032Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1505313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1505399Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1505402Z 2025-08-26T20:28:49.1505513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1505714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1505788Z return mod(**inputs) 2025-08-26T20:28:49.1506064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1506132Z outputs = self.deberta( 2025-08-26T20:28:49.1506412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1506486Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1506782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1506868Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1507091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1507179Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1507454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1507600Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1507873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1507964Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1507968Z 2025-08-26T20:28:49.1508071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1508274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1508347Z return mod(**inputs) 2025-08-26T20:28:49.1508621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1508698Z outputs = self.deberta( 2025-08-26T20:28:49.1508965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1509039Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1509317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1509405Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1509634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1509716Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1509993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1510113Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1510383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1510507Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1510738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1510818Z return self.act(input) 2025-08-26T20:28:49.1510822Z 2025-08-26T20:28:49.1510927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1511143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1511226Z return mod(**inputs) 2025-08-26T20:28:49.1511506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1511582Z outputs = self.deberta( 2025-08-26T20:28:49.1511854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1511930Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1512214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1512303Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1512537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1512619Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1512902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1513056Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1513327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1513418Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1513422Z 2025-08-26T20:28:49.1513543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1513752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1513821Z return mod(**inputs) 2025-08-26T20:28:49.1514099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1514175Z outputs = self.deberta( 2025-08-26T20:28:49.1514449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1514532Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1514818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1514916Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1515154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1515240Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1515538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1515637Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1515932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1516027Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1516300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1516502Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1516506Z 2025-08-26T20:28:49.1516611Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1516822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1516887Z return mod(**inputs) 2025-08-26T20:28:49.1517185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1517258Z outputs = self.deberta( 2025-08-26T20:28:49.1517546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1517630Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1517916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1518015Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1518254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1518339Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1518636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1518737Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1519040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1519124Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1519511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1519716Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1519720Z 2025-08-26T20:28:49.1519832Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1520061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1520157Z return mod(**inputs) 2025-08-26T20:28:49.1520474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1520550Z outputs = self.deberta( 2025-08-26T20:28:49.1520850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1520948Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1521224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1521322Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1521545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1521635Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1521911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1522006Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1522287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1522367Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1522651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1522841Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1523158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1523298Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1523303Z 2025-08-26T20:28:49.1523416Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1523665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1523739Z return mod(**inputs) 2025-08-26T20:28:49.1524069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1524149Z outputs = self.deberta( 2025-08-26T20:28:49.1524449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1524534Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1524834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1524936Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1525181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1525275Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1525574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1525678Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1525986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1526102Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1526406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1526641Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1526645Z 2025-08-26T20:28:49.1526780Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1527008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1527081Z return mod(**inputs) 2025-08-26T20:28:49.1527392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1527468Z outputs = self.deberta( 2025-08-26T20:28:49.1527775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1527854Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1528152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1528256Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1528500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1528597Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1528896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1528998Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1529304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1529390Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1529696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1529927Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1529931Z 2025-08-26T20:28:49.1530051Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1530274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1530360Z return mod(**inputs) 2025-08-26T20:28:49.1530672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1530764Z outputs = self.deberta( 2025-08-26T20:28:49.1531072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1531153Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1531451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1531555Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1531784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1531869Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1532127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1532221Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1532484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1532578Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1532848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1533038Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1533042Z 2025-08-26T20:28:49.1533148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1533345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1533427Z return mod(**inputs) 2025-08-26T20:28:49.1533711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1533778Z outputs = self.deberta( 2025-08-26T20:28:49.1534056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1534127Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1534405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1534490Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1534711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1534798Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1535071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1535169Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1535439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1535514Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1535793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1535989Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1536295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1536423Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1536428Z 2025-08-26T20:28:49.1536535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1536740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1536805Z return mod(**inputs) 2025-08-26T20:28:49.1537093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1537160Z outputs = self.deberta( 2025-08-26T20:28:49.1537423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1537492Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1537749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1537839Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1538053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1538136Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1538393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1538488Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1538762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1538835Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1539103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1539175Z context_layer = torch.bmm( 2025-08-26T20:28:49.1539179Z 2025-08-26T20:28:49.1539286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1539497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1539562Z return mod(**inputs) 2025-08-26T20:28:49.1539837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1539905Z outputs = self.deberta( 2025-08-26T20:28:49.1540175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1540248Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1540518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1540603Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1540821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1540907Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1541175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1541272Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1541538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1541613Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1541883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1542077Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1542081Z 2025-08-26T20:28:49.1542188Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1542378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1542450Z return mod(**inputs) 2025-08-26T20:28:49.1542732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1542798Z outputs = self.deberta( 2025-08-26T20:28:49.1543080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1543155Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1543425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1543510Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1543728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1543814Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1544082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1544180Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1544445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1544569Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1544855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1544936Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1544940Z 2025-08-26T20:28:49.1545049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1545247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1545338Z return mod(**inputs) 2025-08-26T20:28:49.1545607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1545674Z outputs = self.deberta( 2025-08-26T20:28:49.1545949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1546019Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1546293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1546385Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1546611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1546689Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1546955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1547088Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1547366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1547458Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1547461Z 2025-08-26T20:28:49.1547566Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1547767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1547839Z return mod(**inputs) 2025-08-26T20:28:49.1548119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1548196Z outputs = self.deberta( 2025-08-26T20:28:49.1548471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1548556Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1548847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1548933Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1549191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1549274Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1549558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1549682Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1549962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1550086Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1550302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1550382Z return self.act(input) 2025-08-26T20:28:49.1550385Z 2025-08-26T20:28:49.1550489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1550702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1550802Z return mod(**inputs) 2025-08-26T20:28:49.1551145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1551224Z outputs = self.deberta( 2025-08-26T20:28:49.1551499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1551579Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1551870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1551958Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1552188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1552269Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1552554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1552691Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1552972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1553054Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1553058Z 2025-08-26T20:28:49.1553162Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1553371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1553436Z return mod(**inputs) 2025-08-26T20:28:49.1553721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1553790Z outputs = self.deberta( 2025-08-26T20:28:49.1554063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1554144Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1554418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1554510Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1554733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1554813Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1555111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1555207Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1555505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1555587Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1555864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1556058Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1556061Z 2025-08-26T20:28:49.1556165Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1556375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1556441Z return mod(**inputs) 2025-08-26T20:28:49.1556761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1556833Z outputs = self.deberta( 2025-08-26T20:28:49.1557137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1557234Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1557522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1557618Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1557844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1557950Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1558227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1558321Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1558602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1558682Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1558962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1559145Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1559148Z 2025-08-26T20:28:49.1559363Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1580036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1580306Z return mod(**inputs) 2025-08-26T20:28:49.1580672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1580751Z outputs = self.deberta( 2025-08-26T20:28:49.1581052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1581136Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1581410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1581515Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1581741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1581834Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1582106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1582323Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1582594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1582719Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1583005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1583210Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1583538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1583680Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1583689Z 2025-08-26T20:28:49.1583806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1584027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1584100Z return mod(**inputs) 2025-08-26T20:28:49.1584390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1584496Z outputs = self.deberta( 2025-08-26T20:28:49.1584779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1584858Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1585132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1585235Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1585503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1585596Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1585876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1585977Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1586266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1586348Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1586638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1586869Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1586874Z 2025-08-26T20:28:49.1586996Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1587211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1587280Z return mod(**inputs) 2025-08-26T20:28:49.1587574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1587646Z outputs = self.deberta( 2025-08-26T20:28:49.1587934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1588009Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1588291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1588389Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1588617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1588709Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1589001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1589102Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1589391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1589470Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1589746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1589958Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1589962Z 2025-08-26T20:28:49.1590073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1590283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1590349Z return mod(**inputs) 2025-08-26T20:28:49.1590628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1590697Z outputs = self.deberta( 2025-08-26T20:28:49.1590977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1591068Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1591349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1591438Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1591661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1591764Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1592062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1592167Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1592463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1592553Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1592852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1593060Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1593065Z 2025-08-26T20:28:49.1593183Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1593398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1593476Z return mod(**inputs) 2025-08-26T20:28:49.1593782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1593855Z outputs = self.deberta( 2025-08-26T20:28:49.1594161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1594239Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1594540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1594632Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1594873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1594958Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1595283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1595395Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1595712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1595804Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1596154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1596520Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1596871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1597017Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1597025Z 2025-08-26T20:28:49.1597143Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1597355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1597430Z return mod(**inputs) 2025-08-26T20:28:49.1597732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1597865Z outputs = self.deberta( 2025-08-26T20:28:49.1598171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1598250Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1598555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1598648Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1598924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1599019Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1599428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1599545Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1599856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1599945Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1600259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1600337Z context_layer = torch.bmm( 2025-08-26T20:28:49.1600342Z 2025-08-26T20:28:49.1600468Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1600695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1600787Z return mod(**inputs) 2025-08-26T20:28:49.1601084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1601160Z outputs = self.deberta( 2025-08-26T20:28:49.1601460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1601540Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1601838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1601932Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1602181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1602269Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1602598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1602710Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1603016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1603109Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1603396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1603605Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1603617Z 2025-08-26T20:28:49.1603728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1603940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1604020Z return mod(**inputs) 2025-08-26T20:28:49.1604314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1604394Z outputs = self.deberta( 2025-08-26T20:28:49.1604684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1604787Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1605084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1605176Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1605422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1605510Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1605819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1605923Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1606220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1606353Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1606639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1606736Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1606740Z 2025-08-26T20:28:49.1606852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1607065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1607145Z return mod(**inputs) 2025-08-26T20:28:49.1607447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1607519Z outputs = self.deberta( 2025-08-26T20:28:49.1607787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1607858Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1608134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1608220Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1608442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1608522Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1608799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1608940Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1609210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1609318Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1609322Z 2025-08-26T20:28:49.1609427Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1609635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1609701Z return mod(**inputs) 2025-08-26T20:28:49.1609974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1610049Z outputs = self.deberta( 2025-08-26T20:28:49.1610328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1610409Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1610687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1610782Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1611010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1611106Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1611387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1611507Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1611788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1611921Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1612137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1612215Z return self.act(input) 2025-08-26T20:28:49.1612219Z 2025-08-26T20:28:49.1612323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1612529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1612597Z return mod(**inputs) 2025-08-26T20:28:49.1612876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1612944Z outputs = self.deberta( 2025-08-26T20:28:49.1613213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1613292Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1613566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1613667Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1613902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1613987Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1614291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1614427Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1614707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1614790Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1614793Z 2025-08-26T20:28:49.1614906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1615124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1615193Z return mod(**inputs) 2025-08-26T20:28:49.1615498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1615569Z outputs = self.deberta( 2025-08-26T20:28:49.1615850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1615925Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1616195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1616289Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1616511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1616599Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1616875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1616967Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1617240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1617332Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1617602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1617788Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1617792Z 2025-08-26T20:28:49.1617897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1618123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1618188Z return mod(**inputs) 2025-08-26T20:28:49.1618473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1618544Z outputs = self.deberta( 2025-08-26T20:28:49.1618822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1618896Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1619168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1619262Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1619493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1619581Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1619844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1619944Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1620209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1620287Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1620558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1620735Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1620739Z 2025-08-26T20:28:49.1620848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1621043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1621114Z return mod(**inputs) 2025-08-26T20:28:49.1621418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1621488Z outputs = self.deberta( 2025-08-26T20:28:49.1621808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1621895Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1622169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1622253Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1622474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1622563Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1622835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1622938Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1623214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1623290Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1623579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1623764Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1624078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1624213Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1624506Z 2025-08-26T20:28:49.1624623Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1624829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1624895Z return mod(**inputs) 2025-08-26T20:28:49.1625183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1625256Z outputs = self.deberta( 2025-08-26T20:28:49.1625538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1625612Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1625894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1625981Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1626208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1626300Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1626572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1626673Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1626946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1627026Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1627311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1627529Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1627533Z 2025-08-26T20:28:49.1627650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1627869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1627946Z return mod(**inputs) 2025-08-26T20:28:49.1628243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1628312Z outputs = self.deberta( 2025-08-26T20:28:49.1628590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1628663Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1628944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1629030Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1629250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1629335Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1629606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1629701Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1629973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1630076Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1630349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1630561Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1630565Z 2025-08-26T20:28:49.1630677Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1630895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1630970Z return mod(**inputs) 2025-08-26T20:28:49.1631246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1631316Z outputs = self.deberta( 2025-08-26T20:28:49.1631593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1631668Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1631944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1632032Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1632262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1632347Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1632619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1632719Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1632997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1633082Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1633371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1633578Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1633590Z 2025-08-26T20:28:49.1633701Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1633909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1633988Z return mod(**inputs) 2025-08-26T20:28:49.1634297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1634379Z outputs = self.deberta( 2025-08-26T20:28:49.1634686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1634773Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1635062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1635159Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1635397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1635487Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1635777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1635874Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1636165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1636245Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1636558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1636763Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1637091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1637244Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1637265Z 2025-08-26T20:28:49.1637378Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1637595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1637664Z return mod(**inputs) 2025-08-26T20:28:49.1637968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1638045Z outputs = self.deberta( 2025-08-26T20:28:49.1638334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1638420Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1638706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1638802Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1639042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1639126Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1639674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1639778Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1640076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1640158Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1640462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1640536Z context_layer = torch.bmm( 2025-08-26T20:28:49.1640541Z 2025-08-26T20:28:49.1640647Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1640855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1640942Z return mod(**inputs) 2025-08-26T20:28:49.1641227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1641311Z outputs = self.deberta( 2025-08-26T20:28:49.1641581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1641662Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1641932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1642023Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1642244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1642333Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1642606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1642699Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1642976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1643081Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1643376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1643579Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1643583Z 2025-08-26T20:28:49.1643693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1643909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1643997Z return mod(**inputs) 2025-08-26T20:28:49.1644302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1644374Z outputs = self.deberta( 2025-08-26T20:28:49.1644673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1644749Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1645047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1645135Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1645358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1645445Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1645724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1645816Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1646107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1646227Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1646511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1646598Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1646602Z 2025-08-26T20:28:49.1646714Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1646922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1646989Z return mod(**inputs) 2025-08-26T20:28:49.1647281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1647350Z outputs = self.deberta( 2025-08-26T20:28:49.1647636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1647709Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1647973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1648067Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1648285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1648369Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1648633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1648759Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1649019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1649102Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1649106Z 2025-08-26T20:28:49.1649210Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1649418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1649489Z return mod(**inputs) 2025-08-26T20:28:49.1649761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1649827Z outputs = self.deberta( 2025-08-26T20:28:49.1650101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1650190Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1650462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1650545Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1650764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1650848Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1651112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1651237Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1651502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1651622Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1651831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1651902Z return self.act(input) 2025-08-26T20:28:49.1651905Z 2025-08-26T20:28:49.1652009Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1652201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1652274Z return mod(**inputs) 2025-08-26T20:28:49.1652544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1652610Z outputs = self.deberta( 2025-08-26T20:28:49.1652888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1652961Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1653239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1653341Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1653583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1653684Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1653976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1654127Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1654413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1654509Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1654514Z 2025-08-26T20:28:49.1654626Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1654842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1654922Z return mod(**inputs) 2025-08-26T20:28:49.1655214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1655301Z outputs = self.deberta( 2025-08-26T20:28:49.1655564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1655666Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1655932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1656012Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1656235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1656327Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1656592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1656682Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1656949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1657034Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1657300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1657489Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1657493Z 2025-08-26T20:28:49.1657593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1657797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1657861Z return mod(**inputs) 2025-08-26T20:28:49.1658139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1658217Z outputs = self.deberta( 2025-08-26T20:28:49.1658495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1658575Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1658847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1658932Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1659169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1659249Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1659538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1659632Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1659922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1660002Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1660265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1660449Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1660453Z 2025-08-26T20:28:49.1660553Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1660751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1660816Z return mod(**inputs) 2025-08-26T20:28:49.1661087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1661162Z outputs = self.deberta( 2025-08-26T20:28:49.1661430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1661522Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1661787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1661879Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1662100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1662177Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1662466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1662558Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1662827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1662903Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1663168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1663374Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1663698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1663844Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1663849Z 2025-08-26T20:28:49.1663957Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1664174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1664244Z return mod(**inputs) 2025-08-26T20:28:49.1664539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1664621Z outputs = self.deberta( 2025-08-26T20:28:49.1664906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1664988Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1665285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1665382Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1665617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1665715Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1666010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1666130Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1666424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1666507Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1666793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1667024Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1667030Z 2025-08-26T20:28:49.1667139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1667359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1667429Z return mod(**inputs) 2025-08-26T20:28:49.1667740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1667813Z outputs = self.deberta( 2025-08-26T20:28:49.1668120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1668206Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1668493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1668589Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1668823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1668928Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1669224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1669323Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1669618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1669701Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1669994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1670213Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1670217Z 2025-08-26T20:28:49.1670328Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1670552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1670624Z return mod(**inputs) 2025-08-26T20:28:49.1670927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1671001Z outputs = self.deberta( 2025-08-26T20:28:49.1671292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1671379Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1671668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1671766Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1672004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1672098Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1672405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1672505Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1672836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1672923Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1673288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1673496Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1673500Z 2025-08-26T20:28:49.1673610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1673828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1673901Z return mod(**inputs) 2025-08-26T20:28:49.1674205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1674273Z outputs = self.deberta( 2025-08-26T20:28:49.1674576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1674669Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1674975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1675073Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1675319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1675409Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1675725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1675823Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1676127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1676208Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1676506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1676708Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1677054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1677194Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1677200Z 2025-08-26T20:28:49.1677311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1677531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1677603Z return mod(**inputs) 2025-08-26T20:28:49.1677911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1677985Z outputs = self.deberta( 2025-08-26T20:28:49.1678280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1678364Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1678660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1678759Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1678998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1679107Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1679504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1679604Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1679897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1679977Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1680291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1680372Z context_layer = torch.bmm( 2025-08-26T20:28:49.1680376Z 2025-08-26T20:28:49.1680491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1680727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1680800Z return mod(**inputs) 2025-08-26T20:28:49.1681101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1681177Z outputs = self.deberta( 2025-08-26T20:28:49.1681471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1681569Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1681868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1681969Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1682208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1682321Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1682621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1682718Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1683023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1683106Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1683394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1683596Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1683600Z 2025-08-26T20:28:49.1683713Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1683925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1683994Z return mod(**inputs) 2025-08-26T20:28:49.1684304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1684376Z outputs = self.deberta( 2025-08-26T20:28:49.1684668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1684741Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1685011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1685105Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1685339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1685431Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1685748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1685862Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1686140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1686254Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1686525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1686607Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1686611Z 2025-08-26T20:28:49.1686719Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1686915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1686983Z return mod(**inputs) 2025-08-26T20:28:49.1687267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1687338Z outputs = self.deberta( 2025-08-26T20:28:49.1687618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1687692Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1687989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1688072Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1688290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1688373Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1688640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1688788Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1689061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1689145Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1689149Z 2025-08-26T20:28:49.1689273Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1689473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1689545Z return mod(**inputs) 2025-08-26T20:28:49.1689818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1689885Z outputs = self.deberta( 2025-08-26T20:28:49.1690168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1690241Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1690522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1690606Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1690838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1690916Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1691188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1691315Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1691589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1691711Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1691949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1692020Z return self.act(input) 2025-08-26T20:28:49.1692032Z 2025-08-26T20:28:49.1692149Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1692352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1692425Z return mod(**inputs) 2025-08-26T20:28:49.1692700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1692777Z outputs = self.deberta( 2025-08-26T20:28:49.1693050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1693124Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1693412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1693501Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1693733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1693814Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1694102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1694244Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1694514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1694607Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1694611Z 2025-08-26T20:28:49.1694735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1694944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1695009Z return mod(**inputs) 2025-08-26T20:28:49.1695284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1695361Z outputs = self.deberta( 2025-08-26T20:28:49.1695631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1695712Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1695983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1696070Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1696485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1696584Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1696858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1696952Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1697225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1697304Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1697568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1697761Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1697765Z 2025-08-26T20:28:49.1697866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1698069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1698135Z return mod(**inputs) 2025-08-26T20:28:49.1698453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1698555Z outputs = self.deberta( 2025-08-26T20:28:49.1698820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1698900Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1699163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1699253Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1699470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1699552Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1699829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1699923Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1700199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1700300Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1700564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1700750Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1700753Z 2025-08-26T20:28:49.1700855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1701057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1701149Z return mod(**inputs) 2025-08-26T20:28:49.1701434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1701501Z outputs = self.deberta( 2025-08-26T20:28:49.1701772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1701853Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1702126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1702220Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1702447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1702528Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1702815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1702909Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1703198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1703278Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1703563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1703752Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1704072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1704214Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1704219Z 2025-08-26T20:28:49.1704325Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1704553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1704629Z return mod(**inputs) 2025-08-26T20:28:49.1704920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1704998Z outputs = self.deberta( 2025-08-26T20:28:49.1705265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1705347Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1705621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1705715Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1705941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1706024Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1706304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1706399Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1706680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1706777Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1707053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1707273Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1707294Z 2025-08-26T20:28:49.1707400Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1707611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1707678Z return mod(**inputs) 2025-08-26T20:28:49.1707966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1708035Z outputs = self.deberta( 2025-08-26T20:28:49.1708306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1708386Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1708658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1708753Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1708977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1709060Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1709338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1709432Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1709711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1709790Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1710067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1710278Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1710281Z 2025-08-26T20:28:49.1710386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1710596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1710679Z return mod(**inputs) 2025-08-26T20:28:49.1710963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1711049Z outputs = self.deberta( 2025-08-26T20:28:49.1711321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1711402Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1711673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1711766Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1711989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1712077Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1712351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1712444Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1712725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1712820Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1713101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1713302Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1713306Z 2025-08-26T20:28:49.1713421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1713633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1713730Z return mod(**inputs) 2025-08-26T20:28:49.1714033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1714106Z outputs = self.deberta( 2025-08-26T20:28:49.1714398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1714476Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1714763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1714862Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1715100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1715192Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1715485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1715582Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1715877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1715959Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1716263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1716466Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1716817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1716959Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1716966Z 2025-08-26T20:28:49.1717077Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1717312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1717383Z return mod(**inputs) 2025-08-26T20:28:49.1717742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1717815Z outputs = self.deberta( 2025-08-26T20:28:49.1718100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1718174Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1718448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1718543Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1718771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1718861Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1719156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1719315Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1719641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1719724Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1720037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1720116Z context_layer = torch.bmm( 2025-08-26T20:28:49.1720121Z 2025-08-26T20:28:49.1720241Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1720484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1720558Z return mod(**inputs) 2025-08-26T20:28:49.1720879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1720955Z outputs = self.deberta( 2025-08-26T20:28:49.1721264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1721344Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1721633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1721732Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1721957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1722051Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1722348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1722447Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1722756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1722840Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1723139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1723339Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1723343Z 2025-08-26T20:28:49.1723460Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1723676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1723746Z return mod(**inputs) 2025-08-26T20:28:49.1724070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1724144Z outputs = self.deberta( 2025-08-26T20:28:49.1724464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1724544Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1724840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1724934Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1725172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1725265Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1725564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1725668Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1725967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1726094Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1726410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1726500Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1726504Z 2025-08-26T20:28:49.1726622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1726834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1726929Z return mod(**inputs) 2025-08-26T20:28:49.1727222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1727295Z outputs = self.deberta( 2025-08-26T20:28:49.1727595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1727671Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1727971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1728063Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1728299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1728392Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1728692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1728830Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1729117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1729208Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1729220Z 2025-08-26T20:28:49.1729329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1729542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1729620Z return mod(**inputs) 2025-08-26T20:28:49.1729923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1730003Z outputs = self.deberta( 2025-08-26T20:28:49.1730302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1730380Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1730695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1730805Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1731048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1731134Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1731420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1731555Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1731857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1731985Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1732211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1732294Z return self.act(input) 2025-08-26T20:28:49.1732298Z 2025-08-26T20:28:49.1732408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1732621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1732727Z return mod(**inputs) 2025-08-26T20:28:49.1733001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1733078Z outputs = self.deberta( 2025-08-26T20:28:49.1733353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1733427Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1733726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1733812Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1734045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1734125Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1734402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1734538Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1734810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1734901Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1734905Z 2025-08-26T20:28:49.1735011Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1735219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1735285Z return mod(**inputs) 2025-08-26T20:28:49.1735560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1735636Z outputs = self.deberta( 2025-08-26T20:28:49.1735908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1735989Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1736260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1736354Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1736576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1736658Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1736956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1737054Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1737353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1737436Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1737707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1737906Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1737910Z 2025-08-26T20:28:49.1738015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1738225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1738290Z return mod(**inputs) 2025-08-26T20:28:49.1738574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1738644Z outputs = self.deberta( 2025-08-26T20:28:49.1738915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1739021Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1739295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1739388Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1739614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1739714Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1739994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1740087Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1740366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1740446Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1740724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1740906Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1740909Z 2025-08-26T20:28:49.1741015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1741219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1741286Z return mod(**inputs) 2025-08-26T20:28:49.1741568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1741638Z outputs = self.deberta( 2025-08-26T20:28:49.1741911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1741992Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1742265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1742359Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1742583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1742669Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1742942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1743056Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1743351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1743431Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1743711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1743904Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1744215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1744354Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1744360Z 2025-08-26T20:28:49.1744467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1744676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1744742Z return mod(**inputs) 2025-08-26T20:28:49.1745025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1745109Z outputs = self.deberta( 2025-08-26T20:28:49.1745387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1745465Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1745725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1745815Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1746056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1746135Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1746415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1746508Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1746789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1746866Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1747137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1747347Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1747352Z 2025-08-26T20:28:49.1747454Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1747658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1747721Z return mod(**inputs) 2025-08-26T20:28:49.1747997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1748064Z outputs = self.deberta( 2025-08-26T20:28:49.1748329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1748405Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1748666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1748759Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1748976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1749064Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1749350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1749461Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1749740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1749819Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1750100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1750309Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1750313Z 2025-08-26T20:28:49.1750424Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1750625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1750694Z return mod(**inputs) 2025-08-26T20:28:49.1750976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1751045Z outputs = self.deberta( 2025-08-26T20:28:49.1751326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1751415Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1751689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1751784Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1752007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1752111Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1752382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1752475Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1752755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1752834Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1753113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1753307Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1753311Z 2025-08-26T20:28:49.1753423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1753625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1753691Z return mod(**inputs) 2025-08-26T20:28:49.1753986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1754058Z outputs = self.deberta( 2025-08-26T20:28:49.1754356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1754434Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1754735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1754826Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1755062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1755152Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1755463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1755572Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1755880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1755964Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1756260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1756464Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1756803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1756945Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1756951Z 2025-08-26T20:28:49.1757071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1757283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1757352Z return mod(**inputs) 2025-08-26T20:28:49.1757655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1757748Z outputs = self.deberta( 2025-08-26T20:28:49.1758047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1758123Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1758411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1758510Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1758767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1758861Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1759147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1759320Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1759622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1759704Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1760004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1760081Z context_layer = torch.bmm( 2025-08-26T20:28:49.1760086Z 2025-08-26T20:28:49.1760208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1760429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1760497Z return mod(**inputs) 2025-08-26T20:28:49.1760782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1760854Z outputs = self.deberta( 2025-08-26T20:28:49.1761135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1761211Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1761496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1761584Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1761810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1761898Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1762199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1762304Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1762616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1762700Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1763005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1763196Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1763199Z 2025-08-26T20:28:49.1763308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1763508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1763579Z return mod(**inputs) 2025-08-26T20:28:49.1763855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1763924Z outputs = self.deberta( 2025-08-26T20:28:49.1764201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1764293Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1764574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1764661Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1764883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1764986Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1765260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1765360Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1765633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1765758Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1766033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1766117Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1766121Z 2025-08-26T20:28:49.1766231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1766431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1766505Z return mod(**inputs) 2025-08-26T20:28:49.1766783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1766856Z outputs = self.deberta( 2025-08-26T20:28:49.1767151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1767229Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1767533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1767619Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1767851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1767929Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1768197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1768343Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1768618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1768734Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1768738Z 2025-08-26T20:28:49.1768850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1769065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1769145Z return mod(**inputs) 2025-08-26T20:28:49.1769436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1769517Z outputs = self.deberta( 2025-08-26T20:28:49.1769806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1769884Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1770188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1770277Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1770508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1770607Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1770884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1771003Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1771271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1771411Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1771628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1771709Z return self.act(input) 2025-08-26T20:28:49.1771712Z 2025-08-26T20:28:49.1771819Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1772022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1772096Z return mod(**inputs) 2025-08-26T20:28:49.1772375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1772454Z outputs = self.deberta( 2025-08-26T20:28:49.1772727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1772809Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1773085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1773171Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1773404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1773484Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1773764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1773897Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1774171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1774262Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1774266Z 2025-08-26T20:28:49.1774372Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1774616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1774685Z return mod(**inputs) 2025-08-26T20:28:49.1774984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1775057Z outputs = self.deberta( 2025-08-26T20:28:49.1775327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1775408Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1775677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1775770Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1775991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1776072Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1776354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1776449Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1776727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1776825Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1777105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1777297Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1777301Z 2025-08-26T20:28:49.1777403Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1777630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1777698Z return mod(**inputs) 2025-08-26T20:28:49.1777990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1778062Z outputs = self.deberta( 2025-08-26T20:28:49.1778325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1778404Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1778667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1778757Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1778993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1779084Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1779373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1779473Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1779768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1779853Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1780145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1780338Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1780342Z 2025-08-26T20:28:49.1780461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1780675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1780757Z return mod(**inputs) 2025-08-26T20:28:49.1781046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1781115Z outputs = self.deberta( 2025-08-26T20:28:49.1781405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1781482Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1781747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1781840Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1782060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1782148Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1782420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1782527Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1782799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1782874Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1783166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1783350Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1783658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1783792Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1783812Z 2025-08-26T20:28:49.1783916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1784120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1784187Z return mod(**inputs) 2025-08-26T20:28:49.1784470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1784540Z outputs = self.deberta( 2025-08-26T20:28:49.1784807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1784887Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1785150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1785244Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1785462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1785548Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1785814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1785906Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1786178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1786256Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1786528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1786733Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1786739Z 2025-08-26T20:28:49.1786848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1787068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1787133Z return mod(**inputs) 2025-08-26T20:28:49.1787425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1787495Z outputs = self.deberta( 2025-08-26T20:28:49.1787771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1787842Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1788108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1788203Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1788427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1788518Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1788792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1788886Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1789165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1789262Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1789548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1789757Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1789760Z 2025-08-26T20:28:49.1789882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1790094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1790160Z return mod(**inputs) 2025-08-26T20:28:49.1790439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1790509Z outputs = self.deberta( 2025-08-26T20:28:49.1790795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1790871Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1791149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1791237Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1791462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1791551Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1791828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1791929Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1792201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1792280Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1792560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1792755Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1792759Z 2025-08-26T20:28:49.1792872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1793079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1793156Z return mod(**inputs) 2025-08-26T20:28:49.1793467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1793542Z outputs = self.deberta( 2025-08-26T20:28:49.1793853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1793933Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1794229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1794321Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1794558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1794650Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1794941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1795047Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1795339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1795428Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1795736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1795941Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1796543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1796690Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1796738Z 2025-08-26T20:28:49.1796862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1797076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1797146Z return mod(**inputs) 2025-08-26T20:28:49.1797461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1797535Z outputs = self.deberta( 2025-08-26T20:28:49.1797831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1797909Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1798207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1798299Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1798538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1798631Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1798919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1799026Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1799359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1799448Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1799760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1799837Z context_layer = torch.bmm( 2025-08-26T20:28:49.1799842Z 2025-08-26T20:28:49.1799962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1800174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1800295Z return mod(**inputs) 2025-08-26T20:28:49.1800598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1800693Z outputs = self.deberta( 2025-08-26T20:28:49.1800974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1801050Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1801333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1801422Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1801647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1801742Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1802042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1802146Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1802443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1802557Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1802854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1803058Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1803062Z 2025-08-26T20:28:49.1803181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1803413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1803489Z return mod(**inputs) 2025-08-26T20:28:49.1803789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1803858Z outputs = self.deberta( 2025-08-26T20:28:49.1804140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1804215Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1804501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1804588Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1804818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1804901Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1805177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1805277Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1805575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1805710Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1806002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1806092Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1806105Z 2025-08-26T20:28:49.1806220Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1806423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1806499Z return mod(**inputs) 2025-08-26T20:28:49.1806795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1806872Z outputs = self.deberta( 2025-08-26T20:28:49.1807186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1807266Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1807568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1807661Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1807908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1807992Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1808294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1808435Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1808733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1808828Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1808832Z 2025-08-26T20:28:49.1808960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1809174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1809243Z return mod(**inputs) 2025-08-26T20:28:49.1809548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1809629Z outputs = self.deberta( 2025-08-26T20:28:49.1809924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1810029Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1810375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1810471Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1810718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1810805Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1811106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1811233Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1811541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1811665Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1811895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1811977Z return self.act(input) 2025-08-26T20:28:49.1811981Z 2025-08-26T20:28:49.1812090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1812309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1812380Z return mod(**inputs) 2025-08-26T20:28:49.1812674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1812755Z outputs = self.deberta( 2025-08-26T20:28:49.1813040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1813122Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1813431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1813524Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1813789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1813874Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1814173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1814314Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1814611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1814701Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1814705Z 2025-08-26T20:28:49.1814818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1815040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1815110Z return mod(**inputs) 2025-08-26T20:28:49.1815412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1815485Z outputs = self.deberta( 2025-08-26T20:28:49.1815791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1815875Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1816175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1816274Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1816512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1816623Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1816914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1817014Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1817313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1817398Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1817694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1817898Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1817903Z 2025-08-26T20:28:49.1818022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1818236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1818309Z return mod(**inputs) 2025-08-26T20:28:49.1818610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1818684Z outputs = self.deberta( 2025-08-26T20:28:49.1818982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1819062Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1819349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1819460Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1819697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1819789Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1820093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1820195Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1820506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1820592Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1820887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1821079Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1821083Z 2025-08-26T20:28:49.1821201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1821411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1821482Z return mod(**inputs) 2025-08-26T20:28:49.1821789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1821862Z outputs = self.deberta( 2025-08-26T20:28:49.1822158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1822253Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1822542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1822641Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1822879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1822972Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1823278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1823386Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1823673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1823756Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1824052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1824249Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1824586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1824727Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1824733Z 2025-08-26T20:28:49.1824850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1825064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1825132Z return mod(**inputs) 2025-08-26T20:28:49.1825432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1825505Z outputs = self.deberta( 2025-08-26T20:28:49.1825797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1825874Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1826158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1826259Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1826495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1826601Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1826888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1827011Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1827299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1827381Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1827676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1827901Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1827906Z 2025-08-26T20:28:49.1828023Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1828238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1828309Z return mod(**inputs) 2025-08-26T20:28:49.1828612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1828685Z outputs = self.deberta( 2025-08-26T20:28:49.1829001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1829079Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1829376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1829470Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1829705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1829816Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1830104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1830208Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1830493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1830577Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1830872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1831094Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1831098Z 2025-08-26T20:28:49.1831217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1831429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1831508Z return mod(**inputs) 2025-08-26T20:28:49.1831797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1831873Z outputs = self.deberta( 2025-08-26T20:28:49.1832165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1832245Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1832539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1832630Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1832865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1832959Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1833269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1833376Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1833680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1833772Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1834060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1834266Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1834270Z 2025-08-26T20:28:49.1834389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1834604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1834682Z return mod(**inputs) 2025-08-26T20:28:49.1834980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1835054Z outputs = self.deberta( 2025-08-26T20:28:49.1835355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1835462Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1835758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1835850Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1836093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1836176Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1836484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1836592Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1836879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1836968Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1837255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1837456Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1837796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1837937Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1837943Z 2025-08-26T20:28:49.1838063Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1838277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1838357Z return mod(**inputs) 2025-08-26T20:28:49.1838655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1838731Z outputs = self.deberta( 2025-08-26T20:28:49.1839029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1839108Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1839493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1839591Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1839869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1839967Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1840281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1840396Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1840705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1840797Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1841086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1841163Z context_layer = torch.bmm( 2025-08-26T20:28:49.1841167Z 2025-08-26T20:28:49.1841287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1841500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1841580Z return mod(**inputs) 2025-08-26T20:28:49.1841871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1841944Z outputs = self.deberta( 2025-08-26T20:28:49.1842241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1842336Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1842632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1842724Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1842968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1843073Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1843360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1843467Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1843752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1843837Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1844107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1844298Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1844309Z 2025-08-26T20:28:49.1844415Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1844617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1844691Z return mod(**inputs) 2025-08-26T20:28:49.1844976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1845051Z outputs = self.deberta( 2025-08-26T20:28:49.1845317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1845389Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1845658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1845740Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1845963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1846040Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1846319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1846418Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1846699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1846823Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1847091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1847177Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1847181Z 2025-08-26T20:28:49.1847282Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1847476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1847551Z return mod(**inputs) 2025-08-26T20:28:49.1847821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1847896Z outputs = self.deberta( 2025-08-26T20:28:49.1848165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1848238Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1848534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1848621Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1848855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1848935Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1849213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1849355Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1849628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1849720Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1849723Z 2025-08-26T20:28:49.1849829Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1850050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1850119Z return mod(**inputs) 2025-08-26T20:28:49.1850412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1850492Z outputs = self.deberta( 2025-08-26T20:28:49.1850777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1850863Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1851151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1851252Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1851489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1851573Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1851871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1851997Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1852292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1852412Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1852657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1852741Z return self.act(input) 2025-08-26T20:28:49.1852745Z 2025-08-26T20:28:49.1852871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1853092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1853164Z return mod(**inputs) 2025-08-26T20:28:49.1853467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1853540Z outputs = self.deberta( 2025-08-26T20:28:49.1853832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1853912Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1854190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1854283Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1854507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1854588Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1854889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1855030Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1855324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1855413Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1855417Z 2025-08-26T20:28:49.1855551Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1855770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1855838Z return mod(**inputs) 2025-08-26T20:28:49.1856127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1856195Z outputs = self.deberta( 2025-08-26T20:28:49.1856482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1856556Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1856835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1856930Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1857155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1857243Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1857521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1857617Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1857901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1857982Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1858266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1858462Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1858466Z 2025-08-26T20:28:49.1858577Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1858784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1858869Z return mod(**inputs) 2025-08-26T20:28:49.1859162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1859254Z outputs = self.deberta( 2025-08-26T20:28:49.1859532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1859606Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1859876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1859969Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1860190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1860277Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1860555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1860659Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1860948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1861052Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1861347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-26T20:28:49.1861541Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1861545Z 2025-08-26T20:28:49.1861663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1861876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1861973Z return mod(**inputs) 2025-08-26T20:28:49.1862277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1862351Z outputs = self.deberta( 2025-08-26T20:28:49.1862653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1862731Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1863035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1863127Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1863368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1863461Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1863758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1863859Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1864138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1864214Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1864501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-26T20:28:49.1864689Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-26T20:28:49.1865014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1865149Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1865154Z 2025-08-26T20:28:49.1865267Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1865489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1865561Z return mod(**inputs) 2025-08-26T20:28:49.1865890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1865969Z outputs = self.deberta( 2025-08-26T20:28:49.1866272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1866351Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1866660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1866753Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1866992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1867089Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1867387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1867494Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1867810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1867891Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1868208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1868445Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1868467Z 2025-08-26T20:28:49.1868590Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1868821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1868896Z return mod(**inputs) 2025-08-26T20:28:49.1869201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1869275Z outputs = self.deberta( 2025-08-26T20:28:49.1869571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1869647Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1869954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1870048Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1870285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1870380Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1870677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1870777Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1871050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1871135Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1871410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-26T20:28:49.1871624Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-26T20:28:49.1871628Z 2025-08-26T20:28:49.1871739Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1871941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1872036Z return mod(**inputs) 2025-08-26T20:28:49.1872340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1872429Z outputs = self.deberta( 2025-08-26T20:28:49.1872730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1872807Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1873106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1873200Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1873443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1873530Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1873822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1873930Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1874217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1874677Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1874973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1875188Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1875201Z 2025-08-26T20:28:49.1875318Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1875575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1875654Z return mod(**inputs) 2025-08-26T20:28:49.1875957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1876041Z outputs = self.deberta( 2025-08-26T20:28:49.1876338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1876419Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1876721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1876815Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1877068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1877155Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1877454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1877562Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1877861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1877954Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1878255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-26T20:28:49.1878470Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-26T20:28:49.1878827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-26T20:28:49.1878977Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-26T20:28:49.1878981Z 2025-08-26T20:28:49.1879120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1879446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1879532Z return mod(**inputs) 2025-08-26T20:28:49.1879862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1879951Z outputs = self.deberta( 2025-08-26T20:28:49.1880249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1880328Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1880636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1880732Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1880989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1881075Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1881371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1881483Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1881799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1881892Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1882189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-26T20:28:49.1882271Z context_layer = torch.bmm( 2025-08-26T20:28:49.1882282Z 2025-08-26T20:28:49.1882417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1882636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1882718Z return mod(**inputs) 2025-08-26T20:28:49.1883021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1883103Z outputs = self.deberta( 2025-08-26T20:28:49.1883401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1883482Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1883786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1883879Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1884131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1884219Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1884515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1884623Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1884920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-26T20:28:49.1885011Z self_output, att_matrix = self.self( 2025-08-26T20:28:49.1885308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-26T20:28:49.1885524Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-26T20:28:49.1885528Z 2025-08-26T20:28:49.1885640Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1885860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1885940Z return mod(**inputs) 2025-08-26T20:28:49.1886269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1886355Z outputs = self.deberta( 2025-08-26T20:28:49.1886673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1886756Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1887068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1887163Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1887412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1887501Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1887818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-26T20:28:49.1887917Z attention_output, att_matrix = self.attention( 2025-08-26T20:28:49.1888208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-26T20:28:49.1888340Z attention_output = self.output(self_output, query_states) 2025-08-26T20:28:49.1888650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-26T20:28:49.1888747Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1888751Z 2025-08-26T20:28:49.1888861Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1889074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1889171Z return mod(**inputs) 2025-08-26T20:28:49.1889462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1889544Z outputs = self.deberta( 2025-08-26T20:28:49.1889835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1889919Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1890208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1890301Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1890547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1890630Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1891148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1891291Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1891578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-26T20:28:49.1891676Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1891681Z 2025-08-26T20:28:49.1891794Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1892011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1892081Z return mod(**inputs) 2025-08-26T20:28:49.1892382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1892456Z outputs = self.deberta( 2025-08-26T20:28:49.1892744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1892832Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1893138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1893258Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1893500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1893587Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1893882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-26T20:28:49.1894010Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:28:49.1894306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-26T20:28:49.1894431Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:28:49.1894666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:28:49.1894742Z return self.act(input) 2025-08-26T20:28:49.1894745Z 2025-08-26T20:28:49.1894858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1895078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1895168Z return mod(**inputs) 2025-08-26T20:28:49.1895468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-26T20:28:49.1895541Z outputs = self.deberta( 2025-08-26T20:28:49.1895839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-26T20:28:49.1895924Z encoder_outputs = self.encoder( 2025-08-26T20:28:49.1896397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-26T20:28:49.1896502Z output_states, attn_weights = layer_module( 2025-08-26T20:28:49.1896740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:28:49.1896825Z return super().__call__(*args, **kwargs) 2025-08-26T20:28:49.1897122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-26T20:28:49.1897265Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:28:49.1897558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-26T20:28:49.1897644Z hidden_states = self.dense(hidden_states) 2025-08-26T20:28:49.1897648Z 2025-08-26T20:28:49.1897770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1897981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1898053Z return mod(**inputs) 2025-08-26T20:28:49.1898355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1244, in forward 2025-08-26T20:28:49.1898445Z logits = self.qa_outputs(sequence_output) 2025-08-26T20:28:49.1898451Z 2025-08-26T20:28:49.1898569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1898778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1898856Z return mod(**inputs) 2025-08-26T20:28:49.1899160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1262, in forward 2025-08-26T20:28:49.1899274Z start_loss = loss_fct(start_logits, start_positions) 2025-08-26T20:28:49.1899280Z 2025-08-26T20:28:49.1899397Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:28:49.1899651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:28:49.1899732Z return mod(**inputs) 2025-08-26T20:28:49.1900052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1263, in forward 2025-08-26T20:28:49.1900154Z end_loss = loss_fct(end_logits, end_positions) 2025-08-26T20:28:49.1900157Z 2025-08-26T20:29:02.0435527Z Compilation time (from dynamo_timed): 26.264198034 2025-08-26T20:29:02.0435865Z pass 2025-08-26T20:29:02.0436198Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:02.0437076Z TIMING: _recursive_pre_grad_passes:0.01366 _recursive_joint_graph_passes:1.19353 _recursive_post_grad_passes:0.31573 async_compile.wait:0.55659 code_gen:11.19094 inductor_compile:14.35138 backend_compile:21.02076 gc:0.00059 entire_frame_compile:26.2642 total_wall_time:26.2642 2025-08-26T20:29:02.0438283Z STATS: call_* op count: 1087 | FakeTensorMode.__torch_dispatch__:30534 | FakeTensor.__torch_dispatch__:10573 | ProxyTorchDispatchMode.__torch_dispatch__:11524 2025-08-26T20:29:02.0438886Z Dynamo produced 1 graphs covering 1087 ops with 0 graph breaks (0 unique) 2025-08-26T20:29:07.9505183Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:29:07.9507561Z from pkg_resources import resource_filename 2025-08-26T20:29:08.5436000Z 2025-08-26T20:29:09.4813703Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:29:09.4818310Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:29:09.4822382Z cpu eval DistilBertForMaskedLM 2025-08-26T20:29:09.6552731Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:09.7114408Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:09.7675483Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:14.8943130Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.8946841Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.8947526Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.8947789Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.8948006Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.8948228Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.8948493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.8948911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.8950710Z return mod(**inputs) 2025-08-26T20:29:14.8951231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.8951700Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.8952161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.8952615Z return self.transformer( 2025-08-26T20:29:14.8953051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.8953494Z layer_outputs = layer_module( 2025-08-26T20:29:14.8953892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.8954316Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.8954787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.8955256Z sa_output = self.attention( 2025-08-26T20:29:14.8956040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:14.8956691Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:14.8956916Z 2025-08-26T20:29:14.8957043Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.8957456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.8957817Z return mod(**inputs) 2025-08-26T20:29:14.8958267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.8958734Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.8959400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.8959892Z return self.transformer( 2025-08-26T20:29:14.8960337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.8960798Z layer_outputs = layer_module( 2025-08-26T20:29:14.8961184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.8961653Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.8962113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.8962572Z sa_output = self.attention( 2025-08-26T20:29:14.8963009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:14.8963560Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.8963752Z 2025-08-26T20:29:14.8963876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.8964266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.8964613Z return mod(**inputs) 2025-08-26T20:29:14.8965034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.8965484Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.8965925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.8966362Z return self.transformer( 2025-08-26T20:29:14.8966788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.8967228Z layer_outputs = layer_module( 2025-08-26T20:29:14.8967599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.8967993Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.8968446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.8968905Z sa_output = self.attention( 2025-08-26T20:29:14.8969349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:14.8969868Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.8970069Z 2025-08-26T20:29:14.8970177Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.8970431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.8970824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.8971177Z return mod(**inputs) 2025-08-26T20:29:14.8971773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.8972231Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.8972714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.8973171Z return self.transformer( 2025-08-26T20:29:14.8973609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.8974081Z layer_outputs = layer_module( 2025-08-26T20:29:14.8974459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.8974862Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.8975338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.8975801Z sa_output = self.attention( 2025-08-26T20:29:14.8976261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:14.8976784Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:14.8976996Z 2025-08-26T20:29:14.8977134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.8977532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.8977894Z return mod(**inputs) 2025-08-26T20:29:14.8978329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.8978799Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.8979256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.8979725Z return self.transformer( 2025-08-26T20:29:14.8980183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.8980603Z layer_outputs = layer_module( 2025-08-26T20:29:14.8980979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.8981365Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.8981800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.8982243Z sa_output = self.attention( 2025-08-26T20:29:14.8982669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:14.8983093Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:14.8983234Z 2025-08-26T20:29:14.8983347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.8983704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.8984038Z return mod(**inputs) 2025-08-26T20:29:14.8984429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.8984847Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.8985249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.8985659Z return self.transformer( 2025-08-26T20:29:14.8986052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.8986463Z layer_outputs = layer_module( 2025-08-26T20:29:14.8986810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.8987228Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.8987656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.8988133Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.8988592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.8989140Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.8989665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.8990078Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.8990500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:14.8990923Z x = self.lin1(input) 2025-08-26T20:29:14.8991038Z 2025-08-26T20:29:14.8991155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.8991539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.8991890Z return mod(**inputs) 2025-08-26T20:29:14.8992330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.8992755Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.8993163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.8993601Z return self.transformer( 2025-08-26T20:29:14.8994025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.8994490Z layer_outputs = layer_module( 2025-08-26T20:29:14.8994865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.8995253Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.8995705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.8996534Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.8997041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.8997626Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.8998180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.8998621Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.8999078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:14.8999618Z x = self.activation(x) 2025-08-26T20:29:14.8999987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:14.9000369Z return self.act(input) 2025-08-26T20:29:14.9000504Z 2025-08-26T20:29:14.9000624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9001032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9001401Z return mod(**inputs) 2025-08-26T20:29:14.9001816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9002273Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9002761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9003210Z return self.transformer( 2025-08-26T20:29:14.9003665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9004122Z layer_outputs = layer_module( 2025-08-26T20:29:14.9004515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9004969Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9005417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9005895Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9006376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9006966Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9007523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9007952Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9008388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:14.9008856Z x = self.lin2(x) 2025-08-26T20:29:14.9008968Z 2025-08-26T20:29:14.9009081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9009477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9009822Z return mod(**inputs) 2025-08-26T20:29:14.9010213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9010688Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9011131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9011576Z return self.transformer( 2025-08-26T20:29:14.9012002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9012443Z layer_outputs = layer_module( 2025-08-26T20:29:14.9012818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9013210Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9013665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9014109Z sa_output = self.attention( 2025-08-26T20:29:14.9014533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:14.9015007Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:14.9015192Z 2025-08-26T20:29:14.9015311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9015681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9016008Z return mod(**inputs) 2025-08-26T20:29:14.9016409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9016835Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9017263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9017709Z return self.transformer( 2025-08-26T20:29:14.9018163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9018603Z layer_outputs = layer_module( 2025-08-26T20:29:14.9018973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9019346Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9019769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9020187Z sa_output = self.attention( 2025-08-26T20:29:14.9020597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:14.9021067Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9021250Z 2025-08-26T20:29:14.9021366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9021752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9022115Z return mod(**inputs) 2025-08-26T20:29:14.9022511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9022953Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9023417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9023823Z return self.transformer( 2025-08-26T20:29:14.9024231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9024647Z layer_outputs = layer_module( 2025-08-26T20:29:14.9025000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9025381Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9025804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9026218Z sa_output = self.attention( 2025-08-26T20:29:14.9026621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:14.9027091Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9027273Z 2025-08-26T20:29:14.9027356Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.9027599Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9027962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9028296Z return mod(**inputs) 2025-08-26T20:29:14.9028693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9029111Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9029527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9029943Z return self.transformer( 2025-08-26T20:29:14.9030356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9030794Z layer_outputs = layer_module( 2025-08-26T20:29:14.9031170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9031564Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9031992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9032437Z sa_output = self.attention( 2025-08-26T20:29:14.9032878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:14.9033390Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:14.9033599Z 2025-08-26T20:29:14.9033731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9034121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9034479Z return mod(**inputs) 2025-08-26T20:29:14.9034893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9035337Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9035774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9036221Z return self.transformer( 2025-08-26T20:29:14.9036640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9037076Z layer_outputs = layer_module( 2025-08-26T20:29:14.9037452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9037842Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9038308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9038737Z sa_output = self.attention( 2025-08-26T20:29:14.9039173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:14.9039755Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:14.9039919Z 2025-08-26T20:29:14.9040047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9040481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9040827Z return mod(**inputs) 2025-08-26T20:29:14.9041245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9041697Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9042145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9042593Z return self.transformer( 2025-08-26T20:29:14.9043014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9043468Z layer_outputs = layer_module( 2025-08-26T20:29:14.9043849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9044239Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9044681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9045165Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9045646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9046236Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9046792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9047213Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9047662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:14.9048107Z x = self.lin1(input) 2025-08-26T20:29:14.9048219Z 2025-08-26T20:29:14.9048341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9048757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9049082Z return mod(**inputs) 2025-08-26T20:29:14.9049485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9049913Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9050325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9050737Z return self.transformer( 2025-08-26T20:29:14.9051138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9051566Z layer_outputs = layer_module( 2025-08-26T20:29:14.9051929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9052317Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9052757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9053241Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9053747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9054336Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9054891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9055312Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9055779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:14.9056231Z x = self.activation(x) 2025-08-26T20:29:14.9056588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:14.9056955Z return self.act(input) 2025-08-26T20:29:14.9057074Z 2025-08-26T20:29:14.9057202Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9057605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9057961Z return mod(**inputs) 2025-08-26T20:29:14.9058382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9058817Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9059259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9059707Z return self.transformer( 2025-08-26T20:29:14.9060134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9060580Z layer_outputs = layer_module( 2025-08-26T20:29:14.9060948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9061340Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9061785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9062265Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9062748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9063320Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9063913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9064343Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9064815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:14.9065258Z x = self.lin2(x) 2025-08-26T20:29:14.9065365Z 2025-08-26T20:29:14.9065478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9065865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9066216Z return mod(**inputs) 2025-08-26T20:29:14.9066638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9067083Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9067612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9068069Z return self.transformer( 2025-08-26T20:29:14.9068534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9068979Z layer_outputs = layer_module( 2025-08-26T20:29:14.9069375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9069779Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9070228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9070680Z sa_output = self.attention( 2025-08-26T20:29:14.9071116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:14.9071656Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:14.9071862Z 2025-08-26T20:29:14.9071975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9072366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9072722Z return mod(**inputs) 2025-08-26T20:29:14.9073133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9073578Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9074017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9074458Z return self.transformer( 2025-08-26T20:29:14.9074897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9075360Z layer_outputs = layer_module( 2025-08-26T20:29:14.9075746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9076152Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9076614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9077068Z sa_output = self.attention( 2025-08-26T20:29:14.9077512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:14.9078028Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9078229Z 2025-08-26T20:29:14.9078344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9078745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9079106Z return mod(**inputs) 2025-08-26T20:29:14.9080439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9080931Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9081419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9081877Z return self.transformer( 2025-08-26T20:29:14.9082315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9082785Z layer_outputs = layer_module( 2025-08-26T20:29:14.9083170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9083584Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9084053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9084519Z sa_output = self.attention( 2025-08-26T20:29:14.9084957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:14.9085480Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9085706Z 2025-08-26T20:29:14.9085803Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.9086071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9086457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9086819Z return mod(**inputs) 2025-08-26T20:29:14.9087211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9087655Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9088065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9088482Z return self.transformer( 2025-08-26T20:29:14.9088882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9089305Z layer_outputs = layer_module( 2025-08-26T20:29:14.9089682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9090062Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9090485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9090901Z sa_output = self.attention( 2025-08-26T20:29:14.9091306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:14.9091786Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:14.9091976Z 2025-08-26T20:29:14.9092086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9092454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9092783Z return mod(**inputs) 2025-08-26T20:29:14.9093180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9093598Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9094013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9094430Z return self.transformer( 2025-08-26T20:29:14.9094838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9095260Z layer_outputs = layer_module( 2025-08-26T20:29:14.9095635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9096010Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9096693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9097128Z sa_output = self.attention( 2025-08-26T20:29:14.9097533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:14.9097968Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:14.9098122Z 2025-08-26T20:29:14.9098230Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9098600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9098940Z return mod(**inputs) 2025-08-26T20:29:14.9099331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9099753Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9100170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9100622Z return self.transformer( 2025-08-26T20:29:14.9101030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9101447Z layer_outputs = layer_module( 2025-08-26T20:29:14.9101801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9102171Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9102593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9103089Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9103540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9104091Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9104635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9105062Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9105506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:14.9105962Z x = self.lin1(input) 2025-08-26T20:29:14.9106085Z 2025-08-26T20:29:14.9106200Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9106604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9106963Z return mod(**inputs) 2025-08-26T20:29:14.9107382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9107849Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9108249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9108658Z return self.transformer( 2025-08-26T20:29:14.9109051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9109481Z layer_outputs = layer_module( 2025-08-26T20:29:14.9109852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9110244Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9110725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9111204Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9111689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9112239Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9112744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9113130Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9113542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:14.9113981Z x = self.activation(x) 2025-08-26T20:29:14.9114333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:14.9114698Z return self.act(input) 2025-08-26T20:29:14.9114815Z 2025-08-26T20:29:14.9114935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9115317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9115699Z return mod(**inputs) 2025-08-26T20:29:14.9116131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9116588Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9117037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9117484Z return self.transformer( 2025-08-26T20:29:14.9117948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9118407Z layer_outputs = layer_module( 2025-08-26T20:29:14.9118795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9119192Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9119726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9120234Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9120724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9121323Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9121894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9122337Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9122797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:14.9123248Z x = self.lin2(x) 2025-08-26T20:29:14.9123360Z 2025-08-26T20:29:14.9123483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9123876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9124237Z return mod(**inputs) 2025-08-26T20:29:14.9124665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9125122Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9125574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9126020Z return self.transformer( 2025-08-26T20:29:14.9126474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9126928Z layer_outputs = layer_module( 2025-08-26T20:29:14.9127337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9127692Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9128104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9128517Z sa_output = self.attention( 2025-08-26T20:29:14.9128918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:14.9129380Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:14.9129564Z 2025-08-26T20:29:14.9129670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9130048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9130398Z return mod(**inputs) 2025-08-26T20:29:14.9130812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9131289Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9131723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9132166Z return self.transformer( 2025-08-26T20:29:14.9132568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9132986Z layer_outputs = layer_module( 2025-08-26T20:29:14.9133351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9133725Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9134145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9134564Z sa_output = self.attention( 2025-08-26T20:29:14.9134968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:14.9135444Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9135630Z 2025-08-26T20:29:14.9135735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9136095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9136420Z return mod(**inputs) 2025-08-26T20:29:14.9136814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9137225Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9137634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9138048Z return self.transformer( 2025-08-26T20:29:14.9138447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9138858Z layer_outputs = layer_module( 2025-08-26T20:29:14.9139202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9139586Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9140010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9140426Z sa_output = self.attention( 2025-08-26T20:29:14.9140836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:14.9141304Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9141492Z 2025-08-26T20:29:14.9141593Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.9141854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9142243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9142578Z return mod(**inputs) 2025-08-26T20:29:14.9142973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9143391Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9143802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9144212Z return self.transformer( 2025-08-26T20:29:14.9144673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9145087Z layer_outputs = layer_module( 2025-08-26T20:29:14.9145439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9145828Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9146265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9146685Z sa_output = self.attention( 2025-08-26T20:29:14.9147093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:14.9147573Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:14.9147775Z 2025-08-26T20:29:14.9147887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9148246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9148577Z return mod(**inputs) 2025-08-26T20:29:14.9148976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9149402Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9149815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9150225Z return self.transformer( 2025-08-26T20:29:14.9150624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9151040Z layer_outputs = layer_module( 2025-08-26T20:29:14.9151393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9151752Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9152171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9152583Z sa_output = self.attention( 2025-08-26T20:29:14.9152985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:14.9153411Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:14.9153550Z 2025-08-26T20:29:14.9153654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9154015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9154343Z return mod(**inputs) 2025-08-26T20:29:14.9154730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9155149Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9155576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9156012Z return self.transformer( 2025-08-26T20:29:14.9156422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9156865Z layer_outputs = layer_module( 2025-08-26T20:29:14.9157229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9157620Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9158062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9158541Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9159019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9159673Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9160241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9160698Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9161150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:14.9161592Z x = self.lin1(input) 2025-08-26T20:29:14.9161704Z 2025-08-26T20:29:14.9161816Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9162210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9162576Z return mod(**inputs) 2025-08-26T20:29:14.9162992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9163433Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9163861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9164299Z return self.transformer( 2025-08-26T20:29:14.9164722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9165162Z layer_outputs = layer_module( 2025-08-26T20:29:14.9165520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9165910Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9166357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9166840Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9167319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9167889Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9168444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9168866Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9169306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:14.9169749Z x = self.activation(x) 2025-08-26T20:29:14.9170094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:14.9170460Z return self.act(input) 2025-08-26T20:29:14.9170588Z 2025-08-26T20:29:14.9170722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9171112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9171472Z return mod(**inputs) 2025-08-26T20:29:14.9171888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9172332Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9172771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9173221Z return self.transformer( 2025-08-26T20:29:14.9173616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9174064Z layer_outputs = layer_module( 2025-08-26T20:29:14.9174438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9174829Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9175277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9175762Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9176241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9176816Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9177361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9177779Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9178197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:14.9178638Z x = self.lin2(x) 2025-08-26T20:29:14.9178751Z 2025-08-26T20:29:14.9178861Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9179255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9179599Z return mod(**inputs) 2025-08-26T20:29:14.9180012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9180448Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9180861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9181277Z return self.transformer( 2025-08-26T20:29:14.9181693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9182133Z layer_outputs = layer_module( 2025-08-26T20:29:14.9182499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9182887Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9183334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9183770Z sa_output = self.attention( 2025-08-26T20:29:14.9184199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:14.9184699Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:14.9184878Z 2025-08-26T20:29:14.9184990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9185375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9185732Z return mod(**inputs) 2025-08-26T20:29:14.9186147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9186623Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9187062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9187493Z return self.transformer( 2025-08-26T20:29:14.9187919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9188356Z layer_outputs = layer_module( 2025-08-26T20:29:14.9188730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9189117Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9189552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9189993Z sa_output = self.attention( 2025-08-26T20:29:14.9190425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:14.9190937Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9191126Z 2025-08-26T20:29:14.9191244Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9191631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9191979Z return mod(**inputs) 2025-08-26T20:29:14.9192396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9192859Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9193291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9193732Z return self.transformer( 2025-08-26T20:29:14.9194160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9194600Z layer_outputs = layer_module( 2025-08-26T20:29:14.9194974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9195357Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9195810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9196430Z sa_output = self.attention( 2025-08-26T20:29:14.9196864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:14.9197380Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9197577Z 2025-08-26T20:29:14.9197668Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.9197941Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9198335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9198699Z return mod(**inputs) 2025-08-26T20:29:14.9199124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9199701Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9200146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9200588Z return self.transformer( 2025-08-26T20:29:14.9201022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9201526Z layer_outputs = layer_module( 2025-08-26T20:29:14.9201900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9202310Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9202763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9203207Z sa_output = self.attention( 2025-08-26T20:29:14.9203630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:14.9204133Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:14.9204342Z 2025-08-26T20:29:14.9204455Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9204839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9205180Z return mod(**inputs) 2025-08-26T20:29:14.9205594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9206040Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9206479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9206922Z return self.transformer( 2025-08-26T20:29:14.9207319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9207736Z layer_outputs = layer_module( 2025-08-26T20:29:14.9208086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9208487Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9208919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9209333Z sa_output = self.attention( 2025-08-26T20:29:14.9209745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:14.9210185Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:14.9210340Z 2025-08-26T20:29:14.9210463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9210863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9211221Z return mod(**inputs) 2025-08-26T20:29:14.9211663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9212101Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9212534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9212957Z return self.transformer( 2025-08-26T20:29:14.9213380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9213838Z layer_outputs = layer_module( 2025-08-26T20:29:14.9214207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9214620Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9215051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9215540Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9216025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9216634Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9217175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9217559Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9217972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:14.9218385Z x = self.lin1(input) 2025-08-26T20:29:14.9218493Z 2025-08-26T20:29:14.9218604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9218962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9219291Z return mod(**inputs) 2025-08-26T20:29:14.9219681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9220102Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9220513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9220924Z return self.transformer( 2025-08-26T20:29:14.9221310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9221737Z layer_outputs = layer_module( 2025-08-26T20:29:14.9222086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9222451Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9222866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9223338Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9223772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9224302Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9224826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9225224Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9225643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:14.9226060Z x = self.activation(x) 2025-08-26T20:29:14.9226394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:14.9226721Z return self.act(input) 2025-08-26T20:29:14.9226840Z 2025-08-26T20:29:14.9226944Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9227305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9227637Z return mod(**inputs) 2025-08-26T20:29:14.9228031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9228439Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9228852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9229269Z return self.transformer( 2025-08-26T20:29:14.9229672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9230083Z layer_outputs = layer_module( 2025-08-26T20:29:14.9230429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9230792Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9231222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9231678Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9232109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9232639Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9233151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9233543Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9233954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:14.9234349Z x = self.lin2(x) 2025-08-26T20:29:14.9234457Z 2025-08-26T20:29:14.9234561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9234918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9235241Z return mod(**inputs) 2025-08-26T20:29:14.9235622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9236052Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9236465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9236878Z return self.transformer( 2025-08-26T20:29:14.9237277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9237706Z layer_outputs = layer_module( 2025-08-26T20:29:14.9238062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9238432Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9238860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9239355Z sa_output = self.attention( 2025-08-26T20:29:14.9239768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:14.9240255Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:14.9240460Z 2025-08-26T20:29:14.9240572Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9240968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9241306Z return mod(**inputs) 2025-08-26T20:29:14.9241702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9242132Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9242556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9242971Z return self.transformer( 2025-08-26T20:29:14.9243361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9243770Z layer_outputs = layer_module( 2025-08-26T20:29:14.9244119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9244481Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9244900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9245343Z sa_output = self.attention( 2025-08-26T20:29:14.9245742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:14.9246209Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9246384Z 2025-08-26T20:29:14.9246495Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9246852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9247171Z return mod(**inputs) 2025-08-26T20:29:14.9247554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9247970Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9248374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9248824Z return self.transformer( 2025-08-26T20:29:14.9249220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9249640Z layer_outputs = layer_module( 2025-08-26T20:29:14.9249992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9250403Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9250849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9251342Z sa_output = self.attention( 2025-08-26T20:29:14.9251774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:14.9252290Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:14.9252477Z 2025-08-26T20:29:14.9252569Z cudagraph partition due to non gpu ops 2025-08-26T20:29:14.9252809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9253172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9253508Z return mod(**inputs) 2025-08-26T20:29:14.9253904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9254329Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9254736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9255149Z return self.transformer( 2025-08-26T20:29:14.9255546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9255970Z layer_outputs = layer_module( 2025-08-26T20:29:14.9256315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9256686Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9257113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9257529Z sa_output = self.attention( 2025-08-26T20:29:14.9257933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:14.9258407Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:14.9258603Z 2025-08-26T20:29:14.9258707Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9259073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9259405Z return mod(**inputs) 2025-08-26T20:29:14.9259815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9260229Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9260661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9261083Z return self.transformer( 2025-08-26T20:29:14.9261484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9261890Z layer_outputs = layer_module( 2025-08-26T20:29:14.9262242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9262611Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9263035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:14.9263453Z sa_output = self.attention( 2025-08-26T20:29:14.9263851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:14.9264284Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:14.9264430Z 2025-08-26T20:29:14.9264535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9264915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9265245Z return mod(**inputs) 2025-08-26T20:29:14.9265629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9266044Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9266462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9266923Z return self.transformer( 2025-08-26T20:29:14.9267351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9267807Z layer_outputs = layer_module( 2025-08-26T20:29:14.9268192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9268603Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9269056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9269514Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9269979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9270537Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9271069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9271468Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9271885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:14.9272313Z x = self.lin1(input) 2025-08-26T20:29:14.9272429Z 2025-08-26T20:29:14.9272536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9272911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9273250Z return mod(**inputs) 2025-08-26T20:29:14.9273647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9274071Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9274510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9274932Z return self.transformer( 2025-08-26T20:29:14.9275348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9275769Z layer_outputs = layer_module( 2025-08-26T20:29:14.9276122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9276491Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9276920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9277395Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9277875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9278478Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9279078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9279612Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9280100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:14.9280575Z x = self.activation(x) 2025-08-26T20:29:14.9280929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:14.9281287Z return self.act(input) 2025-08-26T20:29:14.9281403Z 2025-08-26T20:29:14.9281520Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9281899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9282256Z return mod(**inputs) 2025-08-26T20:29:14.9282680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-26T20:29:14.9283127Z dlbrt_output = self.distilbert( 2025-08-26T20:29:14.9283560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:14.9284002Z return self.transformer( 2025-08-26T20:29:14.9284410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:14.9284828Z layer_outputs = layer_module( 2025-08-26T20:29:14.9285194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:14.9285558Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:14.9286002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:14.9286479Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:14.9286957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:14.9287531Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:14.9288074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:14.9288500Z return forward_fn(*input_tensors) 2025-08-26T20:29:14.9288945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:14.9289384Z x = self.lin2(x) 2025-08-26T20:29:14.9289492Z 2025-08-26T20:29:14.9289610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9290014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9290366Z return mod(**inputs) 2025-08-26T20:29:14.9290799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 836, in forward 2025-08-26T20:29:14.9291339Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-08-26T20:29:14.9291582Z 2025-08-26T20:29:14.9291689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9292065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9292406Z return mod(**inputs) 2025-08-26T20:29:14.9292812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 839, in forward 2025-08-26T20:29:14.9293357Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-08-26T20:29:14.9293610Z 2025-08-26T20:29:14.9293717Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:14.9294091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:14.9294426Z return mod(**inputs) 2025-08-26T20:29:14.9294830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 843, in forward 2025-08-26T20:29:14.9295408Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-08-26T20:29:14.9295677Z 2025-08-26T20:29:22.4264688Z Compilation time (from dynamo_timed): 11.497413627 2025-08-26T20:29:22.4284545Z pass 2025-08-26T20:29:22.4285123Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:22.4286387Z TIMING: _recursive_pre_grad_passes:0.00534 _recursive_joint_graph_passes:0.26059 _recursive_post_grad_passes:0.05086 async_compile.wait:0.79992 code_gen:7.1881 inductor_compile:8.18794 backend_compile:10.02357 gc:0.00026 entire_frame_compile:11.49741 total_wall_time:11.49741 2025-08-26T20:29:22.4287521Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:6654 | FakeTensor.__torch_dispatch__:2344 | ProxyTorchDispatchMode.__torch_dispatch__:2359 2025-08-26T20:29:22.4288470Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-08-26T20:29:27.7600905Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:29:27.7601904Z from pkg_resources import resource_filename 2025-08-26T20:29:28.3568620Z 2025-08-26T20:29:29.1159162Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:29:29.1159811Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:29:29.1165991Z cpu eval DistilBertForQuestionAnswering 2025-08-26T20:29:29.2443390Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:29.3046929Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:29.3536194Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:34.4341014Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4341366Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4341585Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4341796Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4341999Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4342210Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4342490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4343262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4343655Z return mod(**inputs) 2025-08-26T20:29:34.4344207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4344656Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4345102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4345527Z return self.transformer( 2025-08-26T20:29:34.4345932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4346361Z layer_outputs = layer_module( 2025-08-26T20:29:34.4346733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4347122Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4347563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4347980Z sa_output = self.attention( 2025-08-26T20:29:34.4348390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:34.4348909Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:34.4349097Z 2025-08-26T20:29:34.4349215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4349587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4349914Z return mod(**inputs) 2025-08-26T20:29:34.4350322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4350805Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4351239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4351661Z return self.transformer( 2025-08-26T20:29:34.4352065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4352489Z layer_outputs = layer_module( 2025-08-26T20:29:34.4352848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4353229Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4353655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4354086Z sa_output = self.attention( 2025-08-26T20:29:34.4354517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:34.4355021Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4355217Z 2025-08-26T20:29:34.4355341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4355739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4356102Z return mod(**inputs) 2025-08-26T20:29:34.4356544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4357022Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4357497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4357958Z return self.transformer( 2025-08-26T20:29:34.4358403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4358903Z layer_outputs = layer_module( 2025-08-26T20:29:34.4359494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4359932Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4360404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4360931Z sa_output = self.attention( 2025-08-26T20:29:34.4361340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:34.4361837Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4362032Z 2025-08-26T20:29:34.4362119Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4362383Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4362917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4363267Z return mod(**inputs) 2025-08-26T20:29:34.4363698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4364140Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4364618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4365061Z return self.transformer( 2025-08-26T20:29:34.4365486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4365947Z layer_outputs = layer_module( 2025-08-26T20:29:34.4366320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4366731Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4367184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4367634Z sa_output = self.attention( 2025-08-26T20:29:34.4368064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:34.4368568Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:34.4368764Z 2025-08-26T20:29:34.4368870Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4369237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4369604Z return mod(**inputs) 2025-08-26T20:29:34.4369995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4370431Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4370860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4371281Z return self.transformer( 2025-08-26T20:29:34.4371675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4372086Z layer_outputs = layer_module( 2025-08-26T20:29:34.4372435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4372799Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4373219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4373625Z sa_output = self.attention( 2025-08-26T20:29:34.4374032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:34.4374474Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:34.4374615Z 2025-08-26T20:29:34.4374728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4375135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4375465Z return mod(**inputs) 2025-08-26T20:29:34.4375868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4376302Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4376727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4377143Z return self.transformer( 2025-08-26T20:29:34.4377550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4377996Z layer_outputs = layer_module( 2025-08-26T20:29:34.4378371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4378770Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4379212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4379737Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4380195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4380744Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4381270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4381785Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4382229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:34.4382672Z x = self.lin1(input) 2025-08-26T20:29:34.4382787Z 2025-08-26T20:29:34.4382907Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4383294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4383624Z return mod(**inputs) 2025-08-26T20:29:34.4384019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4384446Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4384871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4385280Z return self.transformer( 2025-08-26T20:29:34.4385686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4386101Z layer_outputs = layer_module( 2025-08-26T20:29:34.4386454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4386815Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4387237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4387704Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4388188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4388732Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4389266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4389668Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4390105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:34.4390529Z x = self.activation(x) 2025-08-26T20:29:34.4390858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:34.4391198Z return self.act(input) 2025-08-26T20:29:34.4391317Z 2025-08-26T20:29:34.4391421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4391786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4392112Z return mod(**inputs) 2025-08-26T20:29:34.4392502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4392931Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4393370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4393807Z return self.transformer( 2025-08-26T20:29:34.4394252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4394681Z layer_outputs = layer_module( 2025-08-26T20:29:34.4395052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4395422Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4395844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4396667Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4397146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4397724Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4398295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4398738Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4399203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:34.4399880Z x = self.lin2(x) 2025-08-26T20:29:34.4400004Z 2025-08-26T20:29:34.4400120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4400528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4400892Z return mod(**inputs) 2025-08-26T20:29:34.4401310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4401776Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4402230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4402677Z return self.transformer( 2025-08-26T20:29:34.4403103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4403538Z layer_outputs = layer_module( 2025-08-26T20:29:34.4403913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4404305Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4404807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4405253Z sa_output = self.attention( 2025-08-26T20:29:34.4405706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:34.4406208Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:34.4406409Z 2025-08-26T20:29:34.4406523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4406924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4407255Z return mod(**inputs) 2025-08-26T20:29:34.4407662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4408122Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4408566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4409004Z return self.transformer( 2025-08-26T20:29:34.4409420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4409859Z layer_outputs = layer_module( 2025-08-26T20:29:34.4410258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4410624Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4411044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4411449Z sa_output = self.attention( 2025-08-26T20:29:34.4411850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:34.4412337Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4412517Z 2025-08-26T20:29:34.4412628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4412992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4413312Z return mod(**inputs) 2025-08-26T20:29:34.4413709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4414135Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4414557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4414964Z return self.transformer( 2025-08-26T20:29:34.4415364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4415781Z layer_outputs = layer_module( 2025-08-26T20:29:34.4416132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4416500Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4416916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4417329Z sa_output = self.attention( 2025-08-26T20:29:34.4417728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:34.4418208Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4418399Z 2025-08-26T20:29:34.4418494Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4418744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4419129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4419501Z return mod(**inputs) 2025-08-26T20:29:34.4419903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4420345Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4420768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4421183Z return self.transformer( 2025-08-26T20:29:34.4421583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4422049Z layer_outputs = layer_module( 2025-08-26T20:29:34.4422415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4422805Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4423258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4423667Z sa_output = self.attention( 2025-08-26T20:29:34.4424105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:34.4424578Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:34.4424789Z 2025-08-26T20:29:34.4424894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4425259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4425590Z return mod(**inputs) 2025-08-26T20:29:34.4425983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4426454Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4426879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4427293Z return self.transformer( 2025-08-26T20:29:34.4427692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4428097Z layer_outputs = layer_module( 2025-08-26T20:29:34.4428449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4428822Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4429233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4429636Z sa_output = self.attention( 2025-08-26T20:29:34.4430016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:34.4430432Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:34.4430577Z 2025-08-26T20:29:34.4430680Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4431035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4431348Z return mod(**inputs) 2025-08-26T20:29:34.4431746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4432171Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4432593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4433007Z return self.transformer( 2025-08-26T20:29:34.4433402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4433819Z layer_outputs = layer_module( 2025-08-26T20:29:34.4434192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4434565Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4435007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4435457Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4435917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4436458Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4436995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4440484Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4440949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:34.4441408Z x = self.lin1(input) 2025-08-26T20:29:34.4441536Z 2025-08-26T20:29:34.4441658Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4442053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4442397Z return mod(**inputs) 2025-08-26T20:29:34.4442815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4443272Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4443722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4444195Z return self.transformer( 2025-08-26T20:29:34.4444646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4445155Z layer_outputs = layer_module( 2025-08-26T20:29:34.4445544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4445945Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4446387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4446873Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4447356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4447919Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4448457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4448846Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4449260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:34.4449672Z x = self.activation(x) 2025-08-26T20:29:34.4449995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:34.4450334Z return self.act(input) 2025-08-26T20:29:34.4450607Z 2025-08-26T20:29:34.4450716Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4451081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4451402Z return mod(**inputs) 2025-08-26T20:29:34.4451785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4452199Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4452641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4453073Z return self.transformer( 2025-08-26T20:29:34.4453477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4453898Z layer_outputs = layer_module( 2025-08-26T20:29:34.4454234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4454592Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4455001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4455442Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4455964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4456499Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4457014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4457409Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4457821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:34.4458224Z x = self.lin2(x) 2025-08-26T20:29:34.4458324Z 2025-08-26T20:29:34.4458429Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4458792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4459144Z return mod(**inputs) 2025-08-26T20:29:34.4459550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4459960Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4460374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4460775Z return self.transformer( 2025-08-26T20:29:34.4461165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4461566Z layer_outputs = layer_module( 2025-08-26T20:29:34.4461907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4462264Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4462682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4463139Z sa_output = self.attention( 2025-08-26T20:29:34.4463556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:34.4464028Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:34.4464220Z 2025-08-26T20:29:34.4464327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4464700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4465027Z return mod(**inputs) 2025-08-26T20:29:34.4465408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4465827Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4466253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4466691Z return self.transformer( 2025-08-26T20:29:34.4467092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4467519Z layer_outputs = layer_module( 2025-08-26T20:29:34.4467875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4468247Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4468675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4469093Z sa_output = self.attention( 2025-08-26T20:29:34.4469497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:34.4469967Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4470187Z 2025-08-26T20:29:34.4470294Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4470657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4470981Z return mod(**inputs) 2025-08-26T20:29:34.4471379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4471807Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4472230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4472647Z return self.transformer( 2025-08-26T20:29:34.4473071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4473515Z layer_outputs = layer_module( 2025-08-26T20:29:34.4473889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4474281Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4474722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4475156Z sa_output = self.attention( 2025-08-26T20:29:34.4475564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:34.4476029Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4476211Z 2025-08-26T20:29:34.4476301Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4476539Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4476913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4477263Z return mod(**inputs) 2025-08-26T20:29:34.4477681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4478130Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4478567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4479012Z return self.transformer( 2025-08-26T20:29:34.4479523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4479992Z layer_outputs = layer_module( 2025-08-26T20:29:34.4480376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4480779Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4481220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4481661Z sa_output = self.attention( 2025-08-26T20:29:34.4482071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:34.4482568Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:34.4482775Z 2025-08-26T20:29:34.4482887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4483276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4483623Z return mod(**inputs) 2025-08-26T20:29:34.4484047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4484514Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4484937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4485382Z return self.transformer( 2025-08-26T20:29:34.4485895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4486311Z layer_outputs = layer_module( 2025-08-26T20:29:34.4486654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4487022Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4487437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4487852Z sa_output = self.attention( 2025-08-26T20:29:34.4488246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:34.4488699Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:34.4488845Z 2025-08-26T20:29:34.4488960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4489305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4489616Z return mod(**inputs) 2025-08-26T20:29:34.4489986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4490401Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4490831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4491236Z return self.transformer( 2025-08-26T20:29:34.4491622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4492032Z layer_outputs = layer_module( 2025-08-26T20:29:34.4492369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4492723Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4493131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4493568Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4494012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4494561Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4495088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4495489Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4495922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:34.4496555Z x = self.lin1(input) 2025-08-26T20:29:34.4496677Z 2025-08-26T20:29:34.4496784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4497224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4497559Z return mod(**inputs) 2025-08-26T20:29:34.4497972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4498452Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4498902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4499341Z return self.transformer( 2025-08-26T20:29:34.4499767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4500257Z layer_outputs = layer_module( 2025-08-26T20:29:34.4500629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4501001Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4501446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4501921Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4502401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4502980Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4503532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4504007Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4504472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:34.4504925Z x = self.activation(x) 2025-08-26T20:29:34.4505296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:34.4505690Z return self.act(input) 2025-08-26T20:29:34.4505810Z 2025-08-26T20:29:34.4505933Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4506322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4506682Z return mod(**inputs) 2025-08-26T20:29:34.4507105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4507571Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4508029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4508468Z return self.transformer( 2025-08-26T20:29:34.4508900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4509346Z layer_outputs = layer_module( 2025-08-26T20:29:34.4509725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4510114Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4510565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4511051Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4511541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4512157Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4512741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4513183Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4513643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:34.4514091Z x = self.lin2(x) 2025-08-26T20:29:34.4514200Z 2025-08-26T20:29:34.4514324Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4514715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4515076Z return mod(**inputs) 2025-08-26T20:29:34.4515507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4515995Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4516452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4516901Z return self.transformer( 2025-08-26T20:29:34.4517335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4517785Z layer_outputs = layer_module( 2025-08-26T20:29:34.4518165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4518557Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4519014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4519561Z sa_output = self.attention( 2025-08-26T20:29:34.4520019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:34.4520537Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:34.4520741Z 2025-08-26T20:29:34.4520860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4521269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4521636Z return mod(**inputs) 2025-08-26T20:29:34.4522076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4522549Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4523005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4523466Z return self.transformer( 2025-08-26T20:29:34.4523909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4524369Z layer_outputs = layer_module( 2025-08-26T20:29:34.4524751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4525160Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4525621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4526076Z sa_output = self.attention( 2025-08-26T20:29:34.4526525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:34.4527031Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4527236Z 2025-08-26T20:29:34.4527346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4527743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4528076Z return mod(**inputs) 2025-08-26T20:29:34.4528489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4528912Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4529335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4529750Z return self.transformer( 2025-08-26T20:29:34.4530148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4530565Z layer_outputs = layer_module( 2025-08-26T20:29:34.4530915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4531308Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4531730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4532146Z sa_output = self.attention( 2025-08-26T20:29:34.4532544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:34.4533010Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4533201Z 2025-08-26T20:29:34.4533287Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4533534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4533898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4534241Z return mod(**inputs) 2025-08-26T20:29:34.4534640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4535073Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4535508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4535916Z return self.transformer( 2025-08-26T20:29:34.4536315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4536734Z layer_outputs = layer_module( 2025-08-26T20:29:34.4537087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4537459Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4537877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4538289Z sa_output = self.attention( 2025-08-26T20:29:34.4538694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:34.4539172Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:34.4539363Z 2025-08-26T20:29:34.4539477Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4539836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4540169Z return mod(**inputs) 2025-08-26T20:29:34.4540574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4541057Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4541469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4541869Z return self.transformer( 2025-08-26T20:29:34.4542281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4542689Z layer_outputs = layer_module( 2025-08-26T20:29:34.4543051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4543401Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4543805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4544210Z sa_output = self.attention( 2025-08-26T20:29:34.4544603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:34.4545019Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:34.4545156Z 2025-08-26T20:29:34.4545259Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4545637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4545962Z return mod(**inputs) 2025-08-26T20:29:34.4546351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4546769Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4547178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4547591Z return self.transformer( 2025-08-26T20:29:34.4547992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4548411Z layer_outputs = layer_module( 2025-08-26T20:29:34.4548757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4549136Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4549545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4549995Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4550440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4550992Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4551509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4551917Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4552345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:34.4552754Z x = self.lin1(input) 2025-08-26T20:29:34.4552860Z 2025-08-26T20:29:34.4552963Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4553322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4553641Z return mod(**inputs) 2025-08-26T20:29:34.4554025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4554451Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4554866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4555280Z return self.transformer( 2025-08-26T20:29:34.4555679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4556103Z layer_outputs = layer_module( 2025-08-26T20:29:34.4556474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4556847Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4557308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4557790Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4558272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4558834Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4559470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4559920Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4560411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:34.4560874Z x = self.activation(x) 2025-08-26T20:29:34.4561576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:34.4562002Z return self.act(input) 2025-08-26T20:29:34.4562180Z 2025-08-26T20:29:34.4562313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4562764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4563187Z return mod(**inputs) 2025-08-26T20:29:34.4563624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4564136Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4564667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4565158Z return self.transformer( 2025-08-26T20:29:34.4565647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4566109Z layer_outputs = layer_module( 2025-08-26T20:29:34.4566538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4567027Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4567522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4568037Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4568585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4569244Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4569865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4570397Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4570910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:34.4571429Z x = self.lin2(x) 2025-08-26T20:29:34.4571570Z 2025-08-26T20:29:34.4571707Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4572144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4583837Z return mod(**inputs) 2025-08-26T20:29:34.4584484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4584993Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4585543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4586019Z return self.transformer( 2025-08-26T20:29:34.4586500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4586958Z layer_outputs = layer_module( 2025-08-26T20:29:34.4587351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4587746Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4588206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4588656Z sa_output = self.attention( 2025-08-26T20:29:34.4589105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:34.4589642Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:34.4589853Z 2025-08-26T20:29:34.4589980Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4590383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4590759Z return mod(**inputs) 2025-08-26T20:29:34.4591199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4591665Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4592123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4592579Z return self.transformer( 2025-08-26T20:29:34.4593051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4593514Z layer_outputs = layer_module( 2025-08-26T20:29:34.4593893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4594297Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4594748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4595207Z sa_output = self.attention( 2025-08-26T20:29:34.4595637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:34.4596152Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4596542Z 2025-08-26T20:29:34.4596661Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4597081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4597464Z return mod(**inputs) 2025-08-26T20:29:34.4597914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4598404Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4598873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4599418Z return self.transformer( 2025-08-26T20:29:34.4599871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4600333Z layer_outputs = layer_module( 2025-08-26T20:29:34.4600729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4601151Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4601677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4602118Z sa_output = self.attention( 2025-08-26T20:29:34.4602581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:34.4603100Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4603308Z 2025-08-26T20:29:34.4603402Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4603675Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4604075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4604440Z return mod(**inputs) 2025-08-26T20:29:34.4604877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4605386Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4605861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4606312Z return self.transformer( 2025-08-26T20:29:34.4606753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4607214Z layer_outputs = layer_module( 2025-08-26T20:29:34.4607603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4608016Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4608473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4608961Z sa_output = self.attention( 2025-08-26T20:29:34.4609405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:34.4609928Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:34.4610135Z 2025-08-26T20:29:34.4610259Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4610649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4611027Z return mod(**inputs) 2025-08-26T20:29:34.4611462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4611926Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4612380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4612820Z return self.transformer( 2025-08-26T20:29:34.4613245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4613685Z layer_outputs = layer_module( 2025-08-26T20:29:34.4614055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4614437Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4614882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4615321Z sa_output = self.attention( 2025-08-26T20:29:34.4615749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:34.4616193Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:34.4616348Z 2025-08-26T20:29:34.4616462Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4616846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4617217Z return mod(**inputs) 2025-08-26T20:29:34.4617641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4618120Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4618568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4619009Z return self.transformer( 2025-08-26T20:29:34.4619443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4619890Z layer_outputs = layer_module( 2025-08-26T20:29:34.4620261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4620659Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4621128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4621611Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4622097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4622678Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4623208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4623612Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4624031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:34.4624465Z x = self.lin1(input) 2025-08-26T20:29:34.4624574Z 2025-08-26T20:29:34.4624681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4625050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4625383Z return mod(**inputs) 2025-08-26T20:29:34.4625778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4626205Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4626628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4627039Z return self.transformer( 2025-08-26T20:29:34.4627441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4627858Z layer_outputs = layer_module( 2025-08-26T20:29:34.4628207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4628579Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4629012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4629494Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4629967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4630546Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4631097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4631505Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4631925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:34.4632360Z x = self.activation(x) 2025-08-26T20:29:34.4632699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:34.4633067Z return self.act(input) 2025-08-26T20:29:34.4633181Z 2025-08-26T20:29:34.4633295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4633663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4633985Z return mod(**inputs) 2025-08-26T20:29:34.4634383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4634810Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4635240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4635713Z return self.transformer( 2025-08-26T20:29:34.4636130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4636587Z layer_outputs = layer_module( 2025-08-26T20:29:34.4636967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4637365Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4637807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4638303Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4638779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4639465Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4640037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4640457Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4640902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:34.4641323Z x = self.lin2(x) 2025-08-26T20:29:34.4641424Z 2025-08-26T20:29:34.4641535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4641896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4642218Z return mod(**inputs) 2025-08-26T20:29:34.4642607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4643026Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4643443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4643852Z return self.transformer( 2025-08-26T20:29:34.4644242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4644646Z layer_outputs = layer_module( 2025-08-26T20:29:34.4644986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4645347Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4645765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4646213Z sa_output = self.attention( 2025-08-26T20:29:34.4646649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-26T20:29:34.4647148Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-26T20:29:34.4647333Z 2025-08-26T20:29:34.4647446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4647821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4648154Z return mod(**inputs) 2025-08-26T20:29:34.4648554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4648983Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4649413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4649827Z return self.transformer( 2025-08-26T20:29:34.4650234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4650674Z layer_outputs = layer_module( 2025-08-26T20:29:34.4651044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4651398Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4651814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4652226Z sa_output = self.attention( 2025-08-26T20:29:34.4652639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-26T20:29:34.4653108Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4653285Z 2025-08-26T20:29:34.4653389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4653755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4654108Z return mod(**inputs) 2025-08-26T20:29:34.4654503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4654933Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4655350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4655768Z return self.transformer( 2025-08-26T20:29:34.4656171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4656589Z layer_outputs = layer_module( 2025-08-26T20:29:34.4656936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4657325Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4657773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4658212Z sa_output = self.attention( 2025-08-26T20:29:34.4658631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-26T20:29:34.4659091Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-26T20:29:34.4659278Z 2025-08-26T20:29:34.4659362Z cudagraph partition due to non gpu ops 2025-08-26T20:29:34.4659605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4659974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4660305Z return mod(**inputs) 2025-08-26T20:29:34.4660696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4661124Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4661566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4661983Z return self.transformer( 2025-08-26T20:29:34.4662400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4662821Z layer_outputs = layer_module( 2025-08-26T20:29:34.4663175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4663543Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4663969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4664384Z sa_output = self.attention( 2025-08-26T20:29:34.4664798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-26T20:29:34.4665307Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:34.4665495Z 2025-08-26T20:29:34.4665607Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4665988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4666334Z return mod(**inputs) 2025-08-26T20:29:34.4666757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4667201Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4667619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4668028Z return self.transformer( 2025-08-26T20:29:34.4668430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4668868Z layer_outputs = layer_module( 2025-08-26T20:29:34.4669229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4669605Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4670029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-26T20:29:34.4670448Z sa_output = self.attention( 2025-08-26T20:29:34.4670857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-26T20:29:34.4671294Z attn_output = self.out_lin(attn_output) 2025-08-26T20:29:34.4671440Z 2025-08-26T20:29:34.4671550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4671926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4672267Z return mod(**inputs) 2025-08-26T20:29:34.4672671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4673107Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4673528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4673957Z return self.transformer( 2025-08-26T20:29:34.4674367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4674795Z layer_outputs = layer_module( 2025-08-26T20:29:34.4675170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4675566Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4676027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4676559Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4677063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4677672Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4678281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4678722Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4679182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-26T20:29:34.4679725Z x = self.lin1(input) 2025-08-26T20:29:34.4679851Z 2025-08-26T20:29:34.4679967Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4680415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4680779Z return mod(**inputs) 2025-08-26T20:29:34.4681215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4681681Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4682118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4682573Z return self.transformer( 2025-08-26T20:29:34.4683007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4683463Z layer_outputs = layer_module( 2025-08-26T20:29:34.4683837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4684249Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4684707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4685192Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4685678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4686263Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4686810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4687238Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4687686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-26T20:29:34.4688130Z x = self.activation(x) 2025-08-26T20:29:34.4688478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:29:34.4688845Z return self.act(input) 2025-08-26T20:29:34.4688969Z 2025-08-26T20:29:34.4689086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4689483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4689832Z return mod(**inputs) 2025-08-26T20:29:34.4690244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-26T20:29:34.4690711Z distilbert_output = self.distilbert( 2025-08-26T20:29:34.4691160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-26T20:29:34.4691603Z return self.transformer( 2025-08-26T20:29:34.4692045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-26T20:29:34.4692486Z layer_outputs = layer_module( 2025-08-26T20:29:34.4692874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:34.4693267Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:34.4693721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-26T20:29:34.4694212Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-26T20:29:34.4694710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-26T20:29:34.4695305Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-26T20:29:34.4695886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:29:34.4696471Z return forward_fn(*input_tensors) 2025-08-26T20:29:34.4696926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-26T20:29:34.4697366Z x = self.lin2(x) 2025-08-26T20:29:34.4697484Z 2025-08-26T20:29:34.4697597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4697994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4698348Z return mod(**inputs) 2025-08-26T20:29:34.4698762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1043, in forward 2025-08-26T20:29:34.4699264Z logits = self.qa_outputs(hidden_states) # (bs, max_query_len, 2) 2025-08-26T20:29:34.4699510Z 2025-08-26T20:29:34.4699624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4700019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4700366Z return mod(**inputs) 2025-08-26T20:29:34.4700782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1061, in forward 2025-08-26T20:29:34.4701233Z start_loss = loss_fct(start_logits, start_positions) 2025-08-26T20:29:34.4701391Z 2025-08-26T20:29:34.4701489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:34.4701844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:29:34.4702157Z return mod(**inputs) 2025-08-26T20:29:34.4702548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1062, in forward 2025-08-26T20:29:34.4702981Z end_loss = loss_fct(end_logits, end_positions) 2025-08-26T20:29:34.4703132Z 2025-08-26T20:29:41.5584915Z Compilation time (from dynamo_timed): 11.040503176 2025-08-26T20:29:41.5585228Z pass 2025-08-26T20:29:41.5585556Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:41.5586414Z TIMING: _recursive_pre_grad_passes:0.00553 _recursive_joint_graph_passes:0.25663 _recursive_post_grad_passes:0.05762 async_compile.wait:0.70616 code_gen:6.78745 inductor_compile:7.78501 backend_compile:9.59702 gc:0.00048 entire_frame_compile:11.0405 total_wall_time:11.0405 2025-08-26T20:29:41.5587463Z STATS: call_* op count: 161 | FakeTensorMode.__torch_dispatch__:6699 | FakeTensor.__torch_dispatch__:2383 | ProxyTorchDispatchMode.__torch_dispatch__:2400 2025-08-26T20:29:41.5588002Z Dynamo produced 1 graphs covering 161 ops with 0 graph breaks (0 unique) 2025-08-26T20:29:46.9497271Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:29:46.9498316Z from pkg_resources import resource_filename 2025-08-26T20:29:47.5460458Z 2025-08-26T20:29:49.6717487Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-08-26T20:29:49.6718264Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-08-26T20:29:49.7102184Z 2025-08-26T20:29:49.7103120Z loading model: 0it [00:02, ?it/s] 2025-08-26T20:29:49.7114903Z cpu eval DistillGPT2 2025-08-26T20:29:50.1456841Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:50.3411909Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:50.5360450Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:29:57.1539797Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1540127Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1540511Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1540727Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1540941Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1541154Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1541399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1541885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1542310Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1542729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1543497Z outputs = block( 2025-08-26T20:29:57.1543861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1544285Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1544726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1545184Z return func(*args, **kwargs) 2025-08-26T20:29:57.1545590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1546049Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1546485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1546913Z return func(*args, **kwargs) 2025-08-26T20:29:57.1547328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:29:57.1547863Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:29:57.1548370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1548818Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1549013Z 2025-08-26T20:29:57.1549109Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1549340Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1549560Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1549784Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1550044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1550497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1550939Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1551467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1551889Z outputs = block( 2025-08-26T20:29:57.1552314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1552725Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1553146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1553593Z return func(*args, **kwargs) 2025-08-26T20:29:57.1554021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1554479Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1554925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1555416Z return func(*args, **kwargs) 2025-08-26T20:29:57.1555827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1556266Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1556745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:29:57.1557273Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:57.1557486Z 2025-08-26T20:29:57.1557605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1558087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1558541Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1558981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1559691Z outputs = block( 2025-08-26T20:29:57.1560070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1560484Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1560920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1561323Z return func(*args, **kwargs) 2025-08-26T20:29:57.1561732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1562311Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1562740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1563142Z return func(*args, **kwargs) 2025-08-26T20:29:57.1563549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1563999Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1564483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:29:57.1564980Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:29:57.1565159Z 2025-08-26T20:29:57.1565279Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1565725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1566168Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1566593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1567002Z outputs = block( 2025-08-26T20:29:57.1567355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1567772Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1568186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1568625Z return func(*args, **kwargs) 2025-08-26T20:29:57.1569074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1569490Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1569910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1570307Z return func(*args, **kwargs) 2025-08-26T20:29:57.1570705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:29:57.1571125Z attn_output = self.c_proj(attn_output) 2025-08-26T20:29:57.1571530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1571966Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1572160Z 2025-08-26T20:29:57.1572275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1572718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1573140Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1573547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1573946Z outputs = block( 2025-08-26T20:29:57.1574290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1574701Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1575103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1575503Z return func(*args, **kwargs) 2025-08-26T20:29:57.1575898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1576339Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1576779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:29:57.1577188Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:29:57.1577577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1578015Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1578206Z 2025-08-26T20:29:57.1578331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1578792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1579224Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1579703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1580123Z outputs = block( 2025-08-26T20:29:57.1580476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1580875Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1581286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1581699Z return func(*args, **kwargs) 2025-08-26T20:29:57.1582106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1582570Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1583038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:29:57.1583468Z hidden_states = self.act(hidden_states) 2025-08-26T20:29:57.1583900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:29:57.1584410Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:29:57.1584664Z 2025-08-26T20:29:57.1584790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1585238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1585678Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1586106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1586539Z outputs = block( 2025-08-26T20:29:57.1586901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1587304Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1587731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1588156Z return func(*args, **kwargs) 2025-08-26T20:29:57.1588564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1589032Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1589486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:29:57.1589947Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:29:57.1590352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1590800Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1590990Z 2025-08-26T20:29:57.1591108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1591566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1592003Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1592431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1592844Z outputs = block( 2025-08-26T20:29:57.1593194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1593594Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1594011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1594422Z return func(*args, **kwargs) 2025-08-26T20:29:57.1594825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1595259Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1595684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1596090Z return func(*args, **kwargs) 2025-08-26T20:29:57.1596778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:29:57.1597322Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:29:57.1597839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1598370Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1598563Z 2025-08-26T20:29:57.1598663Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1598901Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1599154Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1599445Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1599709Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1600177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1600608Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1601051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1601467Z outputs = block( 2025-08-26T20:29:57.1601835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1603015Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1603436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1603862Z return func(*args, **kwargs) 2025-08-26T20:29:57.1604282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1604757Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1605170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1605581Z return func(*args, **kwargs) 2025-08-26T20:29:57.1605988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1606473Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1606958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:29:57.1607467Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:57.1607699Z 2025-08-26T20:29:57.1607814Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1608257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1608687Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1609109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1609499Z outputs = block( 2025-08-26T20:29:57.1609851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1610245Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1610650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1611049Z return func(*args, **kwargs) 2025-08-26T20:29:57.1611433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1611857Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1612275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1612672Z return func(*args, **kwargs) 2025-08-26T20:29:57.1613063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1613504Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1613982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:29:57.1614499Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:29:57.1614673Z 2025-08-26T20:29:57.1614792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1615244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1615673Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1616083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1616479Z outputs = block( 2025-08-26T20:29:57.1616826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1617219Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1617624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1618052Z return func(*args, **kwargs) 2025-08-26T20:29:57.1618451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1618874Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1619296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1619693Z return func(*args, **kwargs) 2025-08-26T20:29:57.1620088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:29:57.1620510Z attn_output = self.c_proj(attn_output) 2025-08-26T20:29:57.1620889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1621318Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1621525Z 2025-08-26T20:29:57.1621642Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1622091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1622516Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1622926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1623328Z outputs = block( 2025-08-26T20:29:57.1623685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1624062Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1624446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1624832Z return func(*args, **kwargs) 2025-08-26T20:29:57.1625211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1625643Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1626063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:29:57.1626459Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:29:57.1626834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1627245Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1627428Z 2025-08-26T20:29:57.1627547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1627987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1628407Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1628820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1629238Z outputs = block( 2025-08-26T20:29:57.1629583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1629990Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1630396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1630796Z return func(*args, **kwargs) 2025-08-26T20:29:57.1631194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1631667Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1632074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:29:57.1632485Z hidden_states = self.act(hidden_states) 2025-08-26T20:29:57.1632884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:29:57.1633371Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:29:57.1633622Z 2025-08-26T20:29:57.1633744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1634184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1634610Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1635024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1635423Z outputs = block( 2025-08-26T20:29:57.1635765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1636167Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1636574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1636976Z return func(*args, **kwargs) 2025-08-26T20:29:57.1637370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1637801Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1638235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:29:57.1638657Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:29:57.1639046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1639569Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1639767Z 2025-08-26T20:29:57.1639885Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1640352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1640802Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1641224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1641629Z outputs = block( 2025-08-26T20:29:57.1641977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1642369Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1642777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1643188Z return func(*args, **kwargs) 2025-08-26T20:29:57.1643586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1644015Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1644502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1644910Z return func(*args, **kwargs) 2025-08-26T20:29:57.1645329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:29:57.1645855Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:29:57.1646354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1646785Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1646967Z 2025-08-26T20:29:57.1647064Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1647297Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1647518Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1647769Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1648026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1648482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1648906Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1649335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1649743Z outputs = block( 2025-08-26T20:29:57.1650091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1650487Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1650886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1651319Z return func(*args, **kwargs) 2025-08-26T20:29:57.1651722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1652122Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1652512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1652881Z return func(*args, **kwargs) 2025-08-26T20:29:57.1653241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1653646Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1654085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:29:57.1654553Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:57.1654741Z 2025-08-26T20:29:57.1654844Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1655260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1655651Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1656035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1656397Z outputs = block( 2025-08-26T20:29:57.1656717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1657079Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1657460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1657838Z return func(*args, **kwargs) 2025-08-26T20:29:57.1658202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1658596Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1659015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1659402Z return func(*args, **kwargs) 2025-08-26T20:29:57.1659812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1660253Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1660730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:29:57.1661230Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:29:57.1661413Z 2025-08-26T20:29:57.1661526Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1661946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1662372Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1662778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1663183Z outputs = block( 2025-08-26T20:29:57.1663519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1663911Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1664316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1664718Z return func(*args, **kwargs) 2025-08-26T20:29:57.1665119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1665509Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1665928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1666309Z return func(*args, **kwargs) 2025-08-26T20:29:57.1666685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:29:57.1667079Z attn_output = self.c_proj(attn_output) 2025-08-26T20:29:57.1667435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1667837Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1668017Z 2025-08-26T20:29:57.1668128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1668573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1668992Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1669412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1669823Z outputs = block( 2025-08-26T20:29:57.1670182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1670575Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1670959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1671337Z return func(*args, **kwargs) 2025-08-26T20:29:57.1671705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1672120Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1672530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:29:57.1672922Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:29:57.1673321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1673748Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1673930Z 2025-08-26T20:29:57.1674067Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1674508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1674936Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1675351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1675764Z outputs = block( 2025-08-26T20:29:57.1676111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1676502Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1676949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1677365Z return func(*args, **kwargs) 2025-08-26T20:29:57.1677780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1678242Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1678691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:29:57.1679144Z hidden_states = self.act(hidden_states) 2025-08-26T20:29:57.1679629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:29:57.1680141Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:29:57.1680425Z 2025-08-26T20:29:57.1680554Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1681011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1681451Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1681875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1682278Z outputs = block( 2025-08-26T20:29:57.1682619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1682996Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1683396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1683806Z return func(*args, **kwargs) 2025-08-26T20:29:57.1684202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1684647Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1685093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:29:57.1685517Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:29:57.1685907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1686329Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1686511Z 2025-08-26T20:29:57.1686622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1687088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1687509Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1687924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1688346Z outputs = block( 2025-08-26T20:29:57.1688686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1689098Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1689567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1689977Z return func(*args, **kwargs) 2025-08-26T20:29:57.1690368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-26T20:29:57.1690812Z hidden_states = residual + feed_forward_hidden_states 2025-08-26T20:29:57.1690991Z 2025-08-26T20:29:57.1691100Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1691544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1691994Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1692404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1692807Z outputs = block( 2025-08-26T20:29:57.1693155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1693541Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1693939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1694341Z return func(*args, **kwargs) 2025-08-26T20:29:57.1694738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1695175Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1695612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1696006Z return func(*args, **kwargs) 2025-08-26T20:29:57.1696611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:29:57.1697158Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:29:57.1697665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1698095Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1698281Z 2025-08-26T20:29:57.1698371Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1698606Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1698832Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1699057Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1699301Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1699759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1700190Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1700607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1701010Z outputs = block( 2025-08-26T20:29:57.1701352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1701741Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1702145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1702560Z return func(*args, **kwargs) 2025-08-26T20:29:57.1702946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1703379Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1703856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1704258Z return func(*args, **kwargs) 2025-08-26T20:29:57.1704681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1705110Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1705588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:29:57.1706105Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:57.1706301Z 2025-08-26T20:29:57.1706423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1706872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1707338Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1707754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1708155Z outputs = block( 2025-08-26T20:29:57.1708496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1708854Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1709237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1709611Z return func(*args, **kwargs) 2025-08-26T20:29:57.1710001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1710425Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1710870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1711267Z return func(*args, **kwargs) 2025-08-26T20:29:57.1711661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1712092Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1712566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:29:57.1713053Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:29:57.1713235Z 2025-08-26T20:29:57.1713347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1713795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1714222Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1714646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1715042Z outputs = block( 2025-08-26T20:29:57.1715392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1715779Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1716184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1716587Z return func(*args, **kwargs) 2025-08-26T20:29:57.1716981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1717409Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1717825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1718226Z return func(*args, **kwargs) 2025-08-26T20:29:57.1718645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:29:57.1719076Z attn_output = self.c_proj(attn_output) 2025-08-26T20:29:57.1719570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1720021Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1720218Z 2025-08-26T20:29:57.1720346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1720812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1721242Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1721656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1722070Z outputs = block( 2025-08-26T20:29:57.1722402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1722783Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1723199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1723611Z return func(*args, **kwargs) 2025-08-26T20:29:57.1724014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1724444Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1724888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:29:57.1725302Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:29:57.1725682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1726130Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1726302Z 2025-08-26T20:29:57.1726406Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1726828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1727224Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1727616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1727986Z outputs = block( 2025-08-26T20:29:57.1728315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1728678Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1729058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1729434Z return func(*args, **kwargs) 2025-08-26T20:29:57.1729802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1730255Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1730703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:29:57.1731132Z hidden_states = self.act(hidden_states) 2025-08-26T20:29:57.1731507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:29:57.1732052Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:29:57.1732294Z 2025-08-26T20:29:57.1732400Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1732824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1733254Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1733650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1734040Z outputs = block( 2025-08-26T20:29:57.1734370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1734762Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1735168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1735569Z return func(*args, **kwargs) 2025-08-26T20:29:57.1735958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1736377Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1736811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:29:57.1737213Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:29:57.1737584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1737994Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1738178Z 2025-08-26T20:29:57.1738286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1738717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1739126Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1739515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1739911Z outputs = block( 2025-08-26T20:29:57.1740241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1740610Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1741012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1741414Z return func(*args, **kwargs) 2025-08-26T20:29:57.1741814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1742250Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1742667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1743064Z return func(*args, **kwargs) 2025-08-26T20:29:57.1743460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:29:57.1743985Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:29:57.1744477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1744901Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1745081Z 2025-08-26T20:29:57.1745175Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1745410Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1745636Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1745859Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1746100Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1746545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1746970Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1747386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1747852Z outputs = block( 2025-08-26T20:29:57.1748196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1748607Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1749015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1749416Z return func(*args, **kwargs) 2025-08-26T20:29:57.1749802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1750229Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1750647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1751056Z return func(*args, **kwargs) 2025-08-26T20:29:57.1751454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1751862Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1752320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:29:57.1752809Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:57.1752998Z 2025-08-26T20:29:57.1753118Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1753566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1753984Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1754400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1754838Z outputs = block( 2025-08-26T20:29:57.1755192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1755574Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1755993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1756403Z return func(*args, **kwargs) 2025-08-26T20:29:57.1756814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1757251Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1757672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1758086Z return func(*args, **kwargs) 2025-08-26T20:29:57.1758493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1758945Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1759526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:29:57.1760032Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:29:57.1760223Z 2025-08-26T20:29:57.1760337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1760785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1761203Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1761599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1761990Z outputs = block( 2025-08-26T20:29:57.1762328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1762711Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1763131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1763536Z return func(*args, **kwargs) 2025-08-26T20:29:57.1763962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1764358Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1764744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1765107Z return func(*args, **kwargs) 2025-08-26T20:29:57.1765459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:29:57.1765842Z attn_output = self.c_proj(attn_output) 2025-08-26T20:29:57.1766196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1766616Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1766785Z 2025-08-26T20:29:57.1766890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1767300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1767691Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1768074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1768457Z outputs = block( 2025-08-26T20:29:57.1768777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1769146Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1769549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1769933Z return func(*args, **kwargs) 2025-08-26T20:29:57.1770323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1770760Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1771191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:29:57.1771582Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:29:57.1771947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1772342Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1772522Z 2025-08-26T20:29:57.1772632Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1773053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1773455Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1773845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1774236Z outputs = block( 2025-08-26T20:29:57.1774579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1774972Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1775377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1775788Z return func(*args, **kwargs) 2025-08-26T20:29:57.1776172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1776619Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1777081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:29:57.1777500Z hidden_states = self.act(hidden_states) 2025-08-26T20:29:57.1777886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:29:57.1778375Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:29:57.1778638Z 2025-08-26T20:29:57.1778745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1779174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1779601Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1780008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1780464Z outputs = block( 2025-08-26T20:29:57.1780808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1781176Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1781559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1781941Z return func(*args, **kwargs) 2025-08-26T20:29:57.1782334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1782782Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1783218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:29:57.1783664Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:29:57.1784023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1784453Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1784636Z 2025-08-26T20:29:57.1784742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1785161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1785565Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1785978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1786380Z outputs = block( 2025-08-26T20:29:57.1786730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1787118Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1787518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1787926Z return func(*args, **kwargs) 2025-08-26T20:29:57.1788328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-26T20:29:57.1788774Z hidden_states = residual + feed_forward_hidden_states 2025-08-26T20:29:57.1788950Z 2025-08-26T20:29:57.1789070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1789511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1789937Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1790352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1790750Z outputs = block( 2025-08-26T20:29:57.1791092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1791482Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1791916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1792319Z return func(*args, **kwargs) 2025-08-26T20:29:57.1792725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1793145Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1793562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1793953Z return func(*args, **kwargs) 2025-08-26T20:29:57.1794345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:29:57.1794871Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:29:57.1795388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1795817Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1796009Z 2025-08-26T20:29:57.1796096Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1796560Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1796787Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1797012Z cudagraph partition due to non gpu ops 2025-08-26T20:29:57.1797269Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1797720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1798147Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1798554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1799016Z outputs = block( 2025-08-26T20:29:57.1799443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1799854Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1800265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1800678Z return func(*args, **kwargs) 2025-08-26T20:29:57.1801074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1801514Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1801932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1802330Z return func(*args, **kwargs) 2025-08-26T20:29:57.1802721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1803157Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1803637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:29:57.1804152Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:29:57.1804349Z 2025-08-26T20:29:57.1804461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1804908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1805330Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1805747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1806155Z outputs = block( 2025-08-26T20:29:57.1806495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1806892Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1807337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1807740Z return func(*args, **kwargs) 2025-08-26T20:29:57.1808158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1808590Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1809010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1809414Z return func(*args, **kwargs) 2025-08-26T20:29:57.1809813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:29:57.1810218Z attn_output, attn_weights = attention_interface( 2025-08-26T20:29:57.1810697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:29:57.1811227Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:29:57.1811406Z 2025-08-26T20:29:57.1811531Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1811988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1812424Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1812827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1813213Z outputs = block( 2025-08-26T20:29:57.1813557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1813948Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1814373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1814755Z return func(*args, **kwargs) 2025-08-26T20:29:57.1815131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:29:57.1815541Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:29:57.1815926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1816301Z return func(*args, **kwargs) 2025-08-26T20:29:57.1816673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:29:57.1817066Z attn_output = self.c_proj(attn_output) 2025-08-26T20:29:57.1817426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1817836Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1818020Z 2025-08-26T20:29:57.1818129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1818551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1818976Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1819381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1819758Z outputs = block( 2025-08-26T20:29:57.1820084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1820453Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1820858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1821253Z return func(*args, **kwargs) 2025-08-26T20:29:57.1821691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1822138Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1822589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:29:57.1822982Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:29:57.1823337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1823739Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1823910Z 2025-08-26T20:29:57.1824023Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1824445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1824840Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1825248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1825623Z outputs = block( 2025-08-26T20:29:57.1825953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1826318Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1826696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1827070Z return func(*args, **kwargs) 2025-08-26T20:29:57.1827442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1827873Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1828285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:29:57.1828721Z hidden_states = self.act(hidden_states) 2025-08-26T20:29:57.1829103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:29:57.1829606Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:29:57.1829854Z 2025-08-26T20:29:57.1829972Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1830420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-26T20:29:57.1830807Z transformer_outputs = self.transformer( 2025-08-26T20:29:57.1831198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:29:57.1831576Z outputs = block( 2025-08-26T20:29:57.1831903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:29:57.1832265Z return super().__call__(*args, **kwargs) 2025-08-26T20:29:57.1832653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:29:57.1833035Z return func(*args, **kwargs) 2025-08-26T20:29:57.1833409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:29:57.1833831Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:29:57.1834237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:29:57.1834668Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:29:57.1835054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:29:57.1835490Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:29:57.1835675Z 2025-08-26T20:29:57.1835795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:29:57.1836272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1207, in forward 2025-08-26T20:29:57.1836747Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-08-26T20:29:57.1836934Z 2025-08-26T20:30:05.1147806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:05.1149273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-26T20:30:05.1149854Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-26T20:30:05.1150417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-26T20:30:05.1150986Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-26T20:30:05.1151655Z 2025-08-26T20:30:06.2877860Z Compilation time (from dynamo_timed): 14.309823887 2025-08-26T20:30:06.3025244Z pass 2025-08-26T20:30:06.3031751Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:06.3032652Z TIMING: gc:0.00458 entire_frame_compile:14.30982 _recursive_pre_grad_passes:0.00761 _recursive_joint_graph_passes:0.22458 _recursive_post_grad_passes:0.05635 async_compile.wait:1.43694 code_gen:8.7876 inductor_compile:9.51052 backend_compile:11.23864 total_wall_time:14.30982 2025-08-26T20:30:06.3033742Z STATS: call_* op count: 299 | FakeTensorMode.__torch_dispatch__:7239 | FakeTensor.__torch_dispatch__:2276 | ProxyTorchDispatchMode.__torch_dispatch__:2190 2025-08-26T20:30:06.3034288Z Dynamo produced 2 graphs covering 299 ops with 2 graph breaks (1 unique) 2025-08-26T20:30:11.8290611Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:30:11.8292358Z from pkg_resources import resource_filename 2025-08-26T20:30:12.4387133Z 2025-08-26T20:30:12.4398232Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-26T20:30:12.4398905Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-26T20:30:12.8579462Z 2025-08-26T20:30:12.8580068Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:30:12.8596170Z cpu eval ElectraForCausalLM 2025-08-26T20:30:13.0162717Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:13.1039490Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:13.1895485Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:21.7557913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7558491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7558885Z return mod(**inputs) 2025-08-26T20:30:21.7559596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7560071Z outputs = self.electra( 2025-08-26T20:30:21.7560518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-08-26T20:30:21.7561020Z hidden_states = self.embeddings_project(hidden_states) 2025-08-26T20:30:21.7561219Z 2025-08-26T20:30:21.7561351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7561762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7562515Z return mod(**inputs) 2025-08-26T20:30:21.7562966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7563477Z outputs = self.electra( 2025-08-26T20:30:21.7563892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7564318Z hidden_states = self.encoder( 2025-08-26T20:30:21.7564754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7565187Z layer_outputs = layer_module( 2025-08-26T20:30:21.7565607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7566015Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7566478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7566888Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7567274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7567685Z return func(*args, **kwargs) 2025-08-26T20:30:21.7568091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7568536Z self_outputs = self.self( 2025-08-26T20:30:21.7568945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7569357Z return func(*args, **kwargs) 2025-08-26T20:30:21.7569765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7570242Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7570396Z 2025-08-26T20:30:21.7570513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7570910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7571332Z return mod(**inputs) 2025-08-26T20:30:21.7571710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7572106Z outputs = self.electra( 2025-08-26T20:30:21.7572482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7572918Z hidden_states = self.encoder( 2025-08-26T20:30:21.7573339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7573782Z layer_outputs = layer_module( 2025-08-26T20:30:21.7574178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7574569Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7575014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7575438Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7575822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7576204Z return func(*args, **kwargs) 2025-08-26T20:30:21.7576603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7577026Z self_outputs = self.self( 2025-08-26T20:30:21.7577413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7577824Z return func(*args, **kwargs) 2025-08-26T20:30:21.7578257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7578692Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7578856Z 2025-08-26T20:30:21.7578978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7579360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7579750Z return mod(**inputs) 2025-08-26T20:30:21.7580152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7580571Z outputs = self.electra( 2025-08-26T20:30:21.7580966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7581379Z hidden_states = self.encoder( 2025-08-26T20:30:21.7581822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7582241Z layer_outputs = layer_module( 2025-08-26T20:30:21.7582613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7582995Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7583432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7584014Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7584438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7584837Z return func(*args, **kwargs) 2025-08-26T20:30:21.7585238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7585682Z self_outputs = self.self( 2025-08-26T20:30:21.7586064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7586463Z return func(*args, **kwargs) 2025-08-26T20:30:21.7586874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7587294Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7587448Z 2025-08-26T20:30:21.7587536Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7587767Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7588022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7588411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7588782Z return mod(**inputs) 2025-08-26T20:30:21.7589196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7589636Z outputs = self.electra( 2025-08-26T20:30:21.7590051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7590490Z hidden_states = self.encoder( 2025-08-26T20:30:21.7590904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7591317Z layer_outputs = layer_module( 2025-08-26T20:30:21.7591686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7592074Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7592513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7592944Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7593378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7593797Z return func(*args, **kwargs) 2025-08-26T20:30:21.7594246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7594757Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7595250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7595698Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7595851Z 2025-08-26T20:30:21.7596002Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7596625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7597044Z return mod(**inputs) 2025-08-26T20:30:21.7597545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7597990Z outputs = self.electra( 2025-08-26T20:30:21.7598426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7598868Z hidden_states = self.encoder( 2025-08-26T20:30:21.7599359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7599806Z layer_outputs = layer_module( 2025-08-26T20:30:21.7600190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7600668Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7601108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7601609Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7602068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7602516Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7602989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7603533Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7604033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7604489Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7604644Z 2025-08-26T20:30:21.7604768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7605168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7605545Z return mod(**inputs) 2025-08-26T20:30:21.7605968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7606413Z outputs = self.electra( 2025-08-26T20:30:21.7606828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7607270Z hidden_states = self.encoder( 2025-08-26T20:30:21.7607713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7608154Z layer_outputs = layer_module( 2025-08-26T20:30:21.7608546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7609033Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7609483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7609974Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7610423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7610891Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7611358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7611887Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7612405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7612959Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7613388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7613799Z return self.act(input) 2025-08-26T20:30:21.7613930Z 2025-08-26T20:30:21.7614056Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7614447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7614802Z return mod(**inputs) 2025-08-26T20:30:21.7615199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7615620Z outputs = self.electra( 2025-08-26T20:30:21.7616055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7616477Z hidden_states = self.encoder( 2025-08-26T20:30:21.7616888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7617326Z layer_outputs = layer_module( 2025-08-26T20:30:21.7617706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7618094Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7618559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7618996Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7619418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7619838Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7620295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7620813Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7621292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7621730Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7621886Z 2025-08-26T20:30:21.7621998Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7622387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7622737Z return mod(**inputs) 2025-08-26T20:30:21.7623126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7623546Z outputs = self.electra( 2025-08-26T20:30:21.7623945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7624363Z hidden_states = self.encoder( 2025-08-26T20:30:21.7624776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7625193Z layer_outputs = layer_module( 2025-08-26T20:30:21.7625587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7625984Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7626444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7626872Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7627288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7627688Z return func(*args, **kwargs) 2025-08-26T20:30:21.7628100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7628520Z self_outputs = self.self( 2025-08-26T20:30:21.7628900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7629323Z return func(*args, **kwargs) 2025-08-26T20:30:21.7629732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7630169Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7630316Z 2025-08-26T20:30:21.7630435Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7630815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7631165Z return mod(**inputs) 2025-08-26T20:30:21.7631567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7631989Z outputs = self.electra( 2025-08-26T20:30:21.7632383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7632825Z hidden_states = self.encoder( 2025-08-26T20:30:21.7633259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7633679Z layer_outputs = layer_module( 2025-08-26T20:30:21.7634050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7634435Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7634885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7635317Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7635727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7636131Z return func(*args, **kwargs) 2025-08-26T20:30:21.7636537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7636965Z self_outputs = self.self( 2025-08-26T20:30:21.7637352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7637757Z return func(*args, **kwargs) 2025-08-26T20:30:21.7638157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7638586Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7638735Z 2025-08-26T20:30:21.7638846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7639318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7639710Z return mod(**inputs) 2025-08-26T20:30:21.7640117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7640558Z outputs = self.electra( 2025-08-26T20:30:21.7641000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7641420Z hidden_states = self.encoder( 2025-08-26T20:30:21.7641866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7642290Z layer_outputs = layer_module( 2025-08-26T20:30:21.7642660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7643044Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7643470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7643899Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7644308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7644728Z return func(*args, **kwargs) 2025-08-26T20:30:21.7645139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7645561Z self_outputs = self.self( 2025-08-26T20:30:21.7645921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7646289Z return func(*args, **kwargs) 2025-08-26T20:30:21.7646665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7647062Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7647198Z 2025-08-26T20:30:21.7647280Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7647520Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7647759Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7648128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7648458Z return mod(**inputs) 2025-08-26T20:30:21.7648841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7649232Z outputs = self.electra( 2025-08-26T20:30:21.7649605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7649996Z hidden_states = self.encoder( 2025-08-26T20:30:21.7650378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7650774Z layer_outputs = layer_module( 2025-08-26T20:30:21.7651126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7651495Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7651896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7652300Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7652689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7653055Z return func(*args, **kwargs) 2025-08-26T20:30:21.7653431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7653873Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7654305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7654709Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7654856Z 2025-08-26T20:30:21.7654980Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7655337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7655652Z return mod(**inputs) 2025-08-26T20:30:21.7656039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7656435Z outputs = self.electra( 2025-08-26T20:30:21.7656799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7657185Z hidden_states = self.encoder( 2025-08-26T20:30:21.7657553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7657941Z layer_outputs = layer_module( 2025-08-26T20:30:21.7658278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7658667Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7659073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7659483Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7659891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7660292Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7660724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7661202Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7661661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7662081Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7662225Z 2025-08-26T20:30:21.7662343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7662714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7663046Z return mod(**inputs) 2025-08-26T20:30:21.7663436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7663845Z outputs = self.electra( 2025-08-26T20:30:21.7664236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7664644Z hidden_states = self.encoder( 2025-08-26T20:30:21.7665032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7665442Z layer_outputs = layer_module( 2025-08-26T20:30:21.7665807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7666180Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7666612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7667032Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7667453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7667853Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7668287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7668763Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7669240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7669735Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7670152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7670578Z return self.act(input) 2025-08-26T20:30:21.7670701Z 2025-08-26T20:30:21.7670812Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7671203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7671537Z return mod(**inputs) 2025-08-26T20:30:21.7671924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7672324Z outputs = self.electra( 2025-08-26T20:30:21.7672742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7673197Z hidden_states = self.encoder( 2025-08-26T20:30:21.7673612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7674033Z layer_outputs = layer_module( 2025-08-26T20:30:21.7674412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7674815Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7675272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7675735Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7676164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7676591Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7677084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7677626Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7678141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7678587Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7678740Z 2025-08-26T20:30:21.7678855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7679331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7679705Z return mod(**inputs) 2025-08-26T20:30:21.7680117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7680543Z outputs = self.electra( 2025-08-26T20:30:21.7680965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7681405Z hidden_states = self.encoder( 2025-08-26T20:30:21.7681832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7682267Z layer_outputs = layer_module( 2025-08-26T20:30:21.7682642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7683046Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7683504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7683960Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7684377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7684791Z return func(*args, **kwargs) 2025-08-26T20:30:21.7685235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7685674Z self_outputs = self.self( 2025-08-26T20:30:21.7686268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7686684Z return func(*args, **kwargs) 2025-08-26T20:30:21.7687112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7687566Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7687718Z 2025-08-26T20:30:21.7687844Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7688249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7688606Z return mod(**inputs) 2025-08-26T20:30:21.7689026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7689488Z outputs = self.electra( 2025-08-26T20:30:21.7689890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7690304Z hidden_states = self.encoder( 2025-08-26T20:30:21.7690717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7691137Z layer_outputs = layer_module( 2025-08-26T20:30:21.7691505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7691893Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7692289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7692725Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7693112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7693493Z return func(*args, **kwargs) 2025-08-26T20:30:21.7693895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7694355Z self_outputs = self.self( 2025-08-26T20:30:21.7694742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7695136Z return func(*args, **kwargs) 2025-08-26T20:30:21.7695546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7695966Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7696118Z 2025-08-26T20:30:21.7696378Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7696782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7697134Z return mod(**inputs) 2025-08-26T20:30:21.7697548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7697941Z outputs = self.electra( 2025-08-26T20:30:21.7698368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7698791Z hidden_states = self.encoder( 2025-08-26T20:30:21.7699220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7699656Z layer_outputs = layer_module( 2025-08-26T20:30:21.7700044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7700438Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7700912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7701332Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7702590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7703016Z return func(*args, **kwargs) 2025-08-26T20:30:21.7703426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7703827Z self_outputs = self.self( 2025-08-26T20:30:21.7704212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7704602Z return func(*args, **kwargs) 2025-08-26T20:30:21.7705010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7705487Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7705632Z 2025-08-26T20:30:21.7705727Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7705952Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7706214Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7706600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7706949Z return mod(**inputs) 2025-08-26T20:30:21.7707353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7707769Z outputs = self.electra( 2025-08-26T20:30:21.7708175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7708633Z hidden_states = self.encoder( 2025-08-26T20:30:21.7709047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7709463Z layer_outputs = layer_module( 2025-08-26T20:30:21.7709836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7710228Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7710658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7711094Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7711494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7711896Z return func(*args, **kwargs) 2025-08-26T20:30:21.7712304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7712794Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7713273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7713694Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7713849Z 2025-08-26T20:30:21.7713960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7714348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7714695Z return mod(**inputs) 2025-08-26T20:30:21.7715089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7715511Z outputs = self.electra( 2025-08-26T20:30:21.7715911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7716336Z hidden_states = self.encoder( 2025-08-26T20:30:21.7716766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7717180Z layer_outputs = layer_module( 2025-08-26T20:30:21.7717572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7717959Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7718401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7718847Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7719332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7719764Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7720221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7720761Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7721252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7721699Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7721859Z 2025-08-26T20:30:21.7721975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7722373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7722710Z return mod(**inputs) 2025-08-26T20:30:21.7723094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7723500Z outputs = self.electra( 2025-08-26T20:30:21.7723890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7724314Z hidden_states = self.encoder( 2025-08-26T20:30:21.7724703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7725094Z layer_outputs = layer_module( 2025-08-26T20:30:21.7725448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7725817Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7726217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7726625Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7727035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7727436Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7727870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7728346Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7728786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7729227Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7729625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7729992Z return self.act(input) 2025-08-26T20:30:21.7730124Z 2025-08-26T20:30:21.7730239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7730599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7730930Z return mod(**inputs) 2025-08-26T20:30:21.7731314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7731753Z outputs = self.electra( 2025-08-26T20:30:21.7732127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7732542Z hidden_states = self.encoder( 2025-08-26T20:30:21.7732935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7733331Z layer_outputs = layer_module( 2025-08-26T20:30:21.7733679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7734040Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7734440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7734852Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7735311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7735704Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7736124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7736615Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7737069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7737480Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7737617Z 2025-08-26T20:30:21.7737729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7738087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7738441Z return mod(**inputs) 2025-08-26T20:30:21.7738833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7739225Z outputs = self.electra( 2025-08-26T20:30:21.7739610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7740032Z hidden_states = self.encoder( 2025-08-26T20:30:21.7740471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7740891Z layer_outputs = layer_module( 2025-08-26T20:30:21.7741265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7741645Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7742059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7742462Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7742847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7743214Z return func(*args, **kwargs) 2025-08-26T20:30:21.7743594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7743992Z self_outputs = self.self( 2025-08-26T20:30:21.7744360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7744740Z return func(*args, **kwargs) 2025-08-26T20:30:21.7745121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7745552Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7745711Z 2025-08-26T20:30:21.7745832Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7746216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7746547Z return mod(**inputs) 2025-08-26T20:30:21.7746937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7747342Z outputs = self.electra( 2025-08-26T20:30:21.7747724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7748125Z hidden_states = self.encoder( 2025-08-26T20:30:21.7748510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7748915Z layer_outputs = layer_module( 2025-08-26T20:30:21.7749268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7749663Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7750064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7750469Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7750857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7751238Z return func(*args, **kwargs) 2025-08-26T20:30:21.7751630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7752021Z self_outputs = self.self( 2025-08-26T20:30:21.7752380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7752779Z return func(*args, **kwargs) 2025-08-26T20:30:21.7753194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7753626Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7753770Z 2025-08-26T20:30:21.7753883Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7754268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7754600Z return mod(**inputs) 2025-08-26T20:30:21.7755003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7755427Z outputs = self.electra( 2025-08-26T20:30:21.7755823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7756245Z hidden_states = self.encoder( 2025-08-26T20:30:21.7756659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7757085Z layer_outputs = layer_module( 2025-08-26T20:30:21.7757458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7757852Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7758295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7758738Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7759159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7759660Z return func(*args, **kwargs) 2025-08-26T20:30:21.7760087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7760536Z self_outputs = self.self( 2025-08-26T20:30:21.7760976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7761406Z return func(*args, **kwargs) 2025-08-26T20:30:21.7761829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7762348Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7762508Z 2025-08-26T20:30:21.7762601Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7762841Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7763095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7763501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7763864Z return mod(**inputs) 2025-08-26T20:30:21.7764281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7764747Z outputs = self.electra( 2025-08-26T20:30:21.7765146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7765589Z hidden_states = self.encoder( 2025-08-26T20:30:21.7766013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7766455Z layer_outputs = layer_module( 2025-08-26T20:30:21.7766828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7767261Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7767698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7768141Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7768573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7768978Z return func(*args, **kwargs) 2025-08-26T20:30:21.7769381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7769862Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7770309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7770715Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7770856Z 2025-08-26T20:30:21.7770960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7771321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7771651Z return mod(**inputs) 2025-08-26T20:30:21.7772026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7772428Z outputs = self.electra( 2025-08-26T20:30:21.7772803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7773198Z hidden_states = self.encoder( 2025-08-26T20:30:21.7773584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7773975Z layer_outputs = layer_module( 2025-08-26T20:30:21.7774316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7774689Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7775089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7775500Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7775927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7776324Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7776772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7777251Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7777698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7778112Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7778251Z 2025-08-26T20:30:21.7778359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7778767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7779137Z return mod(**inputs) 2025-08-26T20:30:21.7779556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7779974Z outputs = self.electra( 2025-08-26T20:30:21.7780379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7780776Z hidden_states = self.encoder( 2025-08-26T20:30:21.7781170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7781567Z layer_outputs = layer_module( 2025-08-26T20:30:21.7781914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7782281Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7782681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7783157Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7783587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7783979Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7784433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7784944Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7785431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7785901Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7786314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7786679Z return self.act(input) 2025-08-26T20:30:21.7786801Z 2025-08-26T20:30:21.7786923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7787315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7787661Z return mod(**inputs) 2025-08-26T20:30:21.7788064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7788494Z outputs = self.electra( 2025-08-26T20:30:21.7788912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7789345Z hidden_states = self.encoder( 2025-08-26T20:30:21.7789772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7790195Z layer_outputs = layer_module( 2025-08-26T20:30:21.7790567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7790959Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7791396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7791849Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7792280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7792695Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7793148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7793657Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7794139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7794597Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7794742Z 2025-08-26T20:30:21.7794864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7795249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7795594Z return mod(**inputs) 2025-08-26T20:30:21.7795998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7796565Z outputs = self.electra( 2025-08-26T20:30:21.7796971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7797388Z hidden_states = self.encoder( 2025-08-26T20:30:21.7797805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7798278Z layer_outputs = layer_module( 2025-08-26T20:30:21.7798652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7799039Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7799514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7799958Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7800380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7800793Z return func(*args, **kwargs) 2025-08-26T20:30:21.7801207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7801602Z self_outputs = self.self( 2025-08-26T20:30:21.7801970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7802355Z return func(*args, **kwargs) 2025-08-26T20:30:21.7802750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7803156Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7803307Z 2025-08-26T20:30:21.7803417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7803782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7804114Z return mod(**inputs) 2025-08-26T20:30:21.7804494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7804889Z outputs = self.electra( 2025-08-26T20:30:21.7805270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7805668Z hidden_states = self.encoder( 2025-08-26T20:30:21.7806100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7806500Z layer_outputs = layer_module( 2025-08-26T20:30:21.7806871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7807239Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7807645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7808052Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7808433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7808817Z return func(*args, **kwargs) 2025-08-26T20:30:21.7809204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7809639Z self_outputs = self.self( 2025-08-26T20:30:21.7810012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7810371Z return func(*args, **kwargs) 2025-08-26T20:30:21.7810748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7811140Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7811272Z 2025-08-26T20:30:21.7811382Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7811739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7812055Z return mod(**inputs) 2025-08-26T20:30:21.7812424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7812835Z outputs = self.electra( 2025-08-26T20:30:21.7813220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7813609Z hidden_states = self.encoder( 2025-08-26T20:30:21.7814002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7814401Z layer_outputs = layer_module( 2025-08-26T20:30:21.7814741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7815102Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7815484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7815898Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7816273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7816645Z return func(*args, **kwargs) 2025-08-26T20:30:21.7817014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7817402Z self_outputs = self.self( 2025-08-26T20:30:21.7817755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7818119Z return func(*args, **kwargs) 2025-08-26T20:30:21.7818492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7818888Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7819034Z 2025-08-26T20:30:21.7819117Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7819339Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7819582Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7819943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7820295Z return mod(**inputs) 2025-08-26T20:30:21.7820680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7821096Z outputs = self.electra( 2025-08-26T20:30:21.7821484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7821865Z hidden_states = self.encoder( 2025-08-26T20:30:21.7822247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7822632Z layer_outputs = layer_module( 2025-08-26T20:30:21.7822973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7823332Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7823738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7824135Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7824510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7824881Z return func(*args, **kwargs) 2025-08-26T20:30:21.7825251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7825691Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7826128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7826528Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7826723Z 2025-08-26T20:30:21.7826834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7827186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7827507Z return mod(**inputs) 2025-08-26T20:30:21.7827878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7828268Z outputs = self.electra( 2025-08-26T20:30:21.7828638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7829027Z hidden_states = self.encoder( 2025-08-26T20:30:21.7829410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7829798Z layer_outputs = layer_module( 2025-08-26T20:30:21.7830141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7830495Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7830895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7831306Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7831711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7832099Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7832515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7832988Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7833432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7833841Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7833987Z 2025-08-26T20:30:21.7834106Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7834505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7834857Z return mod(**inputs) 2025-08-26T20:30:21.7835279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7835709Z outputs = self.electra( 2025-08-26T20:30:21.7836110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7836537Z hidden_states = self.encoder( 2025-08-26T20:30:21.7836955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7837384Z layer_outputs = layer_module( 2025-08-26T20:30:21.7837764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7838178Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7838602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7839034Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7839548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7839991Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7840452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7840927Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7841345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7841787Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7842158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7842483Z return self.act(input) 2025-08-26T20:30:21.7842599Z 2025-08-26T20:30:21.7842701Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7843047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7843358Z return mod(**inputs) 2025-08-26T20:30:21.7843712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7844090Z outputs = self.electra( 2025-08-26T20:30:21.7844457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7844853Z hidden_states = self.encoder( 2025-08-26T20:30:21.7845253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7845676Z layer_outputs = layer_module( 2025-08-26T20:30:21.7846073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7846459Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7846884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7847309Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7847749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7848180Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7848648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7849212Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7849701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7850136Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7850286Z 2025-08-26T20:30:21.7850392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7850756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7851087Z return mod(**inputs) 2025-08-26T20:30:21.7851464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7851882Z outputs = self.electra( 2025-08-26T20:30:21.7852281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7852706Z hidden_states = self.encoder( 2025-08-26T20:30:21.7853098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7853489Z layer_outputs = layer_module( 2025-08-26T20:30:21.7853839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7854209Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7854613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7855013Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7855398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7855776Z return func(*args, **kwargs) 2025-08-26T20:30:21.7856184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7856584Z self_outputs = self.self( 2025-08-26T20:30:21.7856943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7857326Z return func(*args, **kwargs) 2025-08-26T20:30:21.7857713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7858119Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7858260Z 2025-08-26T20:30:21.7858375Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7858734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7859067Z return mod(**inputs) 2025-08-26T20:30:21.7859466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7859891Z outputs = self.electra( 2025-08-26T20:30:21.7860290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7860708Z hidden_states = self.encoder( 2025-08-26T20:30:21.7861123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7861554Z layer_outputs = layer_module( 2025-08-26T20:30:21.7861938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7862324Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7862764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7863210Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7863637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7864076Z return func(*args, **kwargs) 2025-08-26T20:30:21.7864502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7864955Z self_outputs = self.self( 2025-08-26T20:30:21.7865356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7865766Z return func(*args, **kwargs) 2025-08-26T20:30:21.7866181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7866622Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7866776Z 2025-08-26T20:30:21.7866891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7867291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7867672Z return mod(**inputs) 2025-08-26T20:30:21.7868081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7868517Z outputs = self.electra( 2025-08-26T20:30:21.7868931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7869366Z hidden_states = self.encoder( 2025-08-26T20:30:21.7869784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7870216Z layer_outputs = layer_module( 2025-08-26T20:30:21.7870596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7870994Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7871455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7871895Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7872317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7872731Z return func(*args, **kwargs) 2025-08-26T20:30:21.7873155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7873591Z self_outputs = self.self( 2025-08-26T20:30:21.7873981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7874388Z return func(*args, **kwargs) 2025-08-26T20:30:21.7874810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7875259Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7875410Z 2025-08-26T20:30:21.7875703Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7875944Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7876205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7876610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7876965Z return mod(**inputs) 2025-08-26T20:30:21.7877361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7877794Z outputs = self.electra( 2025-08-26T20:30:21.7878214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7878652Z hidden_states = self.encoder( 2025-08-26T20:30:21.7879072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7879587Z layer_outputs = layer_module( 2025-08-26T20:30:21.7880008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7880415Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7880887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7881327Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7881740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7882143Z return func(*args, **kwargs) 2025-08-26T20:30:21.7882552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7883041Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7883542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7883980Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7884138Z 2025-08-26T20:30:21.7884253Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7884642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7884987Z return mod(**inputs) 2025-08-26T20:30:21.7885388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7885818Z outputs = self.electra( 2025-08-26T20:30:21.7886226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7886646Z hidden_states = self.encoder( 2025-08-26T20:30:21.7887076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7887511Z layer_outputs = layer_module( 2025-08-26T20:30:21.7887910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7888301Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7888723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7889251Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7889681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7890099Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7890565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7891091Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7891564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7891996Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7892142Z 2025-08-26T20:30:21.7892262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7892648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7892986Z return mod(**inputs) 2025-08-26T20:30:21.7893389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7893814Z outputs = self.electra( 2025-08-26T20:30:21.7894218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7894638Z hidden_states = self.encoder( 2025-08-26T20:30:21.7895068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7895493Z layer_outputs = layer_module( 2025-08-26T20:30:21.7895894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7896566Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7896997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7897441Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7897875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7898299Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7898763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7899326Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7899810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7900290Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7900700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7901068Z return self.act(input) 2025-08-26T20:30:21.7901191Z 2025-08-26T20:30:21.7901304Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7901689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7902037Z return mod(**inputs) 2025-08-26T20:30:21.7902441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7902899Z outputs = self.electra( 2025-08-26T20:30:21.7903305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7903734Z hidden_states = self.encoder( 2025-08-26T20:30:21.7904131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7904530Z layer_outputs = layer_module( 2025-08-26T20:30:21.7904874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7905242Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7905648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7906060Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7906468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7906862Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7907301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7907793Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7908249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7908666Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7908808Z 2025-08-26T20:30:21.7908915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7909283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7909618Z return mod(**inputs) 2025-08-26T20:30:21.7910012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7910458Z outputs = self.electra( 2025-08-26T20:30:21.7910861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7911309Z hidden_states = self.encoder( 2025-08-26T20:30:21.7911728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7912138Z layer_outputs = layer_module( 2025-08-26T20:30:21.7912484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7912856Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7913304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7913740Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7914175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7914569Z return func(*args, **kwargs) 2025-08-26T20:30:21.7914982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7915400Z self_outputs = self.self( 2025-08-26T20:30:21.7915787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7916188Z return func(*args, **kwargs) 2025-08-26T20:30:21.7916593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7917026Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7917173Z 2025-08-26T20:30:21.7917291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7917707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7918069Z return mod(**inputs) 2025-08-26T20:30:21.7918476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7918910Z outputs = self.electra( 2025-08-26T20:30:21.7919385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7919826Z hidden_states = self.encoder( 2025-08-26T20:30:21.7920260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7920702Z layer_outputs = layer_module( 2025-08-26T20:30:21.7921095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7921511Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7921964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7922419Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7922850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7923276Z return func(*args, **kwargs) 2025-08-26T20:30:21.7923701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7924131Z self_outputs = self.self( 2025-08-26T20:30:21.7924548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7924972Z return func(*args, **kwargs) 2025-08-26T20:30:21.7925402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7925847Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7926036Z 2025-08-26T20:30:21.7926154Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7926573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7926936Z return mod(**inputs) 2025-08-26T20:30:21.7927352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7927779Z outputs = self.electra( 2025-08-26T20:30:21.7928194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7928626Z hidden_states = self.encoder( 2025-08-26T20:30:21.7929035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7929458Z layer_outputs = layer_module( 2025-08-26T20:30:21.7929848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7930244Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7930696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7931135Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7931543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7931920Z return func(*args, **kwargs) 2025-08-26T20:30:21.7932311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7932712Z self_outputs = self.self( 2025-08-26T20:30:21.7933096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7933508Z return func(*args, **kwargs) 2025-08-26T20:30:21.7933921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7934350Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7934496Z 2025-08-26T20:30:21.7934593Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7934818Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7935076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7935470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7935821Z return mod(**inputs) 2025-08-26T20:30:21.7936217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7936611Z outputs = self.electra( 2025-08-26T20:30:21.7936995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7937416Z hidden_states = self.encoder( 2025-08-26T20:30:21.7937828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7938248Z layer_outputs = layer_module( 2025-08-26T20:30:21.7938620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7939014Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7939454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7939898Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7940308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7940723Z return func(*args, **kwargs) 2025-08-26T20:30:21.7941154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7941639Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7942140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7942575Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7942745Z 2025-08-26T20:30:21.7942856Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7943237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7943596Z return mod(**inputs) 2025-08-26T20:30:21.7943999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7944432Z outputs = self.electra( 2025-08-26T20:30:21.7944854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7945276Z hidden_states = self.encoder( 2025-08-26T20:30:21.7945694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7946108Z layer_outputs = layer_module( 2025-08-26T20:30:21.7946481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7946872Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7947317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7947767Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7948194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7948637Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7949092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7949603Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7950070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7950504Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7950657Z 2025-08-26T20:30:21.7950768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7951152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7951499Z return mod(**inputs) 2025-08-26T20:30:21.7951891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7952313Z outputs = self.electra( 2025-08-26T20:30:21.7952736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7953155Z hidden_states = self.encoder( 2025-08-26T20:30:21.7953564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7953970Z layer_outputs = layer_module( 2025-08-26T20:30:21.7954336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7954716Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7955154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7955595Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7956069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7956505Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7956995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7957521Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7958002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7958480Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7958900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7959351Z return self.act(input) 2025-08-26T20:30:21.7959487Z 2025-08-26T20:30:21.7959616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7960046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7960415Z return mod(**inputs) 2025-08-26T20:30:21.7960834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7961272Z outputs = self.electra( 2025-08-26T20:30:21.7961691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7962121Z hidden_states = self.encoder( 2025-08-26T20:30:21.7962550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7962982Z layer_outputs = layer_module( 2025-08-26T20:30:21.7963371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7963785Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7964236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7964691Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7965139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7965580Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7966049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7966595Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7967103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7967559Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7967717Z 2025-08-26T20:30:21.7967845Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7968247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7968619Z return mod(**inputs) 2025-08-26T20:30:21.7969042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7969485Z outputs = self.electra( 2025-08-26T20:30:21.7969897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7970343Z hidden_states = self.encoder( 2025-08-26T20:30:21.7970761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7971189Z layer_outputs = layer_module( 2025-08-26T20:30:21.7971566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7971951Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7972404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7972841Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7973272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7973672Z return func(*args, **kwargs) 2025-08-26T20:30:21.7974074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7974496Z self_outputs = self.self( 2025-08-26T20:30:21.7974762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7974838Z return func(*args, **kwargs) 2025-08-26T20:30:21.7975123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.7975286Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.7975290Z 2025-08-26T20:30:21.7975403Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7975626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7975699Z return mod(**inputs) 2025-08-26T20:30:21.7975986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7976069Z outputs = self.electra( 2025-08-26T20:30:21.7976351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7976437Z hidden_states = self.encoder( 2025-08-26T20:30:21.7976717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7976817Z layer_outputs = layer_module( 2025-08-26T20:30:21.7977066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7977152Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7977439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7977530Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7977795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7977870Z return func(*args, **kwargs) 2025-08-26T20:30:21.7978151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7978240Z self_outputs = self.self( 2025-08-26T20:30:21.7978499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7978583Z return func(*args, **kwargs) 2025-08-26T20:30:21.7978867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.7978953Z key_layer = self.key(current_states) 2025-08-26T20:30:21.7978957Z 2025-08-26T20:30:21.7979076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7979293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7979371Z return mod(**inputs) 2025-08-26T20:30:21.7979655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7979737Z outputs = self.electra( 2025-08-26T20:30:21.7980015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7980094Z hidden_states = self.encoder( 2025-08-26T20:30:21.7980403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7980501Z layer_outputs = layer_module( 2025-08-26T20:30:21.7980746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7980829Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7981105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7981202Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7981458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7981540Z return func(*args, **kwargs) 2025-08-26T20:30:21.7981842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.7981917Z self_outputs = self.self( 2025-08-26T20:30:21.7982182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7982255Z return func(*args, **kwargs) 2025-08-26T20:30:21.7982540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.7982626Z value_layer = self.value(current_states) 2025-08-26T20:30:21.7982630Z 2025-08-26T20:30:21.7982724Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7982810Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.7982920Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7983140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7983232Z return mod(**inputs) 2025-08-26T20:30:21.7983529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7983606Z outputs = self.electra( 2025-08-26T20:30:21.7983890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7983977Z hidden_states = self.encoder( 2025-08-26T20:30:21.7984263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7984346Z layer_outputs = layer_module( 2025-08-26T20:30:21.7984588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7984671Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7984962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7985052Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7985323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7985398Z return func(*args, **kwargs) 2025-08-26T20:30:21.7985683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.7985832Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.7986118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.7986216Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7986220Z 2025-08-26T20:30:21.7986330Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7986555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7986628Z return mod(**inputs) 2025-08-26T20:30:21.7986936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7987019Z outputs = self.electra( 2025-08-26T20:30:21.7987318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7987403Z hidden_states = self.encoder( 2025-08-26T20:30:21.7987681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7987756Z layer_outputs = layer_module( 2025-08-26T20:30:21.7987999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7988082Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7988364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7988475Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7988758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7988839Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7989151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7989288Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7989568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.7989663Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7989667Z 2025-08-26T20:30:21.7989809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7990029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7990106Z return mod(**inputs) 2025-08-26T20:30:21.7990387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7990467Z outputs = self.electra( 2025-08-26T20:30:21.7990746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7990829Z hidden_states = self.encoder( 2025-08-26T20:30:21.7991105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7991181Z layer_outputs = layer_module( 2025-08-26T20:30:21.7991424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7991510Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7991802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7991890Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7992167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7992256Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7992575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.7992712Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.7993050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.7993192Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.7993423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.7993519Z return self.act(input) 2025-08-26T20:30:21.7993524Z 2025-08-26T20:30:21.7993645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7993881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7993961Z return mod(**inputs) 2025-08-26T20:30:21.7994247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7994321Z outputs = self.electra( 2025-08-26T20:30:21.7994608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7994685Z hidden_states = self.encoder( 2025-08-26T20:30:21.7994973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7995071Z layer_outputs = layer_module( 2025-08-26T20:30:21.7995312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7995404Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7995683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.7995780Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.7996058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.7996146Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.7996615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.7996817Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.7997110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.7997201Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.7997206Z 2025-08-26T20:30:21.7997331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.7997550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.7997625Z return mod(**inputs) 2025-08-26T20:30:21.7997924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.7998001Z outputs = self.electra( 2025-08-26T20:30:21.7998301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.7998378Z hidden_states = self.encoder( 2025-08-26T20:30:21.7998682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.7998763Z layer_outputs = layer_module( 2025-08-26T20:30:21.7999005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.7999099Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.7999441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.7999548Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.7999825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.7999900Z return func(*args, **kwargs) 2025-08-26T20:30:21.8000197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8000279Z self_outputs = self.self( 2025-08-26T20:30:21.8000593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8000673Z return func(*args, **kwargs) 2025-08-26T20:30:21.8001006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.8001098Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.8001103Z 2025-08-26T20:30:21.8001219Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8001459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8001526Z return mod(**inputs) 2025-08-26T20:30:21.8001801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8001873Z outputs = self.electra( 2025-08-26T20:30:21.8002170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8002250Z hidden_states = self.encoder( 2025-08-26T20:30:21.8002516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8002594Z layer_outputs = layer_module( 2025-08-26T20:30:21.8002817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8002896Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8003166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8003246Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8003496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8003587Z return func(*args, **kwargs) 2025-08-26T20:30:21.8003862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8003932Z self_outputs = self.self( 2025-08-26T20:30:21.8004180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8004257Z return func(*args, **kwargs) 2025-08-26T20:30:21.8004525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.8004613Z key_layer = self.key(current_states) 2025-08-26T20:30:21.8004617Z 2025-08-26T20:30:21.8004720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8004919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8004996Z return mod(**inputs) 2025-08-26T20:30:21.8005270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8005346Z outputs = self.electra( 2025-08-26T20:30:21.8005612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8005683Z hidden_states = self.encoder( 2025-08-26T20:30:21.8005954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8006025Z layer_outputs = layer_module( 2025-08-26T20:30:21.8006256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8006333Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8006607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8006693Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8006962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8007042Z return func(*args, **kwargs) 2025-08-26T20:30:21.8007325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8007406Z self_outputs = self.self( 2025-08-26T20:30:21.8007649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8007718Z return func(*args, **kwargs) 2025-08-26T20:30:21.8007990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.8008071Z value_layer = self.value(current_states) 2025-08-26T20:30:21.8008077Z 2025-08-26T20:30:21.8008165Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8008265Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8008372Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8008579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8008647Z return mod(**inputs) 2025-08-26T20:30:21.8008922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8008991Z outputs = self.electra( 2025-08-26T20:30:21.8009262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8009335Z hidden_states = self.encoder( 2025-08-26T20:30:21.8009606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8009705Z layer_outputs = layer_module( 2025-08-26T20:30:21.8009927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8010011Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8010275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8010357Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8010606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8010676Z return func(*args, **kwargs) 2025-08-26T20:30:21.8010946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.8011076Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.8011349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.8011436Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8011442Z 2025-08-26T20:30:21.8011546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8011756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8011822Z return mod(**inputs) 2025-08-26T20:30:21.8012094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8012172Z outputs = self.electra( 2025-08-26T20:30:21.8012429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8012506Z hidden_states = self.encoder( 2025-08-26T20:30:21.8012759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8012837Z layer_outputs = layer_module( 2025-08-26T20:30:21.8013074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8013151Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8013428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8013513Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8013778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8013856Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8014156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8014275Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8014537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.8014656Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8014660Z 2025-08-26T20:30:21.8014762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8014965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8015030Z return mod(**inputs) 2025-08-26T20:30:21.8015286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8015360Z outputs = self.electra( 2025-08-26T20:30:21.8015614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8015691Z hidden_states = self.encoder( 2025-08-26T20:30:21.8015942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8016040Z layer_outputs = layer_module( 2025-08-26T20:30:21.8016262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8016340Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8016605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8016687Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8016947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8017021Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8017306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8017433Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8017692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.8017809Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.8018023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.8018105Z return self.act(input) 2025-08-26T20:30:21.8018109Z 2025-08-26T20:30:21.8018214Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8018419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8018494Z return mod(**inputs) 2025-08-26T20:30:21.8018760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8018837Z outputs = self.electra( 2025-08-26T20:30:21.8019126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8019197Z hidden_states = self.encoder( 2025-08-26T20:30:21.8019480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8019551Z layer_outputs = layer_module( 2025-08-26T20:30:21.8019775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8019852Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8020104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8020195Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8020447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8020551Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8020841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.8020977Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.8021234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.8021314Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8021317Z 2025-08-26T20:30:21.8021425Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8021618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8021691Z return mod(**inputs) 2025-08-26T20:30:21.8021950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8022037Z outputs = self.electra( 2025-08-26T20:30:21.8022303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8022373Z hidden_states = self.encoder( 2025-08-26T20:30:21.8022638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8022707Z layer_outputs = layer_module( 2025-08-26T20:30:21.8022936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8023012Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8023268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8023356Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8023599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8023677Z return func(*args, **kwargs) 2025-08-26T20:30:21.8023935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8024005Z self_outputs = self.self( 2025-08-26T20:30:21.8024250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8024320Z return func(*args, **kwargs) 2025-08-26T20:30:21.8024589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.8024669Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.8024672Z 2025-08-26T20:30:21.8024782Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8024980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8025047Z return mod(**inputs) 2025-08-26T20:30:21.8025333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8025404Z outputs = self.electra( 2025-08-26T20:30:21.8025685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8025757Z hidden_states = self.encoder( 2025-08-26T20:30:21.8026013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8026092Z layer_outputs = layer_module( 2025-08-26T20:30:21.8026310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8026395Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8026656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8026761Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8027009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8027081Z return func(*args, **kwargs) 2025-08-26T20:30:21.8027347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8027416Z self_outputs = self.self( 2025-08-26T20:30:21.8027662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8027744Z return func(*args, **kwargs) 2025-08-26T20:30:21.8028011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.8028122Z key_layer = self.key(current_states) 2025-08-26T20:30:21.8028127Z 2025-08-26T20:30:21.8028233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8028441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8028508Z return mod(**inputs) 2025-08-26T20:30:21.8028774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8028852Z outputs = self.electra( 2025-08-26T20:30:21.8029114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8029201Z hidden_states = self.encoder( 2025-08-26T20:30:21.8029456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8029525Z layer_outputs = layer_module( 2025-08-26T20:30:21.8029752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8029834Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8030100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8030182Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8030427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8030497Z return func(*args, **kwargs) 2025-08-26T20:30:21.8030754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8030831Z self_outputs = self.self( 2025-08-26T20:30:21.8031068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8031145Z return func(*args, **kwargs) 2025-08-26T20:30:21.8031427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.8031511Z value_layer = self.value(current_states) 2025-08-26T20:30:21.8031514Z 2025-08-26T20:30:21.8031601Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8031700Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8031811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8032009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8032075Z return mod(**inputs) 2025-08-26T20:30:21.8032344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8032413Z outputs = self.electra( 2025-08-26T20:30:21.8032684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8032774Z hidden_states = self.encoder( 2025-08-26T20:30:21.8033029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8033096Z layer_outputs = layer_module( 2025-08-26T20:30:21.8033312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8033399Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8033651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8033735Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8033973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8034040Z return func(*args, **kwargs) 2025-08-26T20:30:21.8034326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.8034455Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.8034719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.8034801Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8034805Z 2025-08-26T20:30:21.8034913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8035107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8035171Z return mod(**inputs) 2025-08-26T20:30:21.8035437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8035504Z outputs = self.electra( 2025-08-26T20:30:21.8035770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8035843Z hidden_states = self.encoder( 2025-08-26T20:30:21.8036100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8036180Z layer_outputs = layer_module( 2025-08-26T20:30:21.8036397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8036483Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8036738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8036822Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8037080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8037158Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8037473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8037593Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8037883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.8037966Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8037970Z 2025-08-26T20:30:21.8038072Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8038276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8038341Z return mod(**inputs) 2025-08-26T20:30:21.8038622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8038696Z outputs = self.electra( 2025-08-26T20:30:21.8038998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8039082Z hidden_states = self.encoder( 2025-08-26T20:30:21.8039446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8039539Z layer_outputs = layer_module( 2025-08-26T20:30:21.8039789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8039884Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8040192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8040284Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8040569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8040683Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8040977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8041098Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8041352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.8041473Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.8041684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.8041763Z return self.act(input) 2025-08-26T20:30:21.8041767Z 2025-08-26T20:30:21.8041869Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8056810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8057061Z return mod(**inputs) 2025-08-26T20:30:21.8057415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8057503Z outputs = self.electra( 2025-08-26T20:30:21.8057784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8057868Z hidden_states = self.encoder( 2025-08-26T20:30:21.8058134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8058213Z layer_outputs = layer_module( 2025-08-26T20:30:21.8058439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8058526Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8058802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8058990Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8059263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8059381Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8059677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.8059825Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.8060097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.8060196Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8060201Z 2025-08-26T20:30:21.8060319Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8060549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8060655Z return mod(**inputs) 2025-08-26T20:30:21.8060923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8061008Z outputs = self.electra( 2025-08-26T20:30:21.8061267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8061349Z hidden_states = self.encoder( 2025-08-26T20:30:21.8061615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8061689Z layer_outputs = layer_module( 2025-08-26T20:30:21.8061925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8062041Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8062322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8062413Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8062666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8062756Z return func(*args, **kwargs) 2025-08-26T20:30:21.8063027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8063113Z self_outputs = self.self( 2025-08-26T20:30:21.8063369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8063452Z return func(*args, **kwargs) 2025-08-26T20:30:21.8063715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.8063803Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.8063807Z 2025-08-26T20:30:21.8063929Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8064137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8064216Z return mod(**inputs) 2025-08-26T20:30:21.8064483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8064555Z outputs = self.electra( 2025-08-26T20:30:21.8064820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8064895Z hidden_states = self.encoder( 2025-08-26T20:30:21.8065166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8065245Z layer_outputs = layer_module( 2025-08-26T20:30:21.8065502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8065584Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8065866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8065958Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8066204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8066282Z return func(*args, **kwargs) 2025-08-26T20:30:21.8066543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8066614Z self_outputs = self.self( 2025-08-26T20:30:21.8066866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8066963Z return func(*args, **kwargs) 2025-08-26T20:30:21.8067244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.8067323Z key_layer = self.key(current_states) 2025-08-26T20:30:21.8067329Z 2025-08-26T20:30:21.8067441Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8067647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8067715Z return mod(**inputs) 2025-08-26T20:30:21.8067991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8068061Z outputs = self.electra( 2025-08-26T20:30:21.8068337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8068429Z hidden_states = self.encoder( 2025-08-26T20:30:21.8068703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8068784Z layer_outputs = layer_module( 2025-08-26T20:30:21.8069015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8069100Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8069370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8069452Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8069713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8069783Z return func(*args, **kwargs) 2025-08-26T20:30:21.8070056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8070131Z self_outputs = self.self( 2025-08-26T20:30:21.8070389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8070458Z return func(*args, **kwargs) 2025-08-26T20:30:21.8070728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.8070817Z value_layer = self.value(current_states) 2025-08-26T20:30:21.8070820Z 2025-08-26T20:30:21.8070905Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8070994Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8071101Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8071308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8071383Z return mod(**inputs) 2025-08-26T20:30:21.8071661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8071757Z outputs = self.electra( 2025-08-26T20:30:21.8072018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8072109Z hidden_states = self.encoder( 2025-08-26T20:30:21.8072390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8072465Z layer_outputs = layer_module( 2025-08-26T20:30:21.8072711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8072794Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8073080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8073169Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8073447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8073529Z return func(*args, **kwargs) 2025-08-26T20:30:21.8073807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.8073954Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.8074231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.8074319Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8074322Z 2025-08-26T20:30:21.8074441Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8074655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8074752Z return mod(**inputs) 2025-08-26T20:30:21.8075034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8075113Z outputs = self.electra( 2025-08-26T20:30:21.8075395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8075471Z hidden_states = self.encoder( 2025-08-26T20:30:21.8075753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8075829Z layer_outputs = layer_module( 2025-08-26T20:30:21.8076070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8076154Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8076428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8076530Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8076804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8076894Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8077209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8077342Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8077628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.8077716Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8077720Z 2025-08-26T20:30:21.8077837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8078050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8078131Z return mod(**inputs) 2025-08-26T20:30:21.8078445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8078521Z outputs = self.electra( 2025-08-26T20:30:21.8078821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8078899Z hidden_states = self.encoder( 2025-08-26T20:30:21.8079185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8079361Z layer_outputs = layer_module( 2025-08-26T20:30:21.8079617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8079711Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8080006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8080129Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8080412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8080517Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8080814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8080938Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8081210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.8081330Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.8081554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.8081648Z return self.act(input) 2025-08-26T20:30:21.8081651Z 2025-08-26T20:30:21.8081761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8081974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8082043Z return mod(**inputs) 2025-08-26T20:30:21.8082321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8082392Z outputs = self.electra( 2025-08-26T20:30:21.8082664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8082736Z hidden_states = self.encoder( 2025-08-26T20:30:21.8083002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8083084Z layer_outputs = layer_module( 2025-08-26T20:30:21.8083311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8083397Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8083662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8083746Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8084013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8084090Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8084396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.8084530Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.8084790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.8084895Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8084899Z 2025-08-26T20:30:21.8085007Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8085230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8085298Z return mod(**inputs) 2025-08-26T20:30:21.8085577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8085645Z outputs = self.electra( 2025-08-26T20:30:21.8085907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8085986Z hidden_states = self.encoder( 2025-08-26T20:30:21.8086249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8086349Z layer_outputs = layer_module( 2025-08-26T20:30:21.8086571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8086652Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8086926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8087008Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8087262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8087332Z return func(*args, **kwargs) 2025-08-26T20:30:21.8087601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8087673Z self_outputs = self.self( 2025-08-26T20:30:21.8087941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8088022Z return func(*args, **kwargs) 2025-08-26T20:30:21.8088285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:21.8088375Z query_layer = self.query(hidden_states) 2025-08-26T20:30:21.8088379Z 2025-08-26T20:30:21.8088483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8088692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8088763Z return mod(**inputs) 2025-08-26T20:30:21.8089025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8089100Z outputs = self.electra( 2025-08-26T20:30:21.8089357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8089429Z hidden_states = self.encoder( 2025-08-26T20:30:21.8089694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8089762Z layer_outputs = layer_module( 2025-08-26T20:30:21.8089991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8090069Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8090338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8090421Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8090668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8090746Z return func(*args, **kwargs) 2025-08-26T20:30:21.8091012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8091111Z self_outputs = self.self( 2025-08-26T20:30:21.8091359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8091451Z return func(*args, **kwargs) 2025-08-26T20:30:21.8091727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:21.8091811Z key_layer = self.key(current_states) 2025-08-26T20:30:21.8091815Z 2025-08-26T20:30:21.8091933Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8092146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8092223Z return mod(**inputs) 2025-08-26T20:30:21.8092509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8092625Z outputs = self.electra( 2025-08-26T20:30:21.8092922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8093007Z hidden_states = self.encoder( 2025-08-26T20:30:21.8093286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8093357Z layer_outputs = layer_module( 2025-08-26T20:30:21.8093596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8093681Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8093943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8094032Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8094287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8094360Z return func(*args, **kwargs) 2025-08-26T20:30:21.8094628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:21.8094697Z self_outputs = self.self( 2025-08-26T20:30:21.8094941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8095009Z return func(*args, **kwargs) 2025-08-26T20:30:21.8095272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:21.8095349Z value_layer = self.value(current_states) 2025-08-26T20:30:21.8095353Z 2025-08-26T20:30:21.8095435Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8095527Z cudagraph partition due to non gpu ops 2025-08-26T20:30:21.8095634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8095846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8095915Z return mod(**inputs) 2025-08-26T20:30:21.8096325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8096415Z outputs = self.electra( 2025-08-26T20:30:21.8096683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8096766Z hidden_states = self.encoder( 2025-08-26T20:30:21.8097031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8097106Z layer_outputs = layer_module( 2025-08-26T20:30:21.8097348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8097438Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8097783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:21.8097905Z self_attention_outputs = self.attention( 2025-08-26T20:30:21.8098182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:21.8098256Z return func(*args, **kwargs) 2025-08-26T20:30:21.8098534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:21.8098686Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:21.8098952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:21.8099048Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8099085Z 2025-08-26T20:30:21.8099197Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8099397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8099473Z return mod(**inputs) 2025-08-26T20:30:21.8099745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8099822Z outputs = self.electra( 2025-08-26T20:30:21.8100083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8100155Z hidden_states = self.encoder( 2025-08-26T20:30:21.8100426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8100498Z layer_outputs = layer_module( 2025-08-26T20:30:21.8100784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8100871Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8101168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8101253Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8101513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8101598Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8101896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8102026Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8102290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:21.8102375Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8102388Z 2025-08-26T20:30:21.8102499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8102722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8102802Z return mod(**inputs) 2025-08-26T20:30:21.8103094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8103170Z outputs = self.electra( 2025-08-26T20:30:21.8103429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8103500Z hidden_states = self.encoder( 2025-08-26T20:30:21.8103769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8103842Z layer_outputs = layer_module( 2025-08-26T20:30:21.8104093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8104174Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8104456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8104550Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8104808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8104892Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8105187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:21.8105315Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:21.8105578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:21.8105714Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:21.8105938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:21.8106012Z return self.act(input) 2025-08-26T20:30:21.8106015Z 2025-08-26T20:30:21.8106129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8106332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8106399Z return mod(**inputs) 2025-08-26T20:30:21.8106676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-26T20:30:21.8106747Z outputs = self.electra( 2025-08-26T20:30:21.8107022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:21.8107112Z hidden_states = self.encoder( 2025-08-26T20:30:21.8107375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:21.8107454Z layer_outputs = layer_module( 2025-08-26T20:30:21.8107676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:21.8107763Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:21.8108024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:21.8108116Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:21.8108376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:21.8108451Z return forward_fn(*input_tensors) 2025-08-26T20:30:21.8108754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:21.8108893Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:21.8109163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:21.8109247Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:21.8109251Z 2025-08-26T20:30:21.8109362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8109563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8109629Z return mod(**inputs) 2025-08-26T20:30:21.8109899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-08-26T20:30:21.8110081Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-08-26T20:30:21.8110370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 640, in forward 2025-08-26T20:30:21.8110476Z hidden_states = self.dense(generator_hidden_states) 2025-08-26T20:30:21.8110480Z 2025-08-26T20:30:21.8110616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8110823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8110890Z return mod(**inputs) 2025-08-26T20:30:21.8111164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-08-26T20:30:21.8111343Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-08-26T20:30:21.8111347Z 2025-08-26T20:30:21.8111459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:21.8111658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:21.8111746Z return mod(**inputs) 2025-08-26T20:30:21.8112040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1564, in forward 2025-08-26T20:30:21.8112118Z lm_loss = self.loss_function( 2025-08-26T20:30:21.8112390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-26T20:30:21.8112577Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-26T20:30:21.8112851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-26T20:30:21.8113066Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-26T20:30:21.8113071Z 2025-08-26T20:30:30.4135402Z Compilation time (from dynamo_timed): 16.006025809 2025-08-26T20:30:30.4227885Z pass 2025-08-26T20:30:30.4228309Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:30.4229342Z TIMING: _recursive_pre_grad_passes:0.00784 _recursive_joint_graph_passes:0.47708 _recursive_post_grad_passes:0.07687 async_compile.wait:0.79739 code_gen:7.96294 inductor_compile:9.25088 backend_compile:12.97402 gc:0.00138 entire_frame_compile:16.00603 total_wall_time:16.00603 2025-08-26T20:30:30.4230288Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:15035 | FakeTensor.__torch_dispatch__:4346 | ProxyTorchDispatchMode.__torch_dispatch__:5671 2025-08-26T20:30:30.4230776Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-08-26T20:30:35.7928760Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:30:35.7929803Z from pkg_resources import resource_filename 2025-08-26T20:30:36.4139379Z 2025-08-26T20:30:36.8358668Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:30:36.8358997Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:30:36.8372043Z cpu eval ElectraForQuestionAnswering 2025-08-26T20:30:36.9583773Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:37.0206040Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:37.0806135Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:45.6501314Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6501869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6502288Z return mod(**inputs) 2025-08-26T20:30:45.6503082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6503590Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6504152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-08-26T20:30:45.6504666Z hidden_states = self.embeddings_project(hidden_states) 2025-08-26T20:30:45.6504880Z 2025-08-26T20:30:45.6505012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6505383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6505724Z return mod(**inputs) 2025-08-26T20:30:45.6506147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6506610Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6507147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6507584Z hidden_states = self.encoder( 2025-08-26T20:30:45.6508020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6508465Z layer_outputs = layer_module( 2025-08-26T20:30:45.6508852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6509265Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6509707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6510172Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6510608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6511099Z return func(*args, **kwargs) 2025-08-26T20:30:45.6511509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6511938Z self_outputs = self.self( 2025-08-26T20:30:45.6512341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6512814Z return func(*args, **kwargs) 2025-08-26T20:30:45.6513248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6513722Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6513891Z 2025-08-26T20:30:45.6514016Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6514434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6514811Z return mod(**inputs) 2025-08-26T20:30:45.6515241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6515695Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6516151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6516725Z hidden_states = self.encoder( 2025-08-26T20:30:45.6517164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6517604Z layer_outputs = layer_module( 2025-08-26T20:30:45.6517986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6518392Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6518838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6519666Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6520152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6520592Z return func(*args, **kwargs) 2025-08-26T20:30:45.6521059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6521519Z self_outputs = self.self( 2025-08-26T20:30:45.6521940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6522343Z return func(*args, **kwargs) 2025-08-26T20:30:45.6522772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6523209Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6523360Z 2025-08-26T20:30:45.6523513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6523923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6524287Z return mod(**inputs) 2025-08-26T20:30:45.6524716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6525179Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6525638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6526071Z hidden_states = self.encoder( 2025-08-26T20:30:45.6526506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6526950Z layer_outputs = layer_module( 2025-08-26T20:30:45.6527341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6527801Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6528245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6528691Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6529123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6529547Z return func(*args, **kwargs) 2025-08-26T20:30:45.6529975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6530408Z self_outputs = self.self( 2025-08-26T20:30:45.6530823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6531246Z return func(*args, **kwargs) 2025-08-26T20:30:45.6531674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6532112Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6532267Z 2025-08-26T20:30:45.6532359Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6532601Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6532863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6533265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6533642Z return mod(**inputs) 2025-08-26T20:30:45.6534060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6534518Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6534971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6535406Z hidden_states = self.encoder( 2025-08-26T20:30:45.6535899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6536323Z layer_outputs = layer_module( 2025-08-26T20:30:45.6536722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6537110Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6537531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6537981Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6538422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6538864Z return func(*args, **kwargs) 2025-08-26T20:30:45.6539286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6539835Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6540325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6540769Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6540931Z 2025-08-26T20:30:45.6541048Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6541453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6541807Z return mod(**inputs) 2025-08-26T20:30:45.6542202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6542660Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6543134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6543574Z hidden_states = self.encoder( 2025-08-26T20:30:45.6544000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6544438Z layer_outputs = layer_module( 2025-08-26T20:30:45.6544815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6545204Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6545630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6546083Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6546524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6546957Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6547442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6547980Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6548457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6548883Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6549035Z 2025-08-26T20:30:45.6549148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6549538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6549889Z return mod(**inputs) 2025-08-26T20:30:45.6550288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6550734Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6551195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6551621Z hidden_states = self.encoder( 2025-08-26T20:30:45.6552049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6552466Z layer_outputs = layer_module( 2025-08-26T20:30:45.6552841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6553235Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6553678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6554132Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6554555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6555017Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6555489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6556026Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6556490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6556956Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6557363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6557734Z return self.act(input) 2025-08-26T20:30:45.6557857Z 2025-08-26T20:30:45.6557979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6558370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6558749Z return mod(**inputs) 2025-08-26T20:30:45.6559168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6559730Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6560184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6560613Z hidden_states = self.encoder( 2025-08-26T20:30:45.6561041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6561478Z layer_outputs = layer_module( 2025-08-26T20:30:45.6561863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6562259Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6562704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6563165Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6563612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6564047Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6564505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6565046Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6565542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6565988Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6566140Z 2025-08-26T20:30:45.6566266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6566736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6567106Z return mod(**inputs) 2025-08-26T20:30:45.6567550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6568012Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6568461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6568886Z hidden_states = self.encoder( 2025-08-26T20:30:45.6569317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6569763Z layer_outputs = layer_module( 2025-08-26T20:30:45.6570137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6570543Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6570980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6571427Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6571851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6572262Z return func(*args, **kwargs) 2025-08-26T20:30:45.6572674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6573105Z self_outputs = self.self( 2025-08-26T20:30:45.6573507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6573916Z return func(*args, **kwargs) 2025-08-26T20:30:45.6574342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6574779Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6574935Z 2025-08-26T20:30:45.6575047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6575442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6575795Z return mod(**inputs) 2025-08-26T20:30:45.6576194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6576645Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6577085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6577533Z hidden_states = self.encoder( 2025-08-26T20:30:45.6577979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6578400Z layer_outputs = layer_module( 2025-08-26T20:30:45.6578776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6579164Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6579610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6580053Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6580472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6580890Z return func(*args, **kwargs) 2025-08-26T20:30:45.6581314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6581746Z self_outputs = self.self( 2025-08-26T20:30:45.6582143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6582565Z return func(*args, **kwargs) 2025-08-26T20:30:45.6582978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6583423Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6583568Z 2025-08-26T20:30:45.6583688Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6584071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6584421Z return mod(**inputs) 2025-08-26T20:30:45.6584822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6585261Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6585691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6586147Z hidden_states = self.encoder( 2025-08-26T20:30:45.6586561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6586984Z layer_outputs = layer_module( 2025-08-26T20:30:45.6587353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6587734Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6588164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6588599Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6589005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6589423Z return func(*args, **kwargs) 2025-08-26T20:30:45.6589832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6590256Z self_outputs = self.self( 2025-08-26T20:30:45.6590646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6591046Z return func(*args, **kwargs) 2025-08-26T20:30:45.6591457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6591904Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6592060Z 2025-08-26T20:30:45.6592151Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6592392Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6592657Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6593048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6593410Z return mod(**inputs) 2025-08-26T20:30:45.6593826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6594281Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6594727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6595161Z hidden_states = self.encoder( 2025-08-26T20:30:45.6595588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6596022Z layer_outputs = layer_module( 2025-08-26T20:30:45.6596646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6597050Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6597496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6598020Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6598451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6598923Z return func(*args, **kwargs) 2025-08-26T20:30:45.6599430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6599946Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6600456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6600918Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6601081Z 2025-08-26T20:30:45.6601193Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6601586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6601996Z return mod(**inputs) 2025-08-26T20:30:45.6602418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6602877Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6603321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6603752Z hidden_states = self.encoder( 2025-08-26T20:30:45.6604178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6604623Z layer_outputs = layer_module( 2025-08-26T20:30:45.6605006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6605434Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6605884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6606334Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6606773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6607204Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6607673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6608201Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6608691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6609138Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6609292Z 2025-08-26T20:30:45.6609409Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6609817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6610181Z return mod(**inputs) 2025-08-26T20:30:45.6610601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6611056Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6611506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6611943Z hidden_states = self.encoder( 2025-08-26T20:30:45.6612377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6612777Z layer_outputs = layer_module( 2025-08-26T20:30:45.6613134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6613526Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6613982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6614429Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6614872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6615289Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6615750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6616262Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6616731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6617201Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6617627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6617999Z return self.act(input) 2025-08-26T20:30:45.6618127Z 2025-08-26T20:30:45.6618243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6618637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6618994Z return mod(**inputs) 2025-08-26T20:30:45.6619403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6619866Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6620332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6620750Z hidden_states = self.encoder( 2025-08-26T20:30:45.6621181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6621603Z layer_outputs = layer_module( 2025-08-26T20:30:45.6621975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6622363Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6622794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6623218Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6623645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6624070Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6624521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6625042Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6625520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6625953Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6626106Z 2025-08-26T20:30:45.6626217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6626600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6626950Z return mod(**inputs) 2025-08-26T20:30:45.6627346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6627786Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6628223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6628647Z hidden_states = self.encoder( 2025-08-26T20:30:45.6629099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6629527Z layer_outputs = layer_module( 2025-08-26T20:30:45.6629924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6630320Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6630753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6631183Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6631611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6632020Z return func(*args, **kwargs) 2025-08-26T20:30:45.6632433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6632881Z self_outputs = self.self( 2025-08-26T20:30:45.6633273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6633677Z return func(*args, **kwargs) 2025-08-26T20:30:45.6634086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6634516Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6634664Z 2025-08-26T20:30:45.6634779Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6635185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6635545Z return mod(**inputs) 2025-08-26T20:30:45.6635959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6636435Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6636881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6637319Z hidden_states = self.encoder( 2025-08-26T20:30:45.6637751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6638186Z layer_outputs = layer_module( 2025-08-26T20:30:45.6638561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6638963Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6639501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6639953Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6640375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6640794Z return func(*args, **kwargs) 2025-08-26T20:30:45.6641204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6641626Z self_outputs = self.self( 2025-08-26T20:30:45.6642008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6642404Z return func(*args, **kwargs) 2025-08-26T20:30:45.6642805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6643236Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6643391Z 2025-08-26T20:30:45.6643507Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6643895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6644236Z return mod(**inputs) 2025-08-26T20:30:45.6644672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6645210Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6645654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6646077Z hidden_states = self.encoder( 2025-08-26T20:30:45.6646505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6646929Z layer_outputs = layer_module( 2025-08-26T20:30:45.6647299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6647693Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6648144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6648577Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6648991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6649389Z return func(*args, **kwargs) 2025-08-26T20:30:45.6649800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6650216Z self_outputs = self.self( 2025-08-26T20:30:45.6650608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6651007Z return func(*args, **kwargs) 2025-08-26T20:30:45.6651421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6651879Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6652025Z 2025-08-26T20:30:45.6652115Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6652361Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6652606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6652978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6653302Z return mod(**inputs) 2025-08-26T20:30:45.6653686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6654106Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6654544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6654961Z hidden_states = self.encoder( 2025-08-26T20:30:45.6655349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6655753Z layer_outputs = layer_module( 2025-08-26T20:30:45.6656114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6656508Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6656941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6657369Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6657776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6658176Z return func(*args, **kwargs) 2025-08-26T20:30:45.6658562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6659012Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6659529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6659952Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6660097Z 2025-08-26T20:30:45.6660233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6660626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6660969Z return mod(**inputs) 2025-08-26T20:30:45.6661368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6661816Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6662230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6662630Z hidden_states = self.encoder( 2025-08-26T20:30:45.6663050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6663470Z layer_outputs = layer_module( 2025-08-26T20:30:45.6663843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6664230Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6664652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6665090Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6665530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6665927Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6666358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6666873Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6667349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6667784Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6667935Z 2025-08-26T20:30:45.6668057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6668477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6668832Z return mod(**inputs) 2025-08-26T20:30:45.6669236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6669682Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6670133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6670780Z hidden_states = self.encoder( 2025-08-26T20:30:45.6671226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6671666Z layer_outputs = layer_module( 2025-08-26T20:30:45.6672070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6672465Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6672910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6673375Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6673814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6674253Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6674750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6675266Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6675782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6676267Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6676691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6677078Z return self.act(input) 2025-08-26T20:30:45.6677202Z 2025-08-26T20:30:45.6677319Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6677737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6678097Z return mod(**inputs) 2025-08-26T20:30:45.6678506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6678982Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6679507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6679950Z hidden_states = self.encoder( 2025-08-26T20:30:45.6680377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6680808Z layer_outputs = layer_module( 2025-08-26T20:30:45.6681190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6681578Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6682041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6682510Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6682940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6683357Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6683814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6684340Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6684824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6685247Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6685401Z 2025-08-26T20:30:45.6685513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6685901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6686239Z return mod(**inputs) 2025-08-26T20:30:45.6686646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6687076Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6687516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6687933Z hidden_states = self.encoder( 2025-08-26T20:30:45.6688344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6688748Z layer_outputs = layer_module( 2025-08-26T20:30:45.6689089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6689459Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6689861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6690303Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6690686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6691102Z return func(*args, **kwargs) 2025-08-26T20:30:45.6691518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6691951Z self_outputs = self.self( 2025-08-26T20:30:45.6692321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6692693Z return func(*args, **kwargs) 2025-08-26T20:30:45.6693081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6693490Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6693666Z 2025-08-26T20:30:45.6693783Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6694167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6694510Z return mod(**inputs) 2025-08-26T20:30:45.6694918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6695361Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6695803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6696380Z hidden_states = self.encoder( 2025-08-26T20:30:45.6696810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6697232Z layer_outputs = layer_module( 2025-08-26T20:30:45.6697679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6698074Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6698494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6698934Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6699318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6699694Z return func(*args, **kwargs) 2025-08-26T20:30:45.6700080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6700469Z self_outputs = self.self( 2025-08-26T20:30:45.6700837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6701216Z return func(*args, **kwargs) 2025-08-26T20:30:45.6701613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6702032Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6702183Z 2025-08-26T20:30:45.6702299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6702683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6703029Z return mod(**inputs) 2025-08-26T20:30:45.6703426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6703837Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6704274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6704698Z hidden_states = self.encoder( 2025-08-26T20:30:45.6705167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6705592Z layer_outputs = layer_module( 2025-08-26T20:30:45.6705985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6706376Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6706808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6707251Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6707643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6708041Z return func(*args, **kwargs) 2025-08-26T20:30:45.6708450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6708905Z self_outputs = self.self( 2025-08-26T20:30:45.6709311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6709713Z return func(*args, **kwargs) 2025-08-26T20:30:45.6710127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6710556Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6710700Z 2025-08-26T20:30:45.6710795Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6711020Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6711277Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6711664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6712043Z return mod(**inputs) 2025-08-26T20:30:45.6712474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6712921Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6713391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6713837Z hidden_states = self.encoder( 2025-08-26T20:30:45.6714274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6714709Z layer_outputs = layer_module( 2025-08-26T20:30:45.6715101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6715509Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6715956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6716415Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6716849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6717263Z return func(*args, **kwargs) 2025-08-26T20:30:45.6717691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6718192Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6718690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6719138Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6719359Z 2025-08-26T20:30:45.6719479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6719887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6720251Z return mod(**inputs) 2025-08-26T20:30:45.6720705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6721145Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6721601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6722025Z hidden_states = self.encoder( 2025-08-26T20:30:45.6722477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6722896Z layer_outputs = layer_module( 2025-08-26T20:30:45.6723270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6723662Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6724095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6724556Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6724979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6725397Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6725829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6726310Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6726758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6727165Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6727310Z 2025-08-26T20:30:45.6727416Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6727807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6728139Z return mod(**inputs) 2025-08-26T20:30:45.6728512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6728927Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6729342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6729740Z hidden_states = self.encoder( 2025-08-26T20:30:45.6730128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6730517Z layer_outputs = layer_module( 2025-08-26T20:30:45.6730867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6731231Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6731636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6732044Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6732442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6732839Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6733266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6733743Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6734185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6734618Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6735007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6735353Z return self.act(input) 2025-08-26T20:30:45.6735490Z 2025-08-26T20:30:45.6735605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6736228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6736558Z return mod(**inputs) 2025-08-26T20:30:45.6736941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6737362Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6737781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6738179Z hidden_states = self.encoder( 2025-08-26T20:30:45.6738574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6738997Z layer_outputs = layer_module( 2025-08-26T20:30:45.6739350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6739719Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6740119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6740531Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6740936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6741333Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6741757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6742262Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6742744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6743156Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6743295Z 2025-08-26T20:30:45.6743405Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6743763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6744092Z return mod(**inputs) 2025-08-26T20:30:45.6744470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6744887Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6745305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6745708Z hidden_states = self.encoder( 2025-08-26T20:30:45.6746101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6746500Z layer_outputs = layer_module( 2025-08-26T20:30:45.6746848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6747212Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6747606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6748017Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6748411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6748817Z return func(*args, **kwargs) 2025-08-26T20:30:45.6749229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6749663Z self_outputs = self.self( 2025-08-26T20:30:45.6750068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6750436Z return func(*args, **kwargs) 2025-08-26T20:30:45.6750832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6751227Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6751370Z 2025-08-26T20:30:45.6751473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6751839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6752170Z return mod(**inputs) 2025-08-26T20:30:45.6752542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6752962Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6753379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6753820Z hidden_states = self.encoder( 2025-08-26T20:30:45.6754231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6754647Z layer_outputs = layer_module( 2025-08-26T20:30:45.6755019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6755409Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6755838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6756284Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6756696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6757146Z return func(*args, **kwargs) 2025-08-26T20:30:45.6757568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6757990Z self_outputs = self.self( 2025-08-26T20:30:45.6758386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6758797Z return func(*args, **kwargs) 2025-08-26T20:30:45.6759289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6759753Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6759901Z 2025-08-26T20:30:45.6760027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6760418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6760782Z return mod(**inputs) 2025-08-26T20:30:45.6761203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6761656Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6762107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6762531Z hidden_states = self.encoder( 2025-08-26T20:30:45.6762965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6763411Z layer_outputs = layer_module( 2025-08-26T20:30:45.6763801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6764195Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6764650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6765098Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6765543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6765957Z return func(*args, **kwargs) 2025-08-26T20:30:45.6766392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6766831Z self_outputs = self.self( 2025-08-26T20:30:45.6767195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6767575Z return func(*args, **kwargs) 2025-08-26T20:30:45.6767983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6768409Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6768564Z 2025-08-26T20:30:45.6768652Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6768906Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6769163Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6769540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6769898Z return mod(**inputs) 2025-08-26T20:30:45.6770281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6770697Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6771114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6771521Z hidden_states = self.encoder( 2025-08-26T20:30:45.6771935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6772372Z layer_outputs = layer_module( 2025-08-26T20:30:45.6772747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6773128Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6773556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6773988Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6774392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6774789Z return func(*args, **kwargs) 2025-08-26T20:30:45.6775183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6775636Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6776087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6776515Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6776663Z 2025-08-26T20:30:45.6776783Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6777165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6777518Z return mod(**inputs) 2025-08-26T20:30:45.6777919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6778359Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6778796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6779199Z hidden_states = self.encoder( 2025-08-26T20:30:45.6779589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6779986Z layer_outputs = layer_module( 2025-08-26T20:30:45.6780374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6780757Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6781201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6781637Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6782063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6782482Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6782927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6783439Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6783933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6784365Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6784513Z 2025-08-26T20:30:45.6784634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6785013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6785359Z return mod(**inputs) 2025-08-26T20:30:45.6785761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6786202Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6786639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6787104Z hidden_states = self.encoder( 2025-08-26T20:30:45.6787518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6787934Z layer_outputs = layer_module( 2025-08-26T20:30:45.6788305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6788686Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6789111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6789543Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6789967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6790385Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6790826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6791335Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6791825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6792296Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6792819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6793204Z return self.act(input) 2025-08-26T20:30:45.6793337Z 2025-08-26T20:30:45.6793453Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6793868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6794237Z return mod(**inputs) 2025-08-26T20:30:45.6794644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6795104Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6795598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6796036Z hidden_states = self.encoder( 2025-08-26T20:30:45.6796670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6797116Z layer_outputs = layer_module( 2025-08-26T20:30:45.6797503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6797911Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6798355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6798812Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6799314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6799809Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6800283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6800816Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6801323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6801794Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6801955Z 2025-08-26T20:30:45.6802075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6802496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6802857Z return mod(**inputs) 2025-08-26T20:30:45.6803314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6803785Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6804243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6804698Z hidden_states = self.encoder( 2025-08-26T20:30:45.6805133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6805575Z layer_outputs = layer_module( 2025-08-26T20:30:45.6805963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6806374Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6806819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6807268Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6807703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6808120Z return func(*args, **kwargs) 2025-08-26T20:30:45.6808553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6808995Z self_outputs = self.self( 2025-08-26T20:30:45.6809398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6809814Z return func(*args, **kwargs) 2025-08-26T20:30:45.6810245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6810694Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6810848Z 2025-08-26T20:30:45.6810976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6811403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6811767Z return mod(**inputs) 2025-08-26T20:30:45.6812202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6812659Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6813101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6813546Z hidden_states = self.encoder( 2025-08-26T20:30:45.6813966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6814402Z layer_outputs = layer_module( 2025-08-26T20:30:45.6814774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6815194Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6815623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6816054Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6816464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6816852Z return func(*args, **kwargs) 2025-08-26T20:30:45.6817260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6817678Z self_outputs = self.self( 2025-08-26T20:30:45.6818065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6818459Z return func(*args, **kwargs) 2025-08-26T20:30:45.6818862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6819317Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6819466Z 2025-08-26T20:30:45.6819580Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6819968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6820317Z return mod(**inputs) 2025-08-26T20:30:45.6820707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6821147Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6821584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6822007Z hidden_states = self.encoder( 2025-08-26T20:30:45.6822411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6822834Z layer_outputs = layer_module( 2025-08-26T20:30:45.6823208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6823596Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6824024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6824450Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6824857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6825251Z return func(*args, **kwargs) 2025-08-26T20:30:45.6825662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6826081Z self_outputs = self.self( 2025-08-26T20:30:45.6826459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6826880Z return func(*args, **kwargs) 2025-08-26T20:30:45.6827284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6827726Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6827872Z 2025-08-26T20:30:45.6827960Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6828191Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6828445Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6828830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6829175Z return mod(**inputs) 2025-08-26T20:30:45.6829568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6830008Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6830471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6830867Z hidden_states = self.encoder( 2025-08-26T20:30:45.6831249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6831641Z layer_outputs = layer_module( 2025-08-26T20:30:45.6831991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6832366Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6832790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6833215Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6833621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6834040Z return func(*args, **kwargs) 2025-08-26T20:30:45.6834447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6834932Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6835411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6835856Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6836018Z 2025-08-26T20:30:45.6836134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6836532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6836885Z return mod(**inputs) 2025-08-26T20:30:45.6837298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6837758Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6838208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6838639Z hidden_states = self.encoder( 2025-08-26T20:30:45.6839074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6839578Z layer_outputs = layer_module( 2025-08-26T20:30:45.6839966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6840370Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6840794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6841209Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6841652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6842054Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6842497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6842981Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6843420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6843830Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6843976Z 2025-08-26T20:30:45.6844083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6844449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6844775Z return mod(**inputs) 2025-08-26T20:30:45.6845157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6845595Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6846007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6846405Z hidden_states = self.encoder( 2025-08-26T20:30:45.6846787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6847187Z layer_outputs = layer_module( 2025-08-26T20:30:45.6847536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6847901Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6848300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6848728Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6849135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6849532Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6849967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6850438Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6850879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6851319Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6851706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6852075Z return self.act(input) 2025-08-26T20:30:45.6852197Z 2025-08-26T20:30:45.6852311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6852704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6853058Z return mod(**inputs) 2025-08-26T20:30:45.6853462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6853907Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6854321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6854745Z hidden_states = self.encoder( 2025-08-26T20:30:45.6855158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6855581Z layer_outputs = layer_module( 2025-08-26T20:30:45.6855951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6856397Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6856892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6857384Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6857829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6858254Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6858720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6859253Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6859748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6860225Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6860376Z 2025-08-26T20:30:45.6860496Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6860893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6861269Z return mod(**inputs) 2025-08-26T20:30:45.6861680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6862139Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6862567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6862998Z hidden_states = self.encoder( 2025-08-26T20:30:45.6863410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6863851Z layer_outputs = layer_module( 2025-08-26T20:30:45.6864219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6864635Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6865109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6865567Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6866001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6866425Z return func(*args, **kwargs) 2025-08-26T20:30:45.6866835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6867257Z self_outputs = self.self( 2025-08-26T20:30:45.6867647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6868044Z return func(*args, **kwargs) 2025-08-26T20:30:45.6868460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6868897Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6869036Z 2025-08-26T20:30:45.6869149Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6869521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6869861Z return mod(**inputs) 2025-08-26T20:30:45.6870263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6870704Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6871141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6871565Z hidden_states = self.encoder( 2025-08-26T20:30:45.6871997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6872420Z layer_outputs = layer_module( 2025-08-26T20:30:45.6872825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6873217Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6873665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6874124Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6874553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6874962Z return func(*args, **kwargs) 2025-08-26T20:30:45.6875377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6875823Z self_outputs = self.self( 2025-08-26T20:30:45.6876221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6876632Z return func(*args, **kwargs) 2025-08-26T20:30:45.6877056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6877497Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6877645Z 2025-08-26T20:30:45.6877761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6878163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6878521Z return mod(**inputs) 2025-08-26T20:30:45.6878938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6879493Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6879952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6880389Z hidden_states = self.encoder( 2025-08-26T20:30:45.6880824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6881249Z layer_outputs = layer_module( 2025-08-26T20:30:45.6881615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6882004Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6882433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6882866Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6883277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6883670Z return func(*args, **kwargs) 2025-08-26T20:30:45.6884083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6884498Z self_outputs = self.self( 2025-08-26T20:30:45.6884884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6885272Z return func(*args, **kwargs) 2025-08-26T20:30:45.6885677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6886106Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6886247Z 2025-08-26T20:30:45.6886341Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6886571Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6886821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6887235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6887586Z return mod(**inputs) 2025-08-26T20:30:45.6888009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6888448Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6888914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6889395Z hidden_states = self.encoder( 2025-08-26T20:30:45.6889831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6890282Z layer_outputs = layer_module( 2025-08-26T20:30:45.6890656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6891072Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6891525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6891969Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6892391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6892801Z return func(*args, **kwargs) 2025-08-26T20:30:45.6893233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6893716Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6894195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6894680Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6894838Z 2025-08-26T20:30:45.6894951Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6895339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6895691Z return mod(**inputs) 2025-08-26T20:30:45.6896093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6896696Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6897141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6897561Z hidden_states = self.encoder( 2025-08-26T20:30:45.6897973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6898394Z layer_outputs = layer_module( 2025-08-26T20:30:45.6898761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6899153Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6899583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6900020Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6900443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6900868Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6901323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6901827Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6902297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6902725Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6902931Z 2025-08-26T20:30:45.6903047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6903468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6903823Z return mod(**inputs) 2025-08-26T20:30:45.6904229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6904673Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6905121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6905553Z hidden_states = self.encoder( 2025-08-26T20:30:45.6905977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6906435Z layer_outputs = layer_module( 2025-08-26T20:30:45.6906801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6907191Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6907621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6908054Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6908477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6908898Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6909354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6909864Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6910366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6910823Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6911236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6911606Z return self.act(input) 2025-08-26T20:30:45.6911729Z 2025-08-26T20:30:45.6911849Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6912240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6912582Z return mod(**inputs) 2025-08-26T20:30:45.6912984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6913427Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6913865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6914281Z hidden_states = self.encoder( 2025-08-26T20:30:45.6914703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6915132Z layer_outputs = layer_module( 2025-08-26T20:30:45.6915500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6915886Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6916326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6916790Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6917227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6917660Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6918153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6918685Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6919268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6919741Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6919892Z 2025-08-26T20:30:45.6920017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6920435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6920800Z return mod(**inputs) 2025-08-26T20:30:45.6921214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6921671Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6922158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6922609Z hidden_states = self.encoder( 2025-08-26T20:30:45.6923058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6923513Z layer_outputs = layer_module( 2025-08-26T20:30:45.6923903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6924313Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6924758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6925288Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6925723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6926172Z return func(*args, **kwargs) 2025-08-26T20:30:45.6926588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6927023Z self_outputs = self.self( 2025-08-26T20:30:45.6927304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6927383Z return func(*args, **kwargs) 2025-08-26T20:30:45.6927680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6927771Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6927776Z 2025-08-26T20:30:45.6927891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6928123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6928200Z return mod(**inputs) 2025-08-26T20:30:45.6928503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6928600Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6928893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6928978Z hidden_states = self.encoder( 2025-08-26T20:30:45.6929267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6929353Z layer_outputs = layer_module( 2025-08-26T20:30:45.6929596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6929683Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6929979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6930108Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6930389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6930485Z return func(*args, **kwargs) 2025-08-26T20:30:45.6930783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6930862Z self_outputs = self.self( 2025-08-26T20:30:45.6931124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6931210Z return func(*args, **kwargs) 2025-08-26T20:30:45.6931507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6931603Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6931642Z 2025-08-26T20:30:45.6931758Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6931976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6932055Z return mod(**inputs) 2025-08-26T20:30:45.6932338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6932441Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6932720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6932802Z hidden_states = self.encoder( 2025-08-26T20:30:45.6933080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6933154Z layer_outputs = layer_module( 2025-08-26T20:30:45.6933420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6933506Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6933791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6933879Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6934135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6934218Z return func(*args, **kwargs) 2025-08-26T20:30:45.6934500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6934583Z self_outputs = self.self( 2025-08-26T20:30:45.6934838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6934912Z return func(*args, **kwargs) 2025-08-26T20:30:45.6935204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6935289Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6935292Z 2025-08-26T20:30:45.6935388Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6935473Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6935594Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6935806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6935876Z return mod(**inputs) 2025-08-26T20:30:45.6936167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6936260Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6936542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6936622Z hidden_states = self.encoder( 2025-08-26T20:30:45.6936920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6937025Z layer_outputs = layer_module( 2025-08-26T20:30:45.6937263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6937355Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6937656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6937746Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6938013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6938090Z return func(*args, **kwargs) 2025-08-26T20:30:45.6938399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6938538Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6938821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6938911Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6938915Z 2025-08-26T20:30:45.6939027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6939246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6939315Z return mod(**inputs) 2025-08-26T20:30:45.6939604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6939697Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6939995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6940080Z hidden_states = self.encoder( 2025-08-26T20:30:45.6940357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6940441Z layer_outputs = layer_module( 2025-08-26T20:30:45.6940676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6940768Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6941049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6941137Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6941416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6941499Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6941818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6941950Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6942227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6942323Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6942327Z 2025-08-26T20:30:45.6942437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6942657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6942727Z return mod(**inputs) 2025-08-26T20:30:45.6943014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6943108Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6943403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6943491Z hidden_states = self.encoder( 2025-08-26T20:30:45.6943787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6943870Z layer_outputs = layer_module( 2025-08-26T20:30:45.6944107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6944192Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6944479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6944569Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6944852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6944956Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6945275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6945413Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6945693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6945822Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6946057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6946142Z return self.act(input) 2025-08-26T20:30:45.6946146Z 2025-08-26T20:30:45.6946262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6946501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6946597Z return mod(**inputs) 2025-08-26T20:30:45.6946880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6946982Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6947260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6947336Z hidden_states = self.encoder( 2025-08-26T20:30:45.6947622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6947696Z layer_outputs = layer_module( 2025-08-26T20:30:45.6947941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6948029Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6948320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6948408Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6948687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6948776Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6949091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6949241Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6949543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6949632Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6949646Z 2025-08-26T20:30:45.6949760Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6949992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6950074Z return mod(**inputs) 2025-08-26T20:30:45.6950384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6950486Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6950767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6950843Z hidden_states = self.encoder( 2025-08-26T20:30:45.6951126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6951203Z layer_outputs = layer_module( 2025-08-26T20:30:45.6951448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6951554Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6951834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6951930Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6952189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6952270Z return func(*args, **kwargs) 2025-08-26T20:30:45.6952552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6952636Z self_outputs = self.self( 2025-08-26T20:30:45.6952895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6953009Z return func(*args, **kwargs) 2025-08-26T20:30:45.6953300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6953389Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6953393Z 2025-08-26T20:30:45.6953510Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6953722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6953792Z return mod(**inputs) 2025-08-26T20:30:45.6954082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6954176Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6954463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6954540Z hidden_states = self.encoder( 2025-08-26T20:30:45.6954822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6954912Z layer_outputs = layer_module( 2025-08-26T20:30:45.6955159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6955253Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6955539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6955639Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6955904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6955979Z return func(*args, **kwargs) 2025-08-26T20:30:45.6956272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6956352Z self_outputs = self.self( 2025-08-26T20:30:45.6956645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6956721Z return func(*args, **kwargs) 2025-08-26T20:30:45.6958191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6958303Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6958307Z 2025-08-26T20:30:45.6958422Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6958648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6958720Z return mod(**inputs) 2025-08-26T20:30:45.6959022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6959117Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6959487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6959612Z hidden_states = self.encoder( 2025-08-26T20:30:45.6959905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6959991Z layer_outputs = layer_module( 2025-08-26T20:30:45.6960236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6960323Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6960620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6960709Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6960984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6961084Z return func(*args, **kwargs) 2025-08-26T20:30:45.6961374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6961463Z self_outputs = self.self( 2025-08-26T20:30:45.6961731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6961815Z return func(*args, **kwargs) 2025-08-26T20:30:45.6962105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6962201Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6962205Z 2025-08-26T20:30:45.6962294Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6962381Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6962504Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6962725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6962805Z return mod(**inputs) 2025-08-26T20:30:45.6963098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6963197Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6963490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6963569Z hidden_states = self.encoder( 2025-08-26T20:30:45.6963862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6963941Z layer_outputs = layer_module( 2025-08-26T20:30:45.6964187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6964281Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6964595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6964695Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6964984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6965069Z return func(*args, **kwargs) 2025-08-26T20:30:45.6965356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6965499Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6965794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6965886Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6965890Z 2025-08-26T20:30:45.6966013Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6966254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6966328Z return mod(**inputs) 2025-08-26T20:30:45.6966634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6966732Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6967031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6967113Z hidden_states = self.encoder( 2025-08-26T20:30:45.6967411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6967493Z layer_outputs = layer_module( 2025-08-26T20:30:45.6967743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6967857Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6968146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6968237Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6968499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6968576Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6968892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6969023Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6969308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6969398Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6969403Z 2025-08-26T20:30:45.6969521Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6969734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6969803Z return mod(**inputs) 2025-08-26T20:30:45.6970095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6970187Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6970469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6970545Z hidden_states = self.encoder( 2025-08-26T20:30:45.6970821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6970903Z layer_outputs = layer_module( 2025-08-26T20:30:45.6971140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6971250Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6971531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6971634Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6971919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6972000Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6972317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6972447Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6972740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6972887Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6973134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6973219Z return self.act(input) 2025-08-26T20:30:45.6973223Z 2025-08-26T20:30:45.6973338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6973557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6973625Z return mod(**inputs) 2025-08-26T20:30:45.6973910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6974010Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6974290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6974406Z hidden_states = self.encoder( 2025-08-26T20:30:45.6974686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6974767Z layer_outputs = layer_module( 2025-08-26T20:30:45.6975004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6975090Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6975376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6975464Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6975744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6975824Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6976136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.6976292Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.6976573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.6976668Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6976672Z 2025-08-26T20:30:45.6976783Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6977003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6977074Z return mod(**inputs) 2025-08-26T20:30:45.6977357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6977456Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6977731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6977839Z hidden_states = self.encoder( 2025-08-26T20:30:45.6978118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6978212Z layer_outputs = layer_module( 2025-08-26T20:30:45.6978457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6978544Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6978828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6978916Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6979183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6979261Z return func(*args, **kwargs) 2025-08-26T20:30:45.6979568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6979654Z self_outputs = self.self( 2025-08-26T20:30:45.6979910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6979992Z return func(*args, **kwargs) 2025-08-26T20:30:45.6980269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.6980356Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.6980360Z 2025-08-26T20:30:45.6980479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6980689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6980764Z return mod(**inputs) 2025-08-26T20:30:45.6981066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6981161Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6981443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6981520Z hidden_states = self.encoder( 2025-08-26T20:30:45.6981805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6981879Z layer_outputs = layer_module( 2025-08-26T20:30:45.6982119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6982204Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6982492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6982589Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6982846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6982936Z return func(*args, **kwargs) 2025-08-26T20:30:45.6983215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6983289Z self_outputs = self.self( 2025-08-26T20:30:45.6983554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6983626Z return func(*args, **kwargs) 2025-08-26T20:30:45.6983910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.6983994Z key_layer = self.key(current_states) 2025-08-26T20:30:45.6983998Z 2025-08-26T20:30:45.6984114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6984328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6984416Z return mod(**inputs) 2025-08-26T20:30:45.6984707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6984816Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6985102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6985178Z hidden_states = self.encoder( 2025-08-26T20:30:45.6985456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6985541Z layer_outputs = layer_module( 2025-08-26T20:30:45.6985777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6985869Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6986167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6986256Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6986532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6986608Z return func(*args, **kwargs) 2025-08-26T20:30:45.6986897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.6986972Z self_outputs = self.self( 2025-08-26T20:30:45.6987245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6987318Z return func(*args, **kwargs) 2025-08-26T20:30:45.6987596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.6987713Z value_layer = self.value(current_states) 2025-08-26T20:30:45.6987717Z 2025-08-26T20:30:45.6987803Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6987894Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.6988007Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6988221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6988297Z return mod(**inputs) 2025-08-26T20:30:45.6988578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6988677Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6988957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6989035Z hidden_states = self.encoder( 2025-08-26T20:30:45.6989326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6989401Z layer_outputs = layer_module( 2025-08-26T20:30:45.6989649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6989736Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6990036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.6990124Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.6990393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.6990472Z return func(*args, **kwargs) 2025-08-26T20:30:45.6990751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.6990896Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.6991192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.6991304Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6991308Z 2025-08-26T20:30:45.6991427Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6991637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6991716Z return mod(**inputs) 2025-08-26T20:30:45.6991997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6992097Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6992375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6992473Z hidden_states = self.encoder( 2025-08-26T20:30:45.6992760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6992837Z layer_outputs = layer_module( 2025-08-26T20:30:45.6993081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6993166Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6993444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6993543Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6993816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6993906Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6994238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6994369Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6994663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.6994752Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.6994756Z 2025-08-26T20:30:45.6994874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6995087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6995165Z return mod(**inputs) 2025-08-26T20:30:45.6995452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6995544Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.6995838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.6995919Z hidden_states = self.encoder( 2025-08-26T20:30:45.6996355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.6996446Z layer_outputs = layer_module( 2025-08-26T20:30:45.6996695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.6996791Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.6997096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.6997196Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.6997480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.6997575Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.6997950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.6998086Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.6998403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.6998531Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.6998774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.6998853Z return self.act(input) 2025-08-26T20:30:45.6998858Z 2025-08-26T20:30:45.6998973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.6999203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.6999356Z return mod(**inputs) 2025-08-26T20:30:45.6999697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.6999793Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7000092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7000171Z hidden_states = self.encoder( 2025-08-26T20:30:45.7000461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7000555Z layer_outputs = layer_module( 2025-08-26T20:30:45.7000797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7000891Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7001176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.7001306Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.7001602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.7001687Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.7002019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.7002169Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.7002464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.7002554Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.7002558Z 2025-08-26T20:30:45.7002672Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7002901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7002974Z return mod(**inputs) 2025-08-26T20:30:45.7003269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7003366Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7003649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7003734Z hidden_states = self.encoder( 2025-08-26T20:30:45.7004018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7004103Z layer_outputs = layer_module( 2025-08-26T20:30:45.7004346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7004434Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7004749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7004841Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7005146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7005224Z return func(*args, **kwargs) 2025-08-26T20:30:45.7005518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.7005597Z self_outputs = self.self( 2025-08-26T20:30:45.7005863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7005947Z return func(*args, **kwargs) 2025-08-26T20:30:45.7006235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.7006353Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.7006357Z 2025-08-26T20:30:45.7006474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7006694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7006773Z return mod(**inputs) 2025-08-26T20:30:45.7007064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7007165Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7007457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7007541Z hidden_states = self.encoder( 2025-08-26T20:30:45.7007828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7007931Z layer_outputs = layer_module( 2025-08-26T20:30:45.7008191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7008277Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7008573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7008664Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7008933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7009017Z return func(*args, **kwargs) 2025-08-26T20:30:45.7009310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.7009394Z self_outputs = self.self( 2025-08-26T20:30:45.7009660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7009741Z return func(*args, **kwargs) 2025-08-26T20:30:45.7010044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.7010133Z key_layer = self.key(current_states) 2025-08-26T20:30:45.7010139Z 2025-08-26T20:30:45.7010261Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7010482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7010560Z return mod(**inputs) 2025-08-26T20:30:45.7010855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7010950Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7011248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7011330Z hidden_states = self.encoder( 2025-08-26T20:30:45.7011653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7011734Z layer_outputs = layer_module( 2025-08-26T20:30:45.7012000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7012094Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7012375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7012470Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7012726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7012809Z return func(*args, **kwargs) 2025-08-26T20:30:45.7013094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.7013195Z self_outputs = self.self( 2025-08-26T20:30:45.7013464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7013539Z return func(*args, **kwargs) 2025-08-26T20:30:45.7013830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.7013916Z value_layer = self.value(current_states) 2025-08-26T20:30:45.7013920Z 2025-08-26T20:30:45.7014006Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.7014100Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.7014212Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7014432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7014522Z return mod(**inputs) 2025-08-26T20:30:45.7014814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7014915Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7015197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7015280Z hidden_states = self.encoder( 2025-08-26T20:30:45.7015563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7015647Z layer_outputs = layer_module( 2025-08-26T20:30:45.7015887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7015971Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7016257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7016347Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7016616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7016689Z return func(*args, **kwargs) 2025-08-26T20:30:45.7016974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.7017116Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.7017398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.7017494Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.7017497Z 2025-08-26T20:30:45.7017607Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7017830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7017903Z return mod(**inputs) 2025-08-26T20:30:45.7018213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7018315Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7018612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7018696Z hidden_states = self.encoder( 2025-08-26T20:30:45.7018984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7019060Z layer_outputs = layer_module( 2025-08-26T20:30:45.7019313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7019395Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7019686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.7019824Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.7020099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.7020189Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.7020502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.7020639Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.7020924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.7021014Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.7021018Z 2025-08-26T20:30:45.7021122Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7021342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7021422Z return mod(**inputs) 2025-08-26T20:30:45.7021688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7021785Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7022049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7022126Z hidden_states = self.encoder( 2025-08-26T20:30:45.7022412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7022487Z layer_outputs = layer_module( 2025-08-26T20:30:45.7022740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7022819Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7023097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.7023180Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.7023443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.7023527Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.7023826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.7023954Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.7024223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.7024337Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.7024566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.7024662Z return self.act(input) 2025-08-26T20:30:45.7024667Z 2025-08-26T20:30:45.7024787Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7025023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7025102Z return mod(**inputs) 2025-08-26T20:30:45.7025387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7025480Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7025766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7025842Z hidden_states = self.encoder( 2025-08-26T20:30:45.7026125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7026221Z layer_outputs = layer_module( 2025-08-26T20:30:45.7026467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7026559Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7026849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.7026945Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.7027230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.7027311Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.7027643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.7027808Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.7028082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.7028164Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.7028168Z 2025-08-26T20:30:45.7028285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7028497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7028567Z return mod(**inputs) 2025-08-26T20:30:45.7028859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7028953Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7029233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7029311Z hidden_states = self.encoder( 2025-08-26T20:30:45.7029594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7029678Z layer_outputs = layer_module( 2025-08-26T20:30:45.7029917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7030010Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7030296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7030392Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7030658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7030731Z return func(*args, **kwargs) 2025-08-26T20:30:45.7031016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.7031095Z self_outputs = self.self( 2025-08-26T20:30:45.7031390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7031465Z return func(*args, **kwargs) 2025-08-26T20:30:45.7031766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-26T20:30:45.7031863Z query_layer = self.query(hidden_states) 2025-08-26T20:30:45.7031867Z 2025-08-26T20:30:45.7031976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7032200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7032270Z return mod(**inputs) 2025-08-26T20:30:45.7032559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7032652Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7032948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7033031Z hidden_states = self.encoder( 2025-08-26T20:30:45.7033308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7033389Z layer_outputs = layer_module( 2025-08-26T20:30:45.7033623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7033707Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7033996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7034084Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7034350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7034449Z return func(*args, **kwargs) 2025-08-26T20:30:45.7034731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.7034814Z self_outputs = self.self( 2025-08-26T20:30:45.7035074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7035159Z return func(*args, **kwargs) 2025-08-26T20:30:45.7035444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-26T20:30:45.7035539Z key_layer = self.key(current_states) 2025-08-26T20:30:45.7035543Z 2025-08-26T20:30:45.7035657Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7035875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7035960Z return mod(**inputs) 2025-08-26T20:30:45.7036251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7036356Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7036642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7036721Z hidden_states = self.encoder( 2025-08-26T20:30:45.7037012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7037088Z layer_outputs = layer_module( 2025-08-26T20:30:45.7037336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7037423Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7037714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7037821Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7038085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7038189Z return func(*args, **kwargs) 2025-08-26T20:30:45.7038481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-26T20:30:45.7038563Z self_outputs = self.self( 2025-08-26T20:30:45.7038816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7038890Z return func(*args, **kwargs) 2025-08-26T20:30:45.7039178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-26T20:30:45.7039338Z value_layer = self.value(current_states) 2025-08-26T20:30:45.7039393Z 2025-08-26T20:30:45.7039498Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.7039587Z cudagraph partition due to non gpu ops 2025-08-26T20:30:45.7039703Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7039939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7040013Z return mod(**inputs) 2025-08-26T20:30:45.7040312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7040410Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7040707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7040784Z hidden_states = self.encoder( 2025-08-26T20:30:45.7041070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7041181Z layer_outputs = layer_module( 2025-08-26T20:30:45.7041427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7041522Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7041819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-26T20:30:45.7041907Z self_attention_outputs = self.attention( 2025-08-26T20:30:45.7042173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:30:45.7042247Z return func(*args, **kwargs) 2025-08-26T20:30:45.7042531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-26T20:30:45.7042667Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:30:45.7042949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-26T20:30:45.7043047Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.7043050Z 2025-08-26T20:30:45.7043159Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7043380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7043451Z return mod(**inputs) 2025-08-26T20:30:45.7043739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7043830Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7044108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7044193Z hidden_states = self.encoder( 2025-08-26T20:30:45.7044472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7044577Z layer_outputs = layer_module( 2025-08-26T20:30:45.7044814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7044916Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7045207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.7045296Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.7045592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.7045675Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.7046013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.7046165Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.7046446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-26T20:30:45.7046543Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.7046547Z 2025-08-26T20:30:45.7046658Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7046883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7046953Z return mod(**inputs) 2025-08-26T20:30:45.7047233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7047335Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7047612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7047715Z hidden_states = self.encoder( 2025-08-26T20:30:45.7048014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7048098Z layer_outputs = layer_module( 2025-08-26T20:30:45.7048342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7048424Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7048712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.7048799Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.7049081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.7049161Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.7049475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-26T20:30:45.7049617Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:30:45.7049892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-26T20:30:45.7050021Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:30:45.7050251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:30:45.7050334Z return self.act(input) 2025-08-26T20:30:45.7050338Z 2025-08-26T20:30:45.7050447Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7050661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7050754Z return mod(**inputs) 2025-08-26T20:30:45.7051035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-26T20:30:45.7051136Z discriminator_hidden_states = self.electra( 2025-08-26T20:30:45.7051433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-26T20:30:45.7051529Z hidden_states = self.encoder( 2025-08-26T20:30:45.7051817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-26T20:30:45.7051893Z layer_outputs = layer_module( 2025-08-26T20:30:45.7052145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:30:45.7052227Z return super().__call__(*args, **kwargs) 2025-08-26T20:30:45.7052492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-26T20:30:45.7052585Z layer_output = apply_chunking_to_forward( 2025-08-26T20:30:45.7052866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:30:45.7052948Z return forward_fn(*input_tensors) 2025-08-26T20:30:45.7053266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-26T20:30:45.7053413Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:30:45.7053691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-26T20:30:45.7053777Z hidden_states = self.dense(hidden_states) 2025-08-26T20:30:45.7053781Z 2025-08-26T20:30:45.7053900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7054110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7054216Z return mod(**inputs) 2025-08-26T20:30:45.7054507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1330, in forward 2025-08-26T20:30:45.7054596Z logits = self.qa_outputs(sequence_output) 2025-08-26T20:30:45.7054607Z 2025-08-26T20:30:45.7054717Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7054938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7055012Z return mod(**inputs) 2025-08-26T20:30:45.7055398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1348, in forward 2025-08-26T20:30:45.7055517Z start_loss = loss_fct(start_logits, start_positions) 2025-08-26T20:30:45.7055521Z 2025-08-26T20:30:45.7055623Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:30:45.7055823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:30:45.7055900Z return mod(**inputs) 2025-08-26T20:30:45.7056176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1349, in forward 2025-08-26T20:30:45.7056287Z end_loss = loss_fct(end_logits, end_positions) 2025-08-26T20:30:45.7056291Z 2025-08-26T20:30:53.4789195Z Compilation time (from dynamo_timed): 15.236378524 2025-08-26T20:30:53.4791152Z pass 2025-08-26T20:30:53.4791530Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:30:53.4792431Z TIMING: _recursive_pre_grad_passes:0.0075 _recursive_joint_graph_passes:0.48068 _recursive_post_grad_passes:0.08614 async_compile.wait:0.00242 code_gen:7.23814 inductor_compile:8.55913 backend_compile:12.30464 gc:0.00122 entire_frame_compile:15.23638 total_wall_time:15.23638 2025-08-26T20:30:53.4793503Z STATS: call_* op count: 378 | FakeTensorMode.__torch_dispatch__:15000 | FakeTensor.__torch_dispatch__:4378 | ProxyTorchDispatchMode.__torch_dispatch__:5698 2025-08-26T20:30:53.4794065Z Dynamo produced 1 graphs covering 378 ops with 0 graph breaks (0 unique) 2025-08-26T20:30:58.9567215Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:30:58.9568463Z from pkg_resources import resource_filename 2025-08-26T20:30:59.5427073Z 2025-08-26T20:31:01.1335881Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:31:01.1338973Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:31:01.1347421Z cpu eval GPT2ForSequenceClassification 2025-08-26T20:31:01.9782506Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:02.3217382Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:02.7498346Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:09.9577056Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9577421Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9577659Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9577936Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9578166Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9578390Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9578616Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9578858Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9579095Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9579331Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9579564Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9579801Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9580686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9581325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9581841Z return mod(**inputs) 2025-08-26T20:31:09.9582305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1509, in forward 2025-08-26T20:31:09.9582808Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-08-26T20:31:09.9583024Z 2025-08-26T20:31:09.9583155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9583575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9583934Z return mod(**inputs) 2025-08-26T20:31:09.9584353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9584803Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9585250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9585675Z outputs = block( 2025-08-26T20:31:09.9586045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9586466Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9586923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9587469Z return func(*args, **kwargs) 2025-08-26T20:31:09.9587879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9588323Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9588769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9589236Z return func(*args, **kwargs) 2025-08-26T20:31:09.9589731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:09.9590294Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:09.9590880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9591341Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9591543Z 2025-08-26T20:31:09.9591644Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9591876Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9592110Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9592337Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9592612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9593018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9593440Z return mod(**inputs) 2025-08-26T20:31:09.9593850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9594298Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9594743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9595153Z outputs = block( 2025-08-26T20:31:09.9595539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9596140Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9597032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9597461Z return func(*args, **kwargs) 2025-08-26T20:31:09.9597868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9598583Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9599423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9599902Z return func(*args, **kwargs) 2025-08-26T20:31:09.9600493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9601198Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9601718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:09.9602257Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:09.9602468Z 2025-08-26T20:31:09.9602612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9603227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9603699Z return mod(**inputs) 2025-08-26T20:31:09.9604104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9604542Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9604968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9605364Z outputs = block( 2025-08-26T20:31:09.9605833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9606463Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9606889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9607293Z return func(*args, **kwargs) 2025-08-26T20:31:09.9607712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9608207Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9608827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9609484Z return func(*args, **kwargs) 2025-08-26T20:31:09.9609969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9610437Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9611108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:09.9611779Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:09.9611960Z 2025-08-26T20:31:09.9612087Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9612488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9612898Z return mod(**inputs) 2025-08-26T20:31:09.9613315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9613749Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9614168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9614620Z outputs = block( 2025-08-26T20:31:09.9614995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9615416Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9616020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9616633Z return func(*args, **kwargs) 2025-08-26T20:31:09.9617127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9617566Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9617989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9618393Z return func(*args, **kwargs) 2025-08-26T20:31:09.9618797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:09.9619231Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:09.9619609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9620056Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9620258Z 2025-08-26T20:31:09.9620385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9620987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9621437Z return mod(**inputs) 2025-08-26T20:31:09.9621822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9622254Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9622823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9623456Z outputs = block( 2025-08-26T20:31:09.9623809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9624200Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9624614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9625026Z return func(*args, **kwargs) 2025-08-26T20:31:09.9625453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9625901Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9626369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:09.9626798Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:09.9627195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9627637Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9627831Z 2025-08-26T20:31:09.9627950Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9628352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9628715Z return mod(**inputs) 2025-08-26T20:31:09.9629116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9629574Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9629990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9630401Z outputs = block( 2025-08-26T20:31:09.9630752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9631157Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9631565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9631982Z return func(*args, **kwargs) 2025-08-26T20:31:09.9632381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9632852Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9633300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:09.9633717Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:09.9634092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:09.9634579Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:09.9634829Z 2025-08-26T20:31:09.9634946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9635332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9635672Z return mod(**inputs) 2025-08-26T20:31:09.9636054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9636477Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9636902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9637303Z outputs = block( 2025-08-26T20:31:09.9637660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9638058Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9638476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9638901Z return func(*args, **kwargs) 2025-08-26T20:31:09.9639473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9639950Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9640407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:09.9640851Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:09.9641266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9641742Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9641938Z 2025-08-26T20:31:09.9642051Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9642438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9642802Z return mod(**inputs) 2025-08-26T20:31:09.9643179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9643614Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9644025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9644453Z outputs = block( 2025-08-26T20:31:09.9644797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9645275Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9645699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9646106Z return func(*args, **kwargs) 2025-08-26T20:31:09.9646504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9646938Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9647364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9647769Z return func(*args, **kwargs) 2025-08-26T20:31:09.9648166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:09.9648740Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:09.9649232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9649668Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9649859Z 2025-08-26T20:31:09.9649950Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9650186Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9650416Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9650632Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9650886Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9651273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9651621Z return mod(**inputs) 2025-08-26T20:31:09.9652002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9652433Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9652851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9653252Z outputs = block( 2025-08-26T20:31:09.9653591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9653986Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9654387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9654795Z return func(*args, **kwargs) 2025-08-26T20:31:09.9655191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9655612Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9656024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9656407Z return func(*args, **kwargs) 2025-08-26T20:31:09.9657098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9657537Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9658006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:09.9658514Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:09.9658709Z 2025-08-26T20:31:09.9658817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9659186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9659516Z return mod(**inputs) 2025-08-26T20:31:09.9659890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9660309Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9660723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9661113Z outputs = block( 2025-08-26T20:31:09.9661447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9661840Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9662217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9662589Z return func(*args, **kwargs) 2025-08-26T20:31:09.9662960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9663374Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9663766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9664137Z return func(*args, **kwargs) 2025-08-26T20:31:09.9664511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9664918Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9665355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:09.9665818Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:09.9665989Z 2025-08-26T20:31:09.9666095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9666459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9666787Z return mod(**inputs) 2025-08-26T20:31:09.9667175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9667595Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9668013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9668407Z outputs = block( 2025-08-26T20:31:09.9668739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9669123Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9669522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9669918Z return func(*args, **kwargs) 2025-08-26T20:31:09.9670302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9670706Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9671114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9671489Z return func(*args, **kwargs) 2025-08-26T20:31:09.9671875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:09.9672281Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:09.9672667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9673093Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9673277Z 2025-08-26T20:31:09.9673398Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9673788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9674156Z return mod(**inputs) 2025-08-26T20:31:09.9674548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9674975Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9675402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9675804Z outputs = block( 2025-08-26T20:31:09.9676161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9676559Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9676979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9677390Z return func(*args, **kwargs) 2025-08-26T20:31:09.9677788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9678265Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9678717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:09.9679144Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:09.9679643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9680086Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9680285Z 2025-08-26T20:31:09.9680401Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9680807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9681160Z return mod(**inputs) 2025-08-26T20:31:09.9681539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9681969Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9682391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9682795Z outputs = block( 2025-08-26T20:31:09.9683146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9683529Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9683938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9684339Z return func(*args, **kwargs) 2025-08-26T20:31:09.9684738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9685176Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9685609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:09.9686046Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:09.9686427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:09.9686935Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:09.9687187Z 2025-08-26T20:31:09.9687328Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9687710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9688057Z return mod(**inputs) 2025-08-26T20:31:09.9688445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9688869Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9689280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9689699Z outputs = block( 2025-08-26T20:31:09.9690045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9690433Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9690835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9691247Z return func(*args, **kwargs) 2025-08-26T20:31:09.9691617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9692034Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9692443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:09.9692882Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:09.9693278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9693705Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9693889Z 2025-08-26T20:31:09.9694011Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9694397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9694739Z return mod(**inputs) 2025-08-26T20:31:09.9695120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9695539Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9695954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9696536Z outputs = block( 2025-08-26T20:31:09.9696882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9697275Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9697685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9698135Z return func(*args, **kwargs) 2025-08-26T20:31:09.9698521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9698961Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9699376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9699784Z return func(*args, **kwargs) 2025-08-26T20:31:09.9700177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:09.9700706Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:09.9701305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9701750Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9701967Z 2025-08-26T20:31:09.9702069Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9702314Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9702555Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9702781Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9703037Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9703429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9703775Z return mod(**inputs) 2025-08-26T20:31:09.9704161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9704623Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9705046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9705438Z outputs = block( 2025-08-26T20:31:09.9705783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9706172Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9706578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9706981Z return func(*args, **kwargs) 2025-08-26T20:31:09.9707368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9707790Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9709037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9709442Z return func(*args, **kwargs) 2025-08-26T20:31:09.9709833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9710260Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9710735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:09.9711247Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:09.9711444Z 2025-08-26T20:31:09.9711564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9711945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9712295Z return mod(**inputs) 2025-08-26T20:31:09.9712677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9713105Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9713528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9713928Z outputs = block( 2025-08-26T20:31:09.9714284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9714686Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9715096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9715505Z return func(*args, **kwargs) 2025-08-26T20:31:09.9715899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9716332Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9716757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9717192Z return func(*args, **kwargs) 2025-08-26T20:31:09.9717595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9718064Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9718558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:09.9719066Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:09.9719335Z 2025-08-26T20:31:09.9719468Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9719867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9720231Z return mod(**inputs) 2025-08-26T20:31:09.9720702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9721184Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9721615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9722024Z outputs = block( 2025-08-26T20:31:09.9722380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9722785Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9723203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9723609Z return func(*args, **kwargs) 2025-08-26T20:31:09.9724019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9724480Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9724913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9725321Z return func(*args, **kwargs) 2025-08-26T20:31:09.9725723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:09.9726158Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:09.9726557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9726995Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9727187Z 2025-08-26T20:31:09.9727310Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9727709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9728080Z return mod(**inputs) 2025-08-26T20:31:09.9728481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9728924Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9729349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9729762Z outputs = block( 2025-08-26T20:31:09.9730123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9730525Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9730959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9731355Z return func(*args, **kwargs) 2025-08-26T20:31:09.9731750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9732191Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9732658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:09.9733071Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:09.9733472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9733896Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9734078Z 2025-08-26T20:31:09.9734199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9734589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9734931Z return mod(**inputs) 2025-08-26T20:31:09.9735321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9735740Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9736179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9736576Z outputs = block( 2025-08-26T20:31:09.9736915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9737305Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9737709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9738131Z return func(*args, **kwargs) 2025-08-26T20:31:09.9738514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9738977Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9739411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:09.9739857Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:09.9740236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:09.9740711Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:09.9740968Z 2025-08-26T20:31:09.9741082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9741471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9741825Z return mod(**inputs) 2025-08-26T20:31:09.9742211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9742626Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9743040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9743434Z outputs = block( 2025-08-26T20:31:09.9743777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9744157Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9744572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9744973Z return func(*args, **kwargs) 2025-08-26T20:31:09.9745384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9745816Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9746253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:09.9746743Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:09.9747130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9747581Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9747770Z 2025-08-26T20:31:09.9747891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9748300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9748675Z return mod(**inputs) 2025-08-26T20:31:09.9749067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9749500Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9749928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9750357Z outputs = block( 2025-08-26T20:31:09.9750708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9751114Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9751527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9751927Z return func(*args, **kwargs) 2025-08-26T20:31:09.9752331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-26T20:31:09.9752783Z hidden_states = residual + feed_forward_hidden_states 2025-08-26T20:31:09.9752957Z 2025-08-26T20:31:09.9753080Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9753476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9753834Z return mod(**inputs) 2025-08-26T20:31:09.9754217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9754680Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9755105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9755504Z outputs = block( 2025-08-26T20:31:09.9755861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9756259Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9756672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9757082Z return func(*args, **kwargs) 2025-08-26T20:31:09.9757480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9757923Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9758353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9758765Z return func(*args, **kwargs) 2025-08-26T20:31:09.9759159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:09.9759804Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:09.9760324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9760781Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9760972Z 2025-08-26T20:31:09.9761073Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9761307Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9761544Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9761794Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9762058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9762457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9762852Z return mod(**inputs) 2025-08-26T20:31:09.9763254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9763711Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9764137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9764537Z outputs = block( 2025-08-26T20:31:09.9764893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9765295Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9765712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9766115Z return func(*args, **kwargs) 2025-08-26T20:31:09.9766522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9766992Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9767429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9767840Z return func(*args, **kwargs) 2025-08-26T20:31:09.9768240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9768690Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9769186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:09.9769725Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:09.9769924Z 2025-08-26T20:31:09.9770044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9770457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9770817Z return mod(**inputs) 2025-08-26T20:31:09.9771216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9771665Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9772079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9772469Z outputs = block( 2025-08-26T20:31:09.9772809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9773199Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9773608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9774011Z return func(*args, **kwargs) 2025-08-26T20:31:09.9774419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9774855Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9775284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9775695Z return func(*args, **kwargs) 2025-08-26T20:31:09.9776094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9776526Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9776999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:09.9777493Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:09.9777671Z 2025-08-26T20:31:09.9777791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9778198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9778551Z return mod(**inputs) 2025-08-26T20:31:09.9778958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9779386Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9779796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9780198Z outputs = block( 2025-08-26T20:31:09.9780544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9780938Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9781338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9781752Z return func(*args, **kwargs) 2025-08-26T20:31:09.9782148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9782574Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9782987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9783385Z return func(*args, **kwargs) 2025-08-26T20:31:09.9783792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:09.9784230Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:09.9784629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9785076Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9785293Z 2025-08-26T20:31:09.9785406Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9785797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9786150Z return mod(**inputs) 2025-08-26T20:31:09.9786536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9786961Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9787382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9787791Z outputs = block( 2025-08-26T20:31:09.9788148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9788549Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9788958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9789369Z return func(*args, **kwargs) 2025-08-26T20:31:09.9789776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9790225Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9790673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:09.9791094Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:09.9791489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9791926Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9792115Z 2025-08-26T20:31:09.9792237Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9792638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9792994Z return mod(**inputs) 2025-08-26T20:31:09.9793423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9793873Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9794370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9794781Z outputs = block( 2025-08-26T20:31:09.9795140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9795539Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9795960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9796535Z return func(*args, **kwargs) 2025-08-26T20:31:09.9796938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9797452Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9797906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:09.9798338Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:09.9798722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:09.9799283Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:09.9799559Z 2025-08-26T20:31:09.9799679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9800082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9800440Z return mod(**inputs) 2025-08-26T20:31:09.9800829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9801312Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9801744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9802154Z outputs = block( 2025-08-26T20:31:09.9802511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9802905Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9803325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9803736Z return func(*args, **kwargs) 2025-08-26T20:31:09.9804146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9804590Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9805040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:09.9805481Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:09.9805880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9806320Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9806510Z 2025-08-26T20:31:09.9806624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9807023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9807385Z return mod(**inputs) 2025-08-26T20:31:09.9807790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9808213Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9808619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9809014Z outputs = block( 2025-08-26T20:31:09.9809384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9809771Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9810193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9810597Z return func(*args, **kwargs) 2025-08-26T20:31:09.9810994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9811423Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9811842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9812236Z return func(*args, **kwargs) 2025-08-26T20:31:09.9812636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:09.9813208Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:09.9813709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9814130Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9814313Z 2025-08-26T20:31:09.9814400Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9814632Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9814858Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9815083Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9815329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9815715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9816086Z return mod(**inputs) 2025-08-26T20:31:09.9816479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9816903Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9817310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9817709Z outputs = block( 2025-08-26T20:31:09.9818056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9818442Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9818842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9819239Z return func(*args, **kwargs) 2025-08-26T20:31:09.9819632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9820061Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9820476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9820866Z return func(*args, **kwargs) 2025-08-26T20:31:09.9821263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9821700Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9822176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:09.9822688Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:09.9822893Z 2025-08-26T20:31:09.9823006Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9823393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9823746Z return mod(**inputs) 2025-08-26T20:31:09.9824162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9824596Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9825073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9825474Z outputs = block( 2025-08-26T20:31:09.9825817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9826204Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9826604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9827008Z return func(*args, **kwargs) 2025-08-26T20:31:09.9827404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9827859Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9828268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9828668Z return func(*args, **kwargs) 2025-08-26T20:31:09.9829062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9829495Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9829972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:09.9830453Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:09.9830637Z 2025-08-26T20:31:09.9830748Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9831160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9831513Z return mod(**inputs) 2025-08-26T20:31:09.9831901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9832326Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9832750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9833164Z outputs = block( 2025-08-26T20:31:09.9833527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9833909Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9834328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9834736Z return func(*args, **kwargs) 2025-08-26T20:31:09.9835140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9835581Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9835999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9836425Z return func(*args, **kwargs) 2025-08-26T20:31:09.9836830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:09.9837261Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:09.9837660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9838093Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9838289Z 2025-08-26T20:31:09.9838405Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9838806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9839166Z return mod(**inputs) 2025-08-26T20:31:09.9839675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9840140Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9840579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9841003Z outputs = block( 2025-08-26T20:31:09.9841363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9841755Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9842171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9842576Z return func(*args, **kwargs) 2025-08-26T20:31:09.9842982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9843449Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9843903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:09.9844332Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:09.9844722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9845169Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9845358Z 2025-08-26T20:31:09.9845472Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9845877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9846238Z return mod(**inputs) 2025-08-26T20:31:09.9846666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9847108Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9847539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9847965Z outputs = block( 2025-08-26T20:31:09.9848323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9848729Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9849141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9849553Z return func(*args, **kwargs) 2025-08-26T20:31:09.9849963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9850405Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9850845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:09.9851256Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:09.9851634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:09.9852130Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:09.9852377Z 2025-08-26T20:31:09.9852499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9852891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9853249Z return mod(**inputs) 2025-08-26T20:31:09.9853656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9854093Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9854535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9854928Z outputs = block( 2025-08-26T20:31:09.9855342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9855736Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9856153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9856641Z return func(*args, **kwargs) 2025-08-26T20:31:09.9857042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9857483Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9857941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:09.9858446Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:09.9858841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9859266Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9859463Z 2025-08-26T20:31:09.9859576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9859965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9860313Z return mod(**inputs) 2025-08-26T20:31:09.9878975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9879618Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9880087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9880635Z outputs = block( 2025-08-26T20:31:09.9880986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9881366Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9881763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9882157Z return func(*args, **kwargs) 2025-08-26T20:31:09.9882545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-26T20:31:09.9882971Z hidden_states = residual + feed_forward_hidden_states 2025-08-26T20:31:09.9883141Z 2025-08-26T20:31:09.9883262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9883637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9883999Z return mod(**inputs) 2025-08-26T20:31:09.9884395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9884845Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9885279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9885677Z outputs = block( 2025-08-26T20:31:09.9886030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9886424Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9886834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9887229Z return func(*args, **kwargs) 2025-08-26T20:31:09.9887630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9888063Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9888516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9888917Z return func(*args, **kwargs) 2025-08-26T20:31:09.9889335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:09.9889875Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:09.9890378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9890818Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9891014Z 2025-08-26T20:31:09.9891116Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9891345Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9891571Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9891793Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9892085Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9892476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9892831Z return mod(**inputs) 2025-08-26T20:31:09.9893224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9893655Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9894071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9894464Z outputs = block( 2025-08-26T20:31:09.9894811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9895202Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9895633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9896031Z return func(*args, **kwargs) 2025-08-26T20:31:09.9896590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9897030Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9897456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9897860Z return func(*args, **kwargs) 2025-08-26T20:31:09.9898255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9898701Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9899191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:09.9899725Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:09.9899929Z 2025-08-26T20:31:09.9900055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9900455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9900832Z return mod(**inputs) 2025-08-26T20:31:09.9901229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9901658Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9902077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9902485Z outputs = block( 2025-08-26T20:31:09.9902838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9903241Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9903666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9904169Z return func(*args, **kwargs) 2025-08-26T20:31:09.9904580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9905062Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9905493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9905914Z return func(*args, **kwargs) 2025-08-26T20:31:09.9906319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9906768Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9907255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:09.9907809Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:09.9907990Z 2025-08-26T20:31:09.9908112Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9908510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9908873Z return mod(**inputs) 2025-08-26T20:31:09.9909275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9909719Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9910141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9910548Z outputs = block( 2025-08-26T20:31:09.9910904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9911350Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9911788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9912201Z return func(*args, **kwargs) 2025-08-26T20:31:09.9912609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9913056Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9913487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9913899Z return func(*args, **kwargs) 2025-08-26T20:31:09.9914306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:09.9914742Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:09.9915142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9915591Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9915789Z 2025-08-26T20:31:09.9915906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9916303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9916675Z return mod(**inputs) 2025-08-26T20:31:09.9917088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9917526Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9917937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9918334Z outputs = block( 2025-08-26T20:31:09.9918681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9919083Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9919601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9920027Z return func(*args, **kwargs) 2025-08-26T20:31:09.9920463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9920912Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9921356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:09.9921778Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:09.9922163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9922595Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9922782Z 2025-08-26T20:31:09.9922909Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9923316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9923668Z return mod(**inputs) 2025-08-26T20:31:09.9924057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9924484Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9924899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9925302Z outputs = block( 2025-08-26T20:31:09.9925649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9926036Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9926441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9926860Z return func(*args, **kwargs) 2025-08-26T20:31:09.9927253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9927692Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9928129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:09.9928544Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:09.9928915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:09.9929400Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:09.9929654Z 2025-08-26T20:31:09.9929768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9930160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9930515Z return mod(**inputs) 2025-08-26T20:31:09.9930897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9931324Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9931737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9932133Z outputs = block( 2025-08-26T20:31:09.9932481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9932860Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9933262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9933661Z return func(*args, **kwargs) 2025-08-26T20:31:09.9934051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9934514Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9934947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:09.9935409Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:09.9935773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9936182Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9936362Z 2025-08-26T20:31:09.9936473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9936850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9937187Z return mod(**inputs) 2025-08-26T20:31:09.9937560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9937995Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9938403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9938808Z outputs = block( 2025-08-26T20:31:09.9939159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9939552Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9939946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9940347Z return func(*args, **kwargs) 2025-08-26T20:31:09.9940741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9941172Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9941623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9942028Z return func(*args, **kwargs) 2025-08-26T20:31:09.9942434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:09.9942994Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:09.9943502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9943940Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9944124Z 2025-08-26T20:31:09.9944214Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9944448Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9944677Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9944901Z cudagraph partition due to non gpu ops 2025-08-26T20:31:09.9945155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9945553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9945904Z return mod(**inputs) 2025-08-26T20:31:09.9946295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9946712Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9947127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9947523Z outputs = block( 2025-08-26T20:31:09.9947870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9948255Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9948648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9949059Z return func(*args, **kwargs) 2025-08-26T20:31:09.9949482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9949912Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9950348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9950740Z return func(*args, **kwargs) 2025-08-26T20:31:09.9951134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9951580Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9952056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:09.9952566Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:09.9952777Z 2025-08-26T20:31:09.9952916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9953319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9953699Z return mod(**inputs) 2025-08-26T20:31:09.9954113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9954565Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9954993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9955399Z outputs = block( 2025-08-26T20:31:09.9955756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9956158Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9956587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9957025Z return func(*args, **kwargs) 2025-08-26T20:31:09.9957444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9957890Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9958319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9958734Z return func(*args, **kwargs) 2025-08-26T20:31:09.9959141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:09.9959685Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:09.9960180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:09.9960693Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:09.9960878Z 2025-08-26T20:31:09.9961005Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9961401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9961768Z return mod(**inputs) 2025-08-26T20:31:09.9962163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9962602Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9963011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9963413Z outputs = block( 2025-08-26T20:31:09.9963765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9964162Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9964565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9964957Z return func(*args, **kwargs) 2025-08-26T20:31:09.9965375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9965821Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:09.9966238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9966635Z return func(*args, **kwargs) 2025-08-26T20:31:09.9967025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:09.9967447Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:09.9967830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9968267Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9968475Z 2025-08-26T20:31:09.9968590Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9968984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9969335Z return mod(**inputs) 2025-08-26T20:31:09.9969726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9970152Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9970560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9970956Z outputs = block( 2025-08-26T20:31:09.9971304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9971691Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9972111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9972513Z return func(*args, **kwargs) 2025-08-26T20:31:09.9972905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9973353Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9973793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:09.9974213Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:09.9974594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9975023Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9975207Z 2025-08-26T20:31:09.9975327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9975718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9976063Z return mod(**inputs) 2025-08-26T20:31:09.9976452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9976878Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9977295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9977687Z outputs = block( 2025-08-26T20:31:09.9978033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9978422Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9978827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9979226Z return func(*args, **kwargs) 2025-08-26T20:31:09.9979629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9980108Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9980558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:09.9981003Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:09.9981381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:09.9981883Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:09.9982150Z 2025-08-26T20:31:09.9982268Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9982672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9983033Z return mod(**inputs) 2025-08-26T20:31:09.9983426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9983885Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9984315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9984729Z outputs = block( 2025-08-26T20:31:09.9985086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9985481Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9985902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9986311Z return func(*args, **kwargs) 2025-08-26T20:31:09.9986713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:09.9987180Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:09.9987629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:09.9988063Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:09.9988462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:09.9988900Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:09.9989090Z 2025-08-26T20:31:09.9989205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9989606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9989965Z return mod(**inputs) 2025-08-26T20:31:09.9990361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9990798Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9991218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9991629Z outputs = block( 2025-08-26T20:31:09.9991980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9992380Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9992783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9993187Z return func(*args, **kwargs) 2025-08-26T20:31:09.9993590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-26T20:31:09.9994045Z hidden_states = residual + feed_forward_hidden_states 2025-08-26T20:31:09.9994225Z 2025-08-26T20:31:09.9994348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:09.9994743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:09.9995153Z return mod(**inputs) 2025-08-26T20:31:09.9995549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:09.9996007Z transformer_outputs = self.transformer( 2025-08-26T20:31:09.9996584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:09.9996995Z outputs = block( 2025-08-26T20:31:09.9997356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:09.9997759Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:09.9998180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:09.9998588Z return func(*args, **kwargs) 2025-08-26T20:31:09.9999006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:09.9999571Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0000010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0000423Z return func(*args, **kwargs) 2025-08-26T20:31:10.0000825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:10.0001376Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:10.0001896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0002341Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0002531Z 2025-08-26T20:31:10.0002673Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0002912Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0003149Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0003381Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0003642Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0004036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0004399Z return mod(**inputs) 2025-08-26T20:31:10.0004811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0005237Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0005659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0006070Z outputs = block( 2025-08-26T20:31:10.0006430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0006835Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0007249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0007651Z return func(*args, **kwargs) 2025-08-26T20:31:10.0008061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0008504Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0008921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0009335Z return func(*args, **kwargs) 2025-08-26T20:31:10.0009723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0010167Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0010701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:10.0011233Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:10.0011434Z 2025-08-26T20:31:10.0011583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0011977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0012339Z return mod(**inputs) 2025-08-26T20:31:10.0012743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0013182Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0013605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0014014Z outputs = block( 2025-08-26T20:31:10.0014370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0014794Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0015210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0015290Z return func(*args, **kwargs) 2025-08-26T20:31:10.0015562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0015668Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0015935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0016021Z return func(*args, **kwargs) 2025-08-26T20:31:10.0016290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0016421Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0016750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:10.0016895Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:10.0016901Z 2025-08-26T20:31:10.0017028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0017254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0017328Z return mod(**inputs) 2025-08-26T20:31:10.0017632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0017726Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0018012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0018085Z outputs = block( 2025-08-26T20:31:10.0018342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0018437Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0018722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0018806Z return func(*args, **kwargs) 2025-08-26T20:31:10.0019098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0019202Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0019481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0019558Z return func(*args, **kwargs) 2025-08-26T20:31:10.0019852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:10.0019946Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:10.0020221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0020357Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0020361Z 2025-08-26T20:31:10.0020494Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0020726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0020800Z return mod(**inputs) 2025-08-26T20:31:10.0021081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0021176Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0021576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0021681Z outputs = block( 2025-08-26T20:31:10.0022035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0022714Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0023363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0024002Z return func(*args, **kwargs) 2025-08-26T20:31:10.0024571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0025209Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0026680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:10.0027653Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:10.0028380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0029228Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0029584Z 2025-08-26T20:31:10.0029777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0030474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0031003Z return mod(**inputs) 2025-08-26T20:31:10.0031568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0032005Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0032434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0032845Z outputs = block( 2025-08-26T20:31:10.0033187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0033586Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0034002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0034404Z return func(*args, **kwargs) 2025-08-26T20:31:10.0034798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0035244Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0036548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:10.0037364Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:10.0037771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:10.0038263Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:10.0038528Z 2025-08-26T20:31:10.0038649Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0039098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0039532Z return mod(**inputs) 2025-08-26T20:31:10.0039967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0040405Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0040839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0041236Z outputs = block( 2025-08-26T20:31:10.0041585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0041968Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0042378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0042813Z return func(*args, **kwargs) 2025-08-26T20:31:10.0043213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0043648Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0044079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:10.0044501Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:10.0044890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0045320Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0045509Z 2025-08-26T20:31:10.0045631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0046016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0046673Z return mod(**inputs) 2025-08-26T20:31:10.0047126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0047870Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0048285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0048698Z outputs = block( 2025-08-26T20:31:10.0049064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0049466Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0049875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0050317Z return func(*args, **kwargs) 2025-08-26T20:31:10.0050904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0051526Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0052132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0052713Z return func(*args, **kwargs) 2025-08-26T20:31:10.0053290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:10.0054108Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:10.0054848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0055472Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0055740Z 2025-08-26T20:31:10.0055852Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0056180Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0056543Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0056813Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0057414Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0057821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0058213Z return mod(**inputs) 2025-08-26T20:31:10.0058624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0059070Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0059487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0059899Z outputs = block( 2025-08-26T20:31:10.0060265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0060752Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0061390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0062011Z return func(*args, **kwargs) 2025-08-26T20:31:10.0062461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0062901Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0063328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0063730Z return func(*args, **kwargs) 2025-08-26T20:31:10.0064137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0064581Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0065077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:10.0065636Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:10.0065840Z 2025-08-26T20:31:10.0065956Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0066354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0066711Z return mod(**inputs) 2025-08-26T20:31:10.0067107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0067542Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0067966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0068374Z outputs = block( 2025-08-26T20:31:10.0068729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0069139Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0069553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0069960Z return func(*args, **kwargs) 2025-08-26T20:31:10.0070417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0070862Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0071293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0071713Z return func(*args, **kwargs) 2025-08-26T20:31:10.0072118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0072562Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0073061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:10.0073589Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:10.0073778Z 2025-08-26T20:31:10.0073895Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0074316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0074681Z return mod(**inputs) 2025-08-26T20:31:10.0075078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0075504Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0075936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0076347Z outputs = block( 2025-08-26T20:31:10.0076703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0077130Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0077541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0077953Z return func(*args, **kwargs) 2025-08-26T20:31:10.0078364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0078798Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0079314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0079783Z return func(*args, **kwargs) 2025-08-26T20:31:10.0080190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:10.0080622Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:10.0081047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0081488Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0081690Z 2025-08-26T20:31:10.0081809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0082216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0082582Z return mod(**inputs) 2025-08-26T20:31:10.0082981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0083437Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0083867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0084275Z outputs = block( 2025-08-26T20:31:10.0084634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0085032Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0085463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0085878Z return func(*args, **kwargs) 2025-08-26T20:31:10.0086289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0086744Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0087188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:10.0087618Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:10.0088016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0088465Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0088659Z 2025-08-26T20:31:10.0088783Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0089202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0089562Z return mod(**inputs) 2025-08-26T20:31:10.0089978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0090417Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0090838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0091252Z outputs = block( 2025-08-26T20:31:10.0091609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0092017Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0092420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0092836Z return func(*args, **kwargs) 2025-08-26T20:31:10.0093256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0093703Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0094153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:10.0094574Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:10.0094966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:10.0095472Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:10.0095728Z 2025-08-26T20:31:10.0095852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0096508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0096935Z return mod(**inputs) 2025-08-26T20:31:10.0097333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0097772Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0098201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0098610Z outputs = block( 2025-08-26T20:31:10.0098950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0099336Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0099738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0100140Z return func(*args, **kwargs) 2025-08-26T20:31:10.0100532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0100976Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0101411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:10.0101828Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:10.0102212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0102629Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0102822Z 2025-08-26T20:31:10.0102934Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0103317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0103660Z return mod(**inputs) 2025-08-26T20:31:10.0104043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0104540Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0104955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0105383Z outputs = block( 2025-08-26T20:31:10.0105733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0106112Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0106521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0106920Z return func(*args, **kwargs) 2025-08-26T20:31:10.0107316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-26T20:31:10.0107763Z hidden_states = residual + feed_forward_hidden_states 2025-08-26T20:31:10.0107989Z 2025-08-26T20:31:10.0108106Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0108493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0108842Z return mod(**inputs) 2025-08-26T20:31:10.0109230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0109648Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0110066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0110463Z outputs = block( 2025-08-26T20:31:10.0110809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0111200Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0111630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0112037Z return func(*args, **kwargs) 2025-08-26T20:31:10.0112431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0112867Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0113286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0113687Z return func(*args, **kwargs) 2025-08-26T20:31:10.0114083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:10.0114628Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:10.0115124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0115550Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0115742Z 2025-08-26T20:31:10.0115832Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0116073Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0116308Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0116538Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0116792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0117197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0117557Z return mod(**inputs) 2025-08-26T20:31:10.0117961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0118393Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0118824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0119333Z outputs = block( 2025-08-26T20:31:10.0119726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0120132Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0120570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0120987Z return func(*args, **kwargs) 2025-08-26T20:31:10.0121398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0121837Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0122261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0122674Z return func(*args, **kwargs) 2025-08-26T20:31:10.0123081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0123560Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0124058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:10.0124594Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:10.0124808Z 2025-08-26T20:31:10.0124927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0125338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0125707Z return mod(**inputs) 2025-08-26T20:31:10.0126110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0126544Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0126977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0127412Z outputs = block( 2025-08-26T20:31:10.0127767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0128161Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0128580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0128995Z return func(*args, **kwargs) 2025-08-26T20:31:10.0129401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0129847Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0130276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0130650Z return func(*args, **kwargs) 2025-08-26T20:31:10.0131023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0131436Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0131911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:10.0132399Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:10.0132581Z 2025-08-26T20:31:10.0132693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0133079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0133435Z return mod(**inputs) 2025-08-26T20:31:10.0133873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0134445Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0135048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0135476Z outputs = block( 2025-08-26T20:31:10.0135823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0136230Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0136638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0137044Z return func(*args, **kwargs) 2025-08-26T20:31:10.0137439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0137878Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0138363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0138839Z return func(*args, **kwargs) 2025-08-26T20:31:10.0139257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:10.0139677Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:10.0140057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0140488Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0140728Z 2025-08-26T20:31:10.0140846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0141252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0141643Z return mod(**inputs) 2025-08-26T20:31:10.0142028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0142453Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0142900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0143312Z outputs = block( 2025-08-26T20:31:10.0143658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0144056Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0144468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0144880Z return func(*args, **kwargs) 2025-08-26T20:31:10.0145284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0145735Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0146160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:10.0146564Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:10.0146945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0147380Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0147569Z 2025-08-26T20:31:10.0147688Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0148083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0148441Z return mod(**inputs) 2025-08-26T20:31:10.0148836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0149264Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0149701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0150116Z outputs = block( 2025-08-26T20:31:10.0150480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0150910Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0151320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0151768Z return func(*args, **kwargs) 2025-08-26T20:31:10.0152168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0152620Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0153054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:10.0153472Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:10.0153851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:10.0154341Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:10.0154613Z 2025-08-26T20:31:10.0154733Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0155124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0155467Z return mod(**inputs) 2025-08-26T20:31:10.0155850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0156273Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0156686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0157085Z outputs = block( 2025-08-26T20:31:10.0157428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0157840Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0158261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0158670Z return func(*args, **kwargs) 2025-08-26T20:31:10.0159071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0159677Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0160123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:10.0160569Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:10.0160965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0161395Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0161592Z 2025-08-26T20:31:10.0161705Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0162099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0162449Z return mod(**inputs) 2025-08-26T20:31:10.0162827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0163254Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0163668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0164067Z outputs = block( 2025-08-26T20:31:10.0164413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0164793Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0165197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0165599Z return func(*args, **kwargs) 2025-08-26T20:31:10.0166025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0166444Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0166900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0167303Z return func(*args, **kwargs) 2025-08-26T20:31:10.0167697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:10.0168251Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:10.0168746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0169182Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0169404Z 2025-08-26T20:31:10.0169494Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0169732Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0169960Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0170179Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0170436Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0170824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0171172Z return mod(**inputs) 2025-08-26T20:31:10.0171554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0171981Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0172403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0172799Z outputs = block( 2025-08-26T20:31:10.0173168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0173568Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0173992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0174401Z return func(*args, **kwargs) 2025-08-26T20:31:10.0174793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0175227Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0175645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0176046Z return func(*args, **kwargs) 2025-08-26T20:31:10.0176443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0176887Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0177373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:10.0177906Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:10.0178109Z 2025-08-26T20:31:10.0178225Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0178611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0178949Z return mod(**inputs) 2025-08-26T20:31:10.0179337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0179765Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0180179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0180577Z outputs = block( 2025-08-26T20:31:10.0180942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0181333Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0181760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0182158Z return func(*args, **kwargs) 2025-08-26T20:31:10.0182561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0182997Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0183428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0183841Z return func(*args, **kwargs) 2025-08-26T20:31:10.0184247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0184706Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0185199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:10.0185703Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:10.0185883Z 2025-08-26T20:31:10.0186008Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0186405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0186755Z return mod(**inputs) 2025-08-26T20:31:10.0187153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0187586Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0188016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0188446Z outputs = block( 2025-08-26T20:31:10.0188796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0189198Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0189614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0190023Z return func(*args, **kwargs) 2025-08-26T20:31:10.0190421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0190865Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0191293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0191701Z return func(*args, **kwargs) 2025-08-26T20:31:10.0192105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:10.0192532Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:10.0192928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0193369Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0193560Z 2025-08-26T20:31:10.0193683Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0194080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0194433Z return mod(**inputs) 2025-08-26T20:31:10.0194822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0195256Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0195681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0196083Z outputs = block( 2025-08-26T20:31:10.0196907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0197324Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0197773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0198187Z return func(*args, **kwargs) 2025-08-26T20:31:10.0198586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0199046Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0199765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:10.0200215Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:10.0200611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0201095Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0201297Z 2025-08-26T20:31:10.0201413Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0201820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0202185Z return mod(**inputs) 2025-08-26T20:31:10.0202586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0203030Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0203464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0203881Z outputs = block( 2025-08-26T20:31:10.0204238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0204666Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0205088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0205501Z return func(*args, **kwargs) 2025-08-26T20:31:10.0205909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0206359Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0206808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:10.0207236Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:10.0207625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:10.0208125Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:10.0208385Z 2025-08-26T20:31:10.0208500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0208907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0209273Z return mod(**inputs) 2025-08-26T20:31:10.0209666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0210090Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0210497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0210895Z outputs = block( 2025-08-26T20:31:10.0211245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0211648Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0212061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0212488Z return func(*args, **kwargs) 2025-08-26T20:31:10.0212883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0213351Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0213800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:10.0214281Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:10.0214668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0215094Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0215279Z 2025-08-26T20:31:10.0215395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0215785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0216146Z return mod(**inputs) 2025-08-26T20:31:10.0216529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0216966Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0217387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0217801Z outputs = block( 2025-08-26T20:31:10.0218155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0218557Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0218973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0219380Z return func(*args, **kwargs) 2025-08-26T20:31:10.0219792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-26T20:31:10.0220240Z hidden_states = residual + feed_forward_hidden_states 2025-08-26T20:31:10.0220420Z 2025-08-26T20:31:10.0220530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0220920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0221268Z return mod(**inputs) 2025-08-26T20:31:10.0221644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0222081Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0222515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0222921Z outputs = block( 2025-08-26T20:31:10.0223252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0223643Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0224058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0224468Z return func(*args, **kwargs) 2025-08-26T20:31:10.0224870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0225298Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0225724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0226133Z return func(*args, **kwargs) 2025-08-26T20:31:10.0226541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-26T20:31:10.0227076Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-26T20:31:10.0227624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0228063Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0228260Z 2025-08-26T20:31:10.0228369Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0228611Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0228836Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0229065Z cudagraph partition due to non gpu ops 2025-08-26T20:31:10.0229325Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0229718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0230067Z return mod(**inputs) 2025-08-26T20:31:10.0230468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0230914Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0231369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0231781Z outputs = block( 2025-08-26T20:31:10.0232133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0232538Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0232956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0233373Z return func(*args, **kwargs) 2025-08-26T20:31:10.0233770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0234206Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0234634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0235062Z return func(*args, **kwargs) 2025-08-26T20:31:10.0235466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0235574Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0235894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:31:10.0236053Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:31:10.0236057Z 2025-08-26T20:31:10.0236174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0236403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0236478Z return mod(**inputs) 2025-08-26T20:31:10.0236764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0236862Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0237132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0237212Z outputs = block( 2025-08-26T20:31:10.0237456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0237555Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0237819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0237896Z return func(*args, **kwargs) 2025-08-26T20:31:10.0238175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0238273Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0238544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0238625Z return func(*args, **kwargs) 2025-08-26T20:31:10.0238925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-26T20:31:10.0239061Z attn_output, attn_weights = attention_interface( 2025-08-26T20:31:10.0239464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:31:10.0239602Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:31:10.0239607Z 2025-08-26T20:31:10.0239725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0239958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0240034Z return mod(**inputs) 2025-08-26T20:31:10.0240311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0240446Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0240729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0240807Z outputs = block( 2025-08-26T20:31:10.0241044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0241132Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0241400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0241485Z return func(*args, **kwargs) 2025-08-26T20:31:10.0241754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-26T20:31:10.0241848Z attn_output, self_attn_weights = self.attn( 2025-08-26T20:31:10.0242132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0242210Z return func(*args, **kwargs) 2025-08-26T20:31:10.0242473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-26T20:31:10.0242571Z attn_output = self.c_proj(attn_output) 2025-08-26T20:31:10.0242806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0242941Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0242945Z 2025-08-26T20:31:10.0243057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0243272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0243351Z return mod(**inputs) 2025-08-26T20:31:10.0243623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0243725Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0243987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0244054Z outputs = block( 2025-08-26T20:31:10.0244304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0244389Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0244657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0244730Z return func(*args, **kwargs) 2025-08-26T20:31:10.0245010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0245122Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0245385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-26T20:31:10.0245506Z hidden_states = self.c_fc(hidden_states) 2025-08-26T20:31:10.0245739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0245889Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0245893Z 2025-08-26T20:31:10.0246006Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0246230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0246310Z return mod(**inputs) 2025-08-26T20:31:10.0246592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0246689Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0246974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0247070Z outputs = block( 2025-08-26T20:31:10.0247304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0247389Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0247651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0247723Z return func(*args, **kwargs) 2025-08-26T20:31:10.0247994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0248100Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0248344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-26T20:31:10.0248453Z hidden_states = self.act(hidden_states) 2025-08-26T20:31:10.0248686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:10.0248886Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:10.0248890Z 2025-08-26T20:31:10.0249001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0249221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0249286Z return mod(**inputs) 2025-08-26T20:31:10.0249534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-26T20:31:10.0249624Z transformer_outputs = self.transformer( 2025-08-26T20:31:10.0249865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-26T20:31:10.0249937Z outputs = block( 2025-08-26T20:31:10.0250155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:10.0250234Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:10.0250478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:31:10.0250546Z return func(*args, **kwargs) 2025-08-26T20:31:10.0250794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-26T20:31:10.0250894Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-26T20:31:10.0251137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-26T20:31:10.0251232Z hidden_states = self.c_proj(hidden_states) 2025-08-26T20:31:10.0251447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-26T20:31:10.0251574Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-26T20:31:10.0251577Z 2025-08-26T20:31:10.0251702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0251912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0252000Z return mod(**inputs) 2025-08-26T20:31:10.0252254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1494, in forward 2025-08-26T20:31:10.0252343Z logits = self.score(hidden_states) 2025-08-26T20:31:10.0252347Z 2025-08-26T20:31:10.0252449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0252657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0252724Z return mod(**inputs) 2025-08-26T20:31:10.0252974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-08-26T20:31:10.0253152Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-26T20:31:10.0253157Z 2025-08-26T20:31:10.0253258Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:10.0253467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:10.0253535Z return mod(**inputs) 2025-08-26T20:31:10.0253795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-08-26T20:31:10.0253937Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-26T20:31:10.0253941Z 2025-08-26T20:31:21.8075600Z Compilation time (from dynamo_timed): 17.610952714 2025-08-26T20:31:21.8076141Z pass 2025-08-26T20:31:21.8076655Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:21.8077933Z TIMING: _recursive_pre_grad_passes:0.01508 _recursive_joint_graph_passes:0.60368 _recursive_post_grad_passes:0.07913 async_compile.wait:0.73856 code_gen:8.67515 inductor_compile:9.90733 backend_compile:13.23202 gc:0.00159 entire_frame_compile:17.61095 total_wall_time:17.61095 2025-08-26T20:31:21.8079067Z STATS: call_* op count: 1138 | FakeTensorMode.__torch_dispatch__:12455 | FakeTensor.__torch_dispatch__:4284 | ProxyTorchDispatchMode.__torch_dispatch__:4144 2025-08-26T20:31:21.8079857Z Dynamo produced 2 graphs covering 1138 ops with 0 graph breaks (0 unique) 2025-08-26T20:31:27.2441005Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:31:27.2442038Z from pkg_resources import resource_filename 2025-08-26T20:31:27.8915486Z 2025-08-26T20:31:29.1329940Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:31:29.1330403Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:31:29.1349685Z cpu eval GoogleFnet 2025-08-26T20:31:29.5585587Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:29.7400978Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:29.9148399Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:35.7462791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7463358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7463736Z return mod(**inputs) 2025-08-26T20:31:35.7464187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7464709Z outputs = self.fnet( 2025-08-26T20:31:35.7465483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7465973Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7466511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7466953Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7467368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7467833Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7468277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7468723Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7469173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7469715Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7470144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7470605Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7470779Z 2025-08-26T20:31:35.7470906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7471297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7471671Z return mod(**inputs) 2025-08-26T20:31:35.7472143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7472564Z outputs = self.fnet( 2025-08-26T20:31:35.7472940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7473425Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7473849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7474296Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7474711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7475104Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7475522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7475968Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7476418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7476852Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7477317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7477789Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7477972Z 2025-08-26T20:31:35.7478092Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7478500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7478863Z return mod(**inputs) 2025-08-26T20:31:35.7479479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7479917Z outputs = self.fnet( 2025-08-26T20:31:35.7480318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7480759Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7481170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7481595Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7482043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7482481Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7482930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7483369Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7483798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7484217Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7484640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7485135Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7485334Z 2025-08-26T20:31:35.7485447Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7485840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7486191Z return mod(**inputs) 2025-08-26T20:31:35.7486586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7486996Z outputs = self.fnet( 2025-08-26T20:31:35.7487372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7487788Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7488204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7488632Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7489024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7489452Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7489877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7490325Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7490765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7491194Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7491630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7492080Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7492250Z 2025-08-26T20:31:35.7492377Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7492777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7493161Z return mod(**inputs) 2025-08-26T20:31:35.7493550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7493953Z outputs = self.fnet( 2025-08-26T20:31:35.7494344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7494756Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7495164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7495586Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7495989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7496558Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7496976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7497458Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7497882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7498332Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7498748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7499204Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7499369Z 2025-08-26T20:31:35.7499490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7499865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7500221Z return mod(**inputs) 2025-08-26T20:31:35.7500602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7501035Z outputs = self.fnet( 2025-08-26T20:31:35.7501404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7501813Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7502224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7502656Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7503064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7503443Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7503856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7504321Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7504759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7505177Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7505586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7506027Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7506201Z 2025-08-26T20:31:35.7506314Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7506704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7507043Z return mod(**inputs) 2025-08-26T20:31:35.7507426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7507835Z outputs = self.fnet( 2025-08-26T20:31:35.7508230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7508656Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7509062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7509502Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7509910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7510301Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7510714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7511141Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7511571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7511990Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7512437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7512886Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7513063Z 2025-08-26T20:31:35.7513202Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7513602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7513962Z return mod(**inputs) 2025-08-26T20:31:35.7514352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7514769Z outputs = self.fnet( 2025-08-26T20:31:35.7515146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7515560Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7515964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7516429Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7516815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7517204Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7517626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7518079Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7518514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7518942Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7519442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7519932Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7520107Z 2025-08-26T20:31:35.7520238Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7520650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7521012Z return mod(**inputs) 2025-08-26T20:31:35.7521400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7521822Z outputs = self.fnet( 2025-08-26T20:31:35.7522218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7522641Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7523058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7523509Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7523922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7524311Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7524782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7525215Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7525636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7526045Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7526437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7526862Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7527032Z 2025-08-26T20:31:35.7527428Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7527849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7528197Z return mod(**inputs) 2025-08-26T20:31:35.7528572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7528959Z outputs = self.fnet( 2025-08-26T20:31:35.7529320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7529706Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7530079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7530481Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7530852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7531241Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7531634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7532038Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7532450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7532846Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7533239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7533651Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7533809Z 2025-08-26T20:31:35.7533915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7534279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7534628Z return mod(**inputs) 2025-08-26T20:31:35.7534987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7535355Z outputs = self.fnet( 2025-08-26T20:31:35.7535714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7536096Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7536473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7536870Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7537232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7537597Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7537996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7538431Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7538861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7539271Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7539685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7540116Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7540275Z 2025-08-26T20:31:35.7540387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7540744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7541074Z return mod(**inputs) 2025-08-26T20:31:35.7541433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7541831Z outputs = self.fnet( 2025-08-26T20:31:35.7542230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7542626Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7543024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7543425Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7543796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7544163Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7544545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7544954Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7545361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7545775Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7546158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7546575Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7546740Z 2025-08-26T20:31:35.7546846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7547214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7547562Z return mod(**inputs) 2025-08-26T20:31:35.7547930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7548331Z outputs = self.fnet( 2025-08-26T20:31:35.7548708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 512, in forward 2025-08-26T20:31:35.7549159Z embedding_output = self.embeddings( 2025-08-26T20:31:35.7549549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 142, in forward 2025-08-26T20:31:35.7549960Z embeddings = self.projection(embeddings) 2025-08-26T20:31:35.7550120Z 2025-08-26T20:31:35.7550212Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7550473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7550862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7551203Z return mod(**inputs) 2025-08-26T20:31:35.7551584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7551985Z outputs = self.fnet( 2025-08-26T20:31:35.7552360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7552774Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7553168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7553587Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7553980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7554366Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7554768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7555205Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7555635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7556056Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7556485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7556920Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7557093Z 2025-08-26T20:31:35.7557207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7557612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7557968Z return mod(**inputs) 2025-08-26T20:31:35.7558347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7558752Z outputs = self.fnet( 2025-08-26T20:31:35.7559129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7559634Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7560048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7560510Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7560917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7561329Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7561743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7562188Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7562614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7563043Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7563468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7563949Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7564107Z 2025-08-26T20:31:35.7564220Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7564580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7564914Z return mod(**inputs) 2025-08-26T20:31:35.7565278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7565653Z outputs = self.fnet( 2025-08-26T20:31:35.7565996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7566385Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7566775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7567180Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7567561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7567925Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7568320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7568739Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7569145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7569540Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7569925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7570337Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7570501Z 2025-08-26T20:31:35.7570611Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7570977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7571320Z return mod(**inputs) 2025-08-26T20:31:35.7571684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7572097Z outputs = self.fnet( 2025-08-26T20:31:35.7572458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7572861Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7573245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7573640Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7574009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7574384Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7574774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7575172Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7575575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7575975Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7576363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7576835Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7577012Z 2025-08-26T20:31:35.7577124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7577509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7577859Z return mod(**inputs) 2025-08-26T20:31:35.7578262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7578665Z outputs = self.fnet( 2025-08-26T20:31:35.7579023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7579412Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7579792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7580186Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7580560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7580923Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7581311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7581711Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7582124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7582535Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7582959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7583424Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7583852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7584247Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7584398Z 2025-08-26T20:31:35.7584506Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7584872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7585208Z return mod(**inputs) 2025-08-26T20:31:35.7585584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7585966Z outputs = self.fnet( 2025-08-26T20:31:35.7586340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7586725Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7587101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7587489Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7587860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7588242Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7588651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7589099Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7589502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7589906Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7590330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7590786Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7591215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7591635Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7592043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7592564Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7592817Z 2025-08-26T20:31:35.7592940Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7593321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7593676Z return mod(**inputs) 2025-08-26T20:31:35.7594061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7594468Z outputs = self.fnet( 2025-08-26T20:31:35.7594847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7595252Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7595657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7596084Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7596631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7597026Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7597436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7597861Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7598295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7598722Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7599171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7599749Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7600226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7600728Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7600885Z 2025-08-26T20:31:35.7600986Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7601260Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7601681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7602035Z return mod(**inputs) 2025-08-26T20:31:35.7602421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7602839Z outputs = self.fnet( 2025-08-26T20:31:35.7603215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7603629Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7604034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7604495Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7604882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7605276Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7605695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7606127Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7606574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7607044Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7607470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7607946Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7608115Z 2025-08-26T20:31:35.7608235Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7608619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7608959Z return mod(**inputs) 2025-08-26T20:31:35.7609346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7609753Z outputs = self.fnet( 2025-08-26T20:31:35.7610170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7610590Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7610997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7611421Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7611819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7612210Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7612610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7613053Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7613493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7613926Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7614309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7614725Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7614892Z 2025-08-26T20:31:35.7614998Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7615367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7615719Z return mod(**inputs) 2025-08-26T20:31:35.7616072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7616480Z outputs = self.fnet( 2025-08-26T20:31:35.7616876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7617288Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7617682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7618076Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7618447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7618824Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7619237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7619688Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7620122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7620543Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7620944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7621360Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7621519Z 2025-08-26T20:31:35.7621628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7621992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7622321Z return mod(**inputs) 2025-08-26T20:31:35.7622703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7623090Z outputs = self.fnet( 2025-08-26T20:31:35.7623438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7623825Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7624203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7624599Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7624960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7625324Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7625710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7626120Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7626527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7626914Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7627300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7627711Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7627870Z 2025-08-26T20:31:35.7627985Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7628349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7628670Z return mod(**inputs) 2025-08-26T20:31:35.7629029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7629409Z outputs = self.fnet( 2025-08-26T20:31:35.7629805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7630179Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7630567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7630959Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7631321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7631680Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7632052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7632450Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7632860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7633291Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7633735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7634215Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7634669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7635088Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7635236Z 2025-08-26T20:31:35.7635365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7635766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7636121Z return mod(**inputs) 2025-08-26T20:31:35.7636525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7636944Z outputs = self.fnet( 2025-08-26T20:31:35.7637325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7637722Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7638126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7638548Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7638944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7639421Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7639836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7640280Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7640699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7641129Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7641533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7641992Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7642416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7642839Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7643219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7643666Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7643905Z 2025-08-26T20:31:35.7644010Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7644371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7644722Z return mod(**inputs) 2025-08-26T20:31:35.7645078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7645464Z outputs = self.fnet( 2025-08-26T20:31:35.7645820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7646195Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7646567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7646956Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7647321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7647681Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7648085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7648469Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7648860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7649252Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7649659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7650123Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7650550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7650927Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7651093Z 2025-08-26T20:31:35.7651176Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7651417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7651773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7652101Z return mod(**inputs) 2025-08-26T20:31:35.7652453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7652831Z outputs = self.fnet( 2025-08-26T20:31:35.7653189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7653576Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7653955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7654343Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7654705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7655064Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7655443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7655843Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7656243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7656626Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7657009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7657414Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7657574Z 2025-08-26T20:31:35.7657681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7658044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7658374Z return mod(**inputs) 2025-08-26T20:31:35.7658751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7659125Z outputs = self.fnet( 2025-08-26T20:31:35.7659502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7659890Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7660268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7660665Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7661025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7661391Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7661786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7662242Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7662667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7663090Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7663500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7663952Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7664121Z 2025-08-26T20:31:35.7664243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7664613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7664946Z return mod(**inputs) 2025-08-26T20:31:35.7665327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7665709Z outputs = self.fnet( 2025-08-26T20:31:35.7666066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7666452Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7666831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7667231Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7667603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7667965Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7668356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7668770Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7669181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7669575Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7669959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7670375Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7670539Z 2025-08-26T20:31:35.7670646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7671010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7671339Z return mod(**inputs) 2025-08-26T20:31:35.7671691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7672068Z outputs = self.fnet( 2025-08-26T20:31:35.7672425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7672836Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7673213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7673637Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7674014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7674381Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7674776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7675184Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7675641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7676062Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7676456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7676872Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7677031Z 2025-08-26T20:31:35.7677140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7677505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7677838Z return mod(**inputs) 2025-08-26T20:31:35.7678199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7678575Z outputs = self.fnet( 2025-08-26T20:31:35.7678952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7679491Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7679915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7680316Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7680688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7681062Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7681475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7681959Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7682391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7682809Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7683263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7683752Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7684205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7684629Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7684792Z 2025-08-26T20:31:35.7684905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7685289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7685639Z return mod(**inputs) 2025-08-26T20:31:35.7686017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7686422Z outputs = self.fnet( 2025-08-26T20:31:35.7686798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7687211Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7687644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7688092Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7688489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7688838Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7689222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7689628Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7690018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7690414Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7690822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7691291Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7691711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7692106Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7692487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7692941Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7693168Z 2025-08-26T20:31:35.7693282Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7693638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7693985Z return mod(**inputs) 2025-08-26T20:31:35.7694341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7694723Z outputs = self.fnet( 2025-08-26T20:31:35.7695060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7695419Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7695783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7696280Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7696659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7697015Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7697389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7697778Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7698176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7698566Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7698969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7699415Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7699839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7700231Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7700363Z 2025-08-26T20:31:35.7700452Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7700684Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7701023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7701340Z return mod(**inputs) 2025-08-26T20:31:35.7701725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7702087Z outputs = self.fnet( 2025-08-26T20:31:35.7702444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7702826Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7703195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7703589Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7703953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7704307Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7704692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7705147Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7705541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7705922Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7706312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7706717Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7706874Z 2025-08-26T20:31:35.7706989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7707349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7707666Z return mod(**inputs) 2025-08-26T20:31:35.7708043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7708413Z outputs = self.fnet( 2025-08-26T20:31:35.7708759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7709139Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7709503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7709891Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7710252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7710605Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7710976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7711381Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7711782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7712166Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7712542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7712936Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7713097Z 2025-08-26T20:31:35.7713198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7713551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7713867Z return mod(**inputs) 2025-08-26T20:31:35.7714215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7714574Z outputs = self.fnet( 2025-08-26T20:31:35.7714954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7715385Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7715785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7716221Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7716625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7716989Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7717378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7717788Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7718192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7718587Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7718997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7719495Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7719668Z 2025-08-26T20:31:35.7719791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7720175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7720522Z return mod(**inputs) 2025-08-26T20:31:35.7720878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7721251Z outputs = self.fnet( 2025-08-26T20:31:35.7721596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7722004Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7722386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7722811Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7723211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7723591Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7724002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7724452Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7724884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7725313Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7725723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7726168Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7726344Z 2025-08-26T20:31:35.7726457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7726846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7727187Z return mod(**inputs) 2025-08-26T20:31:35.7727570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7727979Z outputs = self.fnet( 2025-08-26T20:31:35.7728354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7728772Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7729162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7729557Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7729939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7730302Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7730687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7731075Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7731472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7731860Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7732265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7732708Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7733125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7733545Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7733681Z 2025-08-26T20:31:35.7733792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7734154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7734482Z return mod(**inputs) 2025-08-26T20:31:35.7734844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7735277Z outputs = self.fnet( 2025-08-26T20:31:35.7735624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7735998Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7736361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7736785Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7737174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7737534Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7737911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7738299Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7738696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7739091Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7739509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7739991Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7740445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7740873Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7741262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7741721Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7741955Z 2025-08-26T20:31:35.7742061Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7742425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7742757Z return mod(**inputs) 2025-08-26T20:31:35.7743121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7743489Z outputs = self.fnet( 2025-08-26T20:31:35.7743865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7744237Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7744628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7745018Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7745373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7745732Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7746111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7746497Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7746890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7747294Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7747704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7748148Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7748580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7748980Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7749119Z 2025-08-26T20:31:35.7749202Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7749448Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7749813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7750145Z return mod(**inputs) 2025-08-26T20:31:35.7750497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7750901Z outputs = self.fnet( 2025-08-26T20:31:35.7751263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7751657Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7752030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7752414Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7752776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7753138Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7753524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7753930Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7754349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7754748Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7755146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7755564Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7755729Z 2025-08-26T20:31:35.7755844Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7756231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7756582Z return mod(**inputs) 2025-08-26T20:31:35.7756969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7757380Z outputs = self.fnet( 2025-08-26T20:31:35.7757758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7758191Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7758603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7759050Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7759535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7759939Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7760363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7760801Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7761213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7761585Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7761988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7762415Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7762585Z 2025-08-26T20:31:35.7762709Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7763098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7763444Z return mod(**inputs) 2025-08-26T20:31:35.7763805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7764193Z outputs = self.fnet( 2025-08-26T20:31:35.7764548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7764958Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7765389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7765811Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7766206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7766593Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7766999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7767439Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7767868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7768291Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7768705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7769136Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7769311Z 2025-08-26T20:31:35.7769424Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7769807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7770159Z return mod(**inputs) 2025-08-26T20:31:35.7770531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7770931Z outputs = self.fnet( 2025-08-26T20:31:35.7771308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7771721Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7772122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7772537Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7772955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7773351Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7773770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7774165Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7774570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7774964Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7775354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7775765Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7775922Z 2025-08-26T20:31:35.7776032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7776443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7776800Z return mod(**inputs) 2025-08-26T20:31:35.7777180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7777585Z outputs = self.fnet( 2025-08-26T20:31:35.7777957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7778375Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7778798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7779219Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7779604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7780012Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7780437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7780870Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7781314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7781746Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7782208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7782693Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7783141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7783561Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7783712Z 2025-08-26T20:31:35.7783825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7784209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7784552Z return mod(**inputs) 2025-08-26T20:31:35.7784932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7785338Z outputs = self.fnet( 2025-08-26T20:31:35.7785725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7786131Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7786530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7786950Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7787331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7787722Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7788149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7788573Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7789039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7789471Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7789937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7790448Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7790894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7791350Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7791818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7792320Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7792597Z 2025-08-26T20:31:35.7792712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7793109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7793463Z return mod(**inputs) 2025-08-26T20:31:35.7793857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7794263Z outputs = self.fnet( 2025-08-26T20:31:35.7794776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7795249Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7795672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7796113Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7796642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7798153Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7798700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7799167Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7800073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7800542Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7801012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7801562Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7802061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7802511Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7802673Z 2025-08-26T20:31:35.7802776Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7803059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7803516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7803910Z return mod(**inputs) 2025-08-26T20:31:35.7804311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7804738Z outputs = self.fnet( 2025-08-26T20:31:35.7805403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7806011Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7806413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7806890Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7807268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7807621Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7808014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7808432Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7808854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7809252Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7809684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7810109Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7810281Z 2025-08-26T20:31:35.7810395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7810774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7811147Z return mod(**inputs) 2025-08-26T20:31:35.7811530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7811962Z outputs = self.fnet( 2025-08-26T20:31:35.7812361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7812797Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7813212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7813659Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7814084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7814466Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7814857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7815265Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7815688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7816083Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7816472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7816891Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7817050Z 2025-08-26T20:31:35.7817160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7817535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7817864Z return mod(**inputs) 2025-08-26T20:31:35.7818215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7818581Z outputs = self.fnet( 2025-08-26T20:31:35.7818930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7819308Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7819682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7820074Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7820459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7820825Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7821226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7821656Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7822067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7822471Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7822871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7823298Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7823461Z 2025-08-26T20:31:35.7823582Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7824002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7824347Z return mod(**inputs) 2025-08-26T20:31:35.7824705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7825088Z outputs = self.fnet( 2025-08-26T20:31:35.7825452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7825836Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7826219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7826632Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7827019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7827400Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7827841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7828264Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7828679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7829145Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7829536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7829968Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7830135Z 2025-08-26T20:31:35.7830242Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7830621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7830962Z return mod(**inputs) 2025-08-26T20:31:35.7831331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7831730Z outputs = self.fnet( 2025-08-26T20:31:35.7832088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7832478Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7832853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7833267Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7833648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7834019Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7834413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7834815Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7835267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7835703Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7837141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7837658Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7838122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7838556Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7838716Z 2025-08-26T20:31:35.7838832Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7839316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7839763Z return mod(**inputs) 2025-08-26T20:31:35.7840180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7840597Z outputs = self.fnet( 2025-08-26T20:31:35.7840990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7841401Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7841804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7842211Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7842589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7842970Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7843385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7843826Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7844261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7844687Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7845131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7845619Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7846073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7846533Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7846955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7847460Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7847715Z 2025-08-26T20:31:35.7847836Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7848238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7848599Z return mod(**inputs) 2025-08-26T20:31:35.7848988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7849395Z outputs = self.fnet( 2025-08-26T20:31:35.7849768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7850182Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7850580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7850976Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7851390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7851747Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7852176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7852583Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7853004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7853427Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7853930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7854406Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7854886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7855322Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7855477Z 2025-08-26T20:31:35.7855561Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7855818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7856186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7856528Z return mod(**inputs) 2025-08-26T20:31:35.7856896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7857284Z outputs = self.fnet( 2025-08-26T20:31:35.7857647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7858047Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7858453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7858863Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7859230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7859594Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7859977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7860380Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7860771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7861158Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7861540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7861951Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7862112Z 2025-08-26T20:31:35.7862225Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7862586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7862920Z return mod(**inputs) 2025-08-26T20:31:35.7863282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7863701Z outputs = self.fnet( 2025-08-26T20:31:35.7864049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7864477Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7864908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7865334Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7865758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7866125Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7866536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7866955Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7867373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7867766Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7868164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7868591Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7868761Z 2025-08-26T20:31:35.7868888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7869309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7869669Z return mod(**inputs) 2025-08-26T20:31:35.7870154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7870589Z outputs = self.fnet( 2025-08-26T20:31:35.7870951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7871343Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7871720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7872124Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7872360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7872470Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7872726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7872835Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7873093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7873185Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7873442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7873547Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7873551Z 2025-08-26T20:31:35.7873672Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7873892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7873973Z return mod(**inputs) 2025-08-26T20:31:35.7874242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7874323Z outputs = self.fnet( 2025-08-26T20:31:35.7874592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7874671Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7874944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7875038Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7875286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7875374Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7875640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7875754Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7876041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7876137Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7876421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7876532Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7876544Z 2025-08-26T20:31:35.7876657Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7876872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7876953Z return mod(**inputs) 2025-08-26T20:31:35.7877223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7877309Z outputs = self.fnet( 2025-08-26T20:31:35.7877603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7877685Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7877963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7878061Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7878312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7878399Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7878670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7878773Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7879061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7879243Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7879569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7879713Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7879985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7880077Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7880083Z 2025-08-26T20:31:35.7880206Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7880426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7880505Z return mod(**inputs) 2025-08-26T20:31:35.7880781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7880857Z outputs = self.fnet( 2025-08-26T20:31:35.7881127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7881211Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7881489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7881579Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7881806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7881899Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7882150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7882245Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7882556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7882670Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7882981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7883124Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7883413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7883534Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7883776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7883976Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7883982Z 2025-08-26T20:31:35.7884103Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7884338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7884413Z return mod(**inputs) 2025-08-26T20:31:35.7884694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7884765Z outputs = self.fnet( 2025-08-26T20:31:35.7885046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7885121Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7885370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7885472Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7885699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7885823Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7886073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7886160Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7886449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7886533Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7886856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7886996Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7887274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7887364Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7887370Z 2025-08-26T20:31:35.7887462Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7887587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7887805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7887885Z return mod(**inputs) 2025-08-26T20:31:35.7888162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7888237Z outputs = self.fnet( 2025-08-26T20:31:35.7888525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7888606Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7888897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7888988Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7889229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7889346Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7889640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7889755Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7890028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7890123Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7890394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7890507Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7890511Z 2025-08-26T20:31:35.7890629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7890849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7890951Z return mod(**inputs) 2025-08-26T20:31:35.7891235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7891310Z outputs = self.fnet( 2025-08-26T20:31:35.7891593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7891672Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7891950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7892043Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7892283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7892396Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7892665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7892782Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7893046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7893144Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7893406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7893518Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7893522Z 2025-08-26T20:31:35.7893641Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7893853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7893933Z return mod(**inputs) 2025-08-26T20:31:35.7894198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7894274Z outputs = self.fnet( 2025-08-26T20:31:35.7894546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7894626Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7894895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7894984Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7895226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7895312Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7895576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7895690Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7895977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7896075Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7896823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7897012Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7897017Z 2025-08-26T20:31:35.7897148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7897363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7897446Z return mod(**inputs) 2025-08-26T20:31:35.7897708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7897791Z outputs = self.fnet( 2025-08-26T20:31:35.7898088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7898167Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7898437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7898530Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7898773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7898857Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7899120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7899230Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7899489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7899612Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7899874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7899986Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7899996Z 2025-08-26T20:31:35.7900104Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7900318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7900397Z return mod(**inputs) 2025-08-26T20:31:35.7900661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7900742Z outputs = self.fnet( 2025-08-26T20:31:35.7901008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7901089Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7901362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7901453Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7901698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7901782Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7902045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7902143Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7902423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7902516Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7902817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7903019Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7903283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7903402Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7903407Z 2025-08-26T20:31:35.7903527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7903740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7903819Z return mod(**inputs) 2025-08-26T20:31:35.7904081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7904156Z outputs = self.fnet( 2025-08-26T20:31:35.7904424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7904538Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7904810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7904904Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7905145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7905238Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7905492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7905586Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7905854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7905938Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7906253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7906372Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7906627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7906740Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7906977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7907171Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7907176Z 2025-08-26T20:31:35.7907294Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7907511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7907585Z return mod(**inputs) 2025-08-26T20:31:35.7907860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7907932Z outputs = self.fnet( 2025-08-26T20:31:35.7908202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7908283Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7908545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7908646Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7908881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7908974Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7909237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7909331Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7909636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7909724Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7910044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7910187Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7910466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7910556Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7910560Z 2025-08-26T20:31:35.7910650Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7910773Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7910990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7911101Z return mod(**inputs) 2025-08-26T20:31:35.7911364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7911437Z outputs = self.fnet( 2025-08-26T20:31:35.7911707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7911785Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7912052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7912142Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7912378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7912491Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7912757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7912870Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7913130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7913223Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7913483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7913592Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7913596Z 2025-08-26T20:31:35.7913711Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7913921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7913998Z return mod(**inputs) 2025-08-26T20:31:35.7914260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7914333Z outputs = self.fnet( 2025-08-26T20:31:35.7914601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7914682Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7914945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7915035Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7915269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7915359Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7915618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7915731Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7916016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7916112Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7916391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7916505Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7916509Z 2025-08-26T20:31:35.7916630Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7916849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7916927Z return mod(**inputs) 2025-08-26T20:31:35.7917193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7917268Z outputs = self.fnet( 2025-08-26T20:31:35.7917573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7917654Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7917937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7918033Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7918284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7918373Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7918644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7918760Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7919031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7919154Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7919499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7919619Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7919626Z 2025-08-26T20:31:35.7919745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7919963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7920043Z return mod(**inputs) 2025-08-26T20:31:35.7920318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7920395Z outputs = self.fnet( 2025-08-26T20:31:35.7920672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7920772Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7921057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7921161Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7921398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7921482Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7921733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7921841Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7922088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7922179Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7922444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7922585Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7922598Z 2025-08-26T20:31:35.7922716Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7923010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7923096Z return mod(**inputs) 2025-08-26T20:31:35.7923368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7923447Z outputs = self.fnet( 2025-08-26T20:31:35.7923723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7923802Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7924083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7924199Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7924462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7924549Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7924830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7924932Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7925221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7925312Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7925619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7925758Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7926123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7926218Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7926222Z 2025-08-26T20:31:35.7926348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7926570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7926651Z return mod(**inputs) 2025-08-26T20:31:35.7926931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7927005Z outputs = self.fnet( 2025-08-26T20:31:35.7927292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7927373Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7927656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7927753Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7927995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7928089Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7928373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7928471Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7928745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7928828Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7929106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7929230Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7929522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7929642Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7929901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7930087Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7930091Z 2025-08-26T20:31:35.7930203Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7930409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7930479Z return mod(**inputs) 2025-08-26T20:31:35.7930754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7930829Z outputs = self.fnet( 2025-08-26T20:31:35.7931132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7931207Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7931460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7931555Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7931781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7931870Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7932118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7932203Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7932475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7932578Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7932866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7933012Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7933260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7933341Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7933345Z 2025-08-26T20:31:35.7933427Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7933537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7933732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7933806Z return mod(**inputs) 2025-08-26T20:31:35.7934049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7934118Z outputs = self.fnet( 2025-08-26T20:31:35.7934366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7934440Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7934685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7934769Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7934986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7935071Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7935310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7935415Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7935677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7935767Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7936032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7936137Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7936141Z 2025-08-26T20:31:35.7936251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7936447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7936522Z return mod(**inputs) 2025-08-26T20:31:35.7936767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7936839Z outputs = self.fnet( 2025-08-26T20:31:35.7937092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7937183Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7937430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7937517Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7937733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7937820Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7938059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7938166Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7938403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7938509Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7938753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7938854Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7938859Z 2025-08-26T20:31:35.7938970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7939172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7939248Z return mod(**inputs) 2025-08-26T20:31:35.7939494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7939563Z outputs = self.fnet( 2025-08-26T20:31:35.7939820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7939897Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7940155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7940242Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7940472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7940554Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7940812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7940916Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7941155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7941241Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7941485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7941606Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7941611Z 2025-08-26T20:31:35.7941724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7941971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7942046Z return mod(**inputs) 2025-08-26T20:31:35.7942295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7942362Z outputs = self.fnet( 2025-08-26T20:31:35.7942619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7942693Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7942947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7943051Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7943284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7943375Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7943617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7943718Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7943958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7944043Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7944287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7944386Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7944414Z 2025-08-26T20:31:35.7944518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7944716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7944791Z return mod(**inputs) 2025-08-26T20:31:35.7945032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7945106Z outputs = self.fnet( 2025-08-26T20:31:35.7945344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7945418Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7945665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7945748Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7945971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7946052Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7946293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7946383Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7946642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7946726Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7946997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7947110Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7947359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7947442Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7947447Z 2025-08-26T20:31:35.7947554Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7947767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7947843Z return mod(**inputs) 2025-08-26T20:31:35.7948101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7948171Z outputs = self.fnet( 2025-08-26T20:31:35.7948426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7948503Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7948758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7948845Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7949073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7949183Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7949435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7949528Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7949793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7949877Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7950160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7950277Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7950533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7950666Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7950893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7951080Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7951084Z 2025-08-26T20:31:35.7951189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7951400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7951468Z return mod(**inputs) 2025-08-26T20:31:35.7951724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7951797Z outputs = self.fnet( 2025-08-26T20:31:35.7952070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7952153Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7952416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7952517Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7952755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7952848Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7953112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7953203Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7953488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7953572Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7953877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7954051Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7954323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7954429Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7954434Z 2025-08-26T20:31:35.7954526Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7954643Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7954861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7954940Z return mod(**inputs) 2025-08-26T20:31:35.7955201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7955272Z outputs = self.fnet( 2025-08-26T20:31:35.7955545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7955647Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7955915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7956008Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7956246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7956339Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7956604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7956718Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7956978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7957094Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7957359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7957469Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7957473Z 2025-08-26T20:31:35.7957591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7957805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7957883Z return mod(**inputs) 2025-08-26T20:31:35.7958145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7958215Z outputs = self.fnet( 2025-08-26T20:31:35.7958483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7958565Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7958846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7958940Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7959268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7959375Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7959650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7959767Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7960038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7960136Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7960407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7960532Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7960574Z 2025-08-26T20:31:35.7960689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7960911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7960990Z return mod(**inputs) 2025-08-26T20:31:35.7961239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7961309Z outputs = self.fnet( 2025-08-26T20:31:35.7961566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7961644Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7961900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7961990Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7962239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7962321Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7962572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7962679Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7962926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7963014Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7963261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7963362Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7963388Z 2025-08-26T20:31:35.7963501Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7963706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7963782Z return mod(**inputs) 2025-08-26T20:31:35.7964030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7964098Z outputs = self.fnet( 2025-08-26T20:31:35.7964352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7964427Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7964683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7964770Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7965000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7965084Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7965330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7965437Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7965683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7965774Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7966021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7966124Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7966136Z 2025-08-26T20:31:35.7966239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7966441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7966517Z return mod(**inputs) 2025-08-26T20:31:35.7966784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7966860Z outputs = self.fnet( 2025-08-26T20:31:35.7967128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7967204Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7967458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7967545Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7967775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7967855Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7968105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7968219Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7968483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7968571Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7968850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7968966Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7969219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7969302Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7969306Z 2025-08-26T20:31:35.7969417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7969637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7969713Z return mod(**inputs) 2025-08-26T20:31:35.7969960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7970030Z outputs = self.fnet( 2025-08-26T20:31:35.7970289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7970366Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7970618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7970704Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7970940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7971032Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7971300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7971399Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7971677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7971769Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7972066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7972189Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7972457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7972574Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7972807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7973022Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7973027Z 2025-08-26T20:31:35.7973137Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7973373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7973447Z return mod(**inputs) 2025-08-26T20:31:35.7973715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7973786Z outputs = self.fnet( 2025-08-26T20:31:35.7974055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7974135Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7974397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7974512Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7974746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7974838Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7975099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7975187Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7975471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7975552Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7975850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.7975985Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.7976275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.7976363Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7976367Z 2025-08-26T20:31:35.7976454Z cudagraph partition due to non gpu ops 2025-08-26T20:31:35.7976575Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7976787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7976863Z return mod(**inputs) 2025-08-26T20:31:35.7977122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7977195Z outputs = self.fnet( 2025-08-26T20:31:35.7977462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7977544Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7977815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7977906Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7978145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7978236Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7981932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7982070Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7982344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7982435Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7982708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7982824Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7982828Z 2025-08-26T20:31:35.7982945Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7983190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7983266Z return mod(**inputs) 2025-08-26T20:31:35.7983540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7983650Z outputs = self.fnet( 2025-08-26T20:31:35.7983917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7984003Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7984266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7984367Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7984624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7984709Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7984977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7985083Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7985358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7985438Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7985709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7985813Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7985850Z 2025-08-26T20:31:35.7985963Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7986167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7986233Z return mod(**inputs) 2025-08-26T20:31:35.7986543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7986614Z outputs = self.fnet( 2025-08-26T20:31:35.7986889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7986970Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7987246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7987345Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7987590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7987683Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7987955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7988066Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7988370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7988456Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7988802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7988915Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7988919Z 2025-08-26T20:31:35.7989035Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7989262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7989337Z return mod(**inputs) 2025-08-26T20:31:35.7989618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7989689Z outputs = self.fnet( 2025-08-26T20:31:35.7989981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7990064Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7990335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7990433Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7990681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7990772Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7991044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-26T20:31:35.7991177Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-26T20:31:35.7991481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-26T20:31:35.7991571Z self_outputs = self.self(hidden_states) 2025-08-26T20:31:35.7991855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-26T20:31:35.7991964Z outputs = self.fourier_transform(hidden_states).real 2025-08-26T20:31:35.7991968Z 2025-08-26T20:31:35.7992085Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7992310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7992380Z return mod(**inputs) 2025-08-26T20:31:35.7992655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7992747Z outputs = self.fnet( 2025-08-26T20:31:35.7993025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7993105Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7993389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7993482Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7993731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7993825Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7994102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7994202Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7994500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7994586Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7994897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7995025Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7995313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-26T20:31:35.7995443Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.7995447Z 2025-08-26T20:31:35.7995568Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.7995782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.7995854Z return mod(**inputs) 2025-08-26T20:31:35.7996124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.7996434Z outputs = self.fnet( 2025-08-26T20:31:35.7996892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.7996979Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.7997297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.7997402Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.7997643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.7997734Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.7998005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.7998100Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.7998435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.7998521Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.7998839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-26T20:31:35.7998969Z intermediate_output = self.intermediate(fourier_output) 2025-08-26T20:31:35.7999305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-26T20:31:35.7999436Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:31:35.7999672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-26T20:31:35.7999882Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-26T20:31:35.7999923Z 2025-08-26T20:31:35.8000040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.8000270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.8000344Z return mod(**inputs) 2025-08-26T20:31:35.8000616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-26T20:31:35.8000698Z outputs = self.fnet( 2025-08-26T20:31:35.8000970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-26T20:31:35.8001062Z encoder_outputs = self.encoder( 2025-08-26T20:31:35.8001340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-26T20:31:35.8001440Z layer_outputs = layer_module(hidden_states) 2025-08-26T20:31:35.8001677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:31:35.8001763Z return super().__call__(*args, **kwargs) 2025-08-26T20:31:35.8002029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-26T20:31:35.8002117Z layer_output = apply_chunking_to_forward( 2025-08-26T20:31:35.8002401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:31:35.8002483Z return forward_fn(*input_tensors) 2025-08-26T20:31:35.8002815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-26T20:31:35.8002963Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-26T20:31:35.8003222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-26T20:31:35.8017635Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.8017672Z 2025-08-26T20:31:35.8017937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.8018199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.8018286Z return mod(**inputs) 2025-08-26T20:31:35.8018676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-08-26T20:31:35.8018787Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:31:35.8019065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-08-26T20:31:35.8019186Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:31:35.8019454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-08-26T20:31:35.8019557Z hidden_states = self.transform(hidden_states) 2025-08-26T20:31:35.8019852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 321, in forward 2025-08-26T20:31:35.8019944Z hidden_states = self.dense(hidden_states) 2025-08-26T20:31:35.8019949Z 2025-08-26T20:31:35.8020063Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.8020289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.8020362Z return mod(**inputs) 2025-08-26T20:31:35.8020634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-08-26T20:31:35.8020733Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:31:35.8021001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-08-26T20:31:35.8021133Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:31:35.8021424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 341, in forward 2025-08-26T20:31:35.8021537Z hidden_states = self.decoder(hidden_states) 2025-08-26T20:31:35.8021541Z 2025-08-26T20:31:35.8021654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:31:35.8021884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:31:35.8021957Z return mod(**inputs) 2025-08-26T20:31:35.8022227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 686, in forward 2025-08-26T20:31:35.8022446Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:31:35.8022450Z 2025-08-26T20:31:44.3944156Z Compilation time (from dynamo_timed): 13.181173263 2025-08-26T20:31:44.3997351Z pass 2025-08-26T20:31:44.4000383Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:44.4001309Z TIMING: _recursive_pre_grad_passes:0.00648 _recursive_joint_graph_passes:0.49915 _recursive_post_grad_passes:0.07274 async_compile.wait:0.76471 code_gen:8.20802 inductor_compile:9.14155 backend_compile:11.33753 gc:0.00029 entire_frame_compile:13.18117 total_wall_time:13.18117 2025-08-26T20:31:44.4002303Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:7515 | FakeTensor.__torch_dispatch__:3268 | ProxyTorchDispatchMode.__torch_dispatch__:2859 2025-08-26T20:31:44.4003145Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-08-26T20:31:49.8354744Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:31:49.8355782Z from pkg_resources import resource_filename 2025-08-26T20:31:50.4438691Z 2025-08-26T20:31:51.8842567Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:31:51.8842894Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:31:51.8855194Z cpu eval LayoutLMForMaskedLM 2025-08-26T20:31:52.4784265Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:52.7121440Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:31:52.9454765Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:01.8015540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8018449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8018886Z return mod(**inputs) 2025-08-26T20:32:01.8024534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8026231Z return func(*args, **kwargs) 2025-08-26T20:32:01.8031250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8034707Z return func(*args, **kwargs) 2025-08-26T20:32:01.8035390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8036207Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8036787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8037260Z outputs = self.layoutlm( 2025-08-26T20:32:01.8037671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8046351Z return func(*args, **kwargs) 2025-08-26T20:32:01.8047025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8053146Z return func(*args, **kwargs) 2025-08-26T20:32:01.8059025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8059682Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8060295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8060794Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8061242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8061679Z return func(*args, **kwargs) 2025-08-26T20:32:01.8062091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8062527Z return func(*args, **kwargs) 2025-08-26T20:32:01.8062945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8063374Z return func(*args, **kwargs) 2025-08-26T20:32:01.8063611Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8064006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8064410Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8065052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8065509Z layer_outputs = layer_module( 2025-08-26T20:32:01.8065891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8066285Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8066720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8067123Z return func(*args, **kwargs) 2025-08-26T20:32:01.8067531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8067928Z return func(*args, **kwargs) 2025-08-26T20:32:01.8068391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8068792Z return func(*args, **kwargs) 2025-08-26T20:32:01.8069215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8069665Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8070125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8070495Z return func(*args, **kwargs) 2025-08-26T20:32:01.8070893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8071330Z return func(*args, **kwargs) 2025-08-26T20:32:01.8071717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8072114Z return func(*args, **kwargs) 2025-08-26T20:32:01.8072527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8072971Z self_outputs = self.self( 2025-08-26T20:32:01.8073379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8073776Z return func(*args, **kwargs) 2025-08-26T20:32:01.8074160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8074582Z return func(*args, **kwargs) 2025-08-26T20:32:01.8075649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8076051Z return func(*args, **kwargs) 2025-08-26T20:32:01.8076464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8076977Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8077200Z 2025-08-26T20:32:01.8077321Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8077726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8078079Z return mod(**inputs) 2025-08-26T20:32:01.8078453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8078851Z return func(*args, **kwargs) 2025-08-26T20:32:01.8079480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8079902Z return func(*args, **kwargs) 2025-08-26T20:32:01.8080273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8080657Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8081097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8081508Z outputs = self.layoutlm( 2025-08-26T20:32:01.8081905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8082284Z return func(*args, **kwargs) 2025-08-26T20:32:01.8082648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8083023Z return func(*args, **kwargs) 2025-08-26T20:32:01.8083369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8083733Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8084166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8084585Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8084971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8085352Z return func(*args, **kwargs) 2025-08-26T20:32:01.8085732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8086130Z return func(*args, **kwargs) 2025-08-26T20:32:01.8086509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8086895Z return func(*args, **kwargs) 2025-08-26T20:32:01.8087122Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8087480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8087862Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8088295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8088733Z layer_outputs = layer_module( 2025-08-26T20:32:01.8089113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8089481Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8089865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8090252Z return func(*args, **kwargs) 2025-08-26T20:32:01.8090654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8091051Z return func(*args, **kwargs) 2025-08-26T20:32:01.8091436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8091833Z return func(*args, **kwargs) 2025-08-26T20:32:01.8092250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8092696Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8093100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8093498Z return func(*args, **kwargs) 2025-08-26T20:32:01.8093885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8094291Z return func(*args, **kwargs) 2025-08-26T20:32:01.8094671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8095069Z return func(*args, **kwargs) 2025-08-26T20:32:01.8095488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8095923Z self_outputs = self.self( 2025-08-26T20:32:01.8096652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8097091Z return func(*args, **kwargs) 2025-08-26T20:32:01.8097484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8097885Z return func(*args, **kwargs) 2025-08-26T20:32:01.8098273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8098669Z return func(*args, **kwargs) 2025-08-26T20:32:01.8099083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8099585Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8099802Z 2025-08-26T20:32:01.8100008Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8100389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8100724Z return mod(**inputs) 2025-08-26T20:32:01.8101082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8101459Z return func(*args, **kwargs) 2025-08-26T20:32:01.8101821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8102199Z return func(*args, **kwargs) 2025-08-26T20:32:01.8102569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8102934Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8103352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8103747Z outputs = self.layoutlm( 2025-08-26T20:32:01.8104097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8104465Z return func(*args, **kwargs) 2025-08-26T20:32:01.8104820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8105194Z return func(*args, **kwargs) 2025-08-26T20:32:01.8105539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8105919Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8106329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8106737Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8107121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8107504Z return func(*args, **kwargs) 2025-08-26T20:32:01.8107854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8108217Z return func(*args, **kwargs) 2025-08-26T20:32:01.8108569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8108935Z return func(*args, **kwargs) 2025-08-26T20:32:01.8109126Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8109487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8109846Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8110252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8110663Z layer_outputs = layer_module( 2025-08-26T20:32:01.8111022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8111399Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8111775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8112144Z return func(*args, **kwargs) 2025-08-26T20:32:01.8112490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8112860Z return func(*args, **kwargs) 2025-08-26T20:32:01.8113217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8113582Z return func(*args, **kwargs) 2025-08-26T20:32:01.8114007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8114424Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8114838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8115241Z return func(*args, **kwargs) 2025-08-26T20:32:01.8115625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8116012Z return func(*args, **kwargs) 2025-08-26T20:32:01.8116403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8116822Z return func(*args, **kwargs) 2025-08-26T20:32:01.8117245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8117682Z self_outputs = self.self( 2025-08-26T20:32:01.8118071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8118460Z return func(*args, **kwargs) 2025-08-26T20:32:01.8118842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8119301Z return func(*args, **kwargs) 2025-08-26T20:32:01.8119698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8120106Z return func(*args, **kwargs) 2025-08-26T20:32:01.8120549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8121095Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8121318Z 2025-08-26T20:32:01.8121422Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8121656Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8121930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8122335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8122727Z return mod(**inputs) 2025-08-26T20:32:01.8123109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8123516Z return func(*args, **kwargs) 2025-08-26T20:32:01.8123905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8124311Z return func(*args, **kwargs) 2025-08-26T20:32:01.8124681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8125064Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8125505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8125941Z outputs = self.layoutlm( 2025-08-26T20:32:01.8126333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8126746Z return func(*args, **kwargs) 2025-08-26T20:32:01.8127107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8127484Z return func(*args, **kwargs) 2025-08-26T20:32:01.8127824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8128188Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8128592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8129001Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8129394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8129771Z return func(*args, **kwargs) 2025-08-26T20:32:01.8130132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8130496Z return func(*args, **kwargs) 2025-08-26T20:32:01.8130859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8131229Z return func(*args, **kwargs) 2025-08-26T20:32:01.8131428Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8131781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8132155Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8132561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8132970Z layer_outputs = layer_module( 2025-08-26T20:32:01.8133323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8133690Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8134074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8134448Z return func(*args, **kwargs) 2025-08-26T20:32:01.8134813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8135240Z return func(*args, **kwargs) 2025-08-26T20:32:01.8135604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8135976Z return func(*args, **kwargs) 2025-08-26T20:32:01.8136371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8136793Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8137174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8137553Z return func(*args, **kwargs) 2025-08-26T20:32:01.8137919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8138296Z return func(*args, **kwargs) 2025-08-26T20:32:01.8138656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8139031Z return func(*args, **kwargs) 2025-08-26T20:32:01.8139434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8139895Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8140348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8140751Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8140899Z 2025-08-26T20:32:01.8141022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8141387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8141712Z return mod(**inputs) 2025-08-26T20:32:01.8142062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8142422Z return func(*args, **kwargs) 2025-08-26T20:32:01.8142778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8143140Z return func(*args, **kwargs) 2025-08-26T20:32:01.8143492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8143846Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8144241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8144641Z outputs = self.layoutlm( 2025-08-26T20:32:01.8144997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8145362Z return func(*args, **kwargs) 2025-08-26T20:32:01.8145706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8146089Z return func(*args, **kwargs) 2025-08-26T20:32:01.8146425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8146778Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8147179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8147572Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8147942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8148307Z return func(*args, **kwargs) 2025-08-26T20:32:01.8148658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8149021Z return func(*args, **kwargs) 2025-08-26T20:32:01.8149374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8149756Z return func(*args, **kwargs) 2025-08-26T20:32:01.8149954Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8150305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8150652Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8151052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8151454Z layer_outputs = layer_module( 2025-08-26T20:32:01.8151807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8152162Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8152541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8152908Z return func(*args, **kwargs) 2025-08-26T20:32:01.8153269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8153633Z return func(*args, **kwargs) 2025-08-26T20:32:01.8153988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8154347Z return func(*args, **kwargs) 2025-08-26T20:32:01.8154767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8155200Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8155618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8156017Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8156461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8156964Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8157428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8157864Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8158024Z 2025-08-26T20:32:01.8158141Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8158531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8158888Z return mod(**inputs) 2025-08-26T20:32:01.8159371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8159815Z return func(*args, **kwargs) 2025-08-26T20:32:01.8160230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8160674Z return func(*args, **kwargs) 2025-08-26T20:32:01.8161050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8161447Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8161852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8162266Z outputs = self.layoutlm( 2025-08-26T20:32:01.8162636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8163012Z return func(*args, **kwargs) 2025-08-26T20:32:01.8163368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8163754Z return func(*args, **kwargs) 2025-08-26T20:32:01.8164088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8164466Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8164863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8165256Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8165632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8166005Z return func(*args, **kwargs) 2025-08-26T20:32:01.8166369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8166740Z return func(*args, **kwargs) 2025-08-26T20:32:01.8167102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8167476Z return func(*args, **kwargs) 2025-08-26T20:32:01.8167681Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8168044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8168400Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8168811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8169226Z layer_outputs = layer_module( 2025-08-26T20:32:01.8169590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8169955Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8170341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8170717Z return func(*args, **kwargs) 2025-08-26T20:32:01.8171083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8171463Z return func(*args, **kwargs) 2025-08-26T20:32:01.8171823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8172197Z return func(*args, **kwargs) 2025-08-26T20:32:01.8172609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8173038Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8173460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8173870Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8174327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8174846Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8175353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8175818Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8176225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8176583Z return self.act(input) 2025-08-26T20:32:01.8176706Z 2025-08-26T20:32:01.8176826Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8177214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8177573Z return mod(**inputs) 2025-08-26T20:32:01.8177968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8178374Z return func(*args, **kwargs) 2025-08-26T20:32:01.8178763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8179902Z return func(*args, **kwargs) 2025-08-26T20:32:01.8180245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8180613Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8181028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8181441Z outputs = self.layoutlm( 2025-08-26T20:32:01.8181807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8182191Z return func(*args, **kwargs) 2025-08-26T20:32:01.8182561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8182944Z return func(*args, **kwargs) 2025-08-26T20:32:01.8183284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8183632Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8184035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8184436Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8184815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8185219Z return func(*args, **kwargs) 2025-08-26T20:32:01.8185572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8185935Z return func(*args, **kwargs) 2025-08-26T20:32:01.8186286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8186659Z return func(*args, **kwargs) 2025-08-26T20:32:01.8186854Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8187212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8187569Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8187992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8188396Z layer_outputs = layer_module( 2025-08-26T20:32:01.8188753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8189125Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8189512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8189888Z return func(*args, **kwargs) 2025-08-26T20:32:01.8190247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8190657Z return func(*args, **kwargs) 2025-08-26T20:32:01.8191020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8191405Z return func(*args, **kwargs) 2025-08-26T20:32:01.8191832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8192277Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8192719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8193157Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8193597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8194112Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8194583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8195013Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8195163Z 2025-08-26T20:32:01.8195289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8195685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8196035Z return mod(**inputs) 2025-08-26T20:32:01.8196608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8197021Z return func(*args, **kwargs) 2025-08-26T20:32:01.8197416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8197816Z return func(*args, **kwargs) 2025-08-26T20:32:01.8198182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8198569Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8199007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8199506Z outputs = self.layoutlm( 2025-08-26T20:32:01.8199894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8200355Z return func(*args, **kwargs) 2025-08-26T20:32:01.8200765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8201242Z return func(*args, **kwargs) 2025-08-26T20:32:01.8201611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8201988Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8202424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8202857Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8203288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8203680Z return func(*args, **kwargs) 2025-08-26T20:32:01.8204066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8204462Z return func(*args, **kwargs) 2025-08-26T20:32:01.8204846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8205248Z return func(*args, **kwargs) 2025-08-26T20:32:01.8205453Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8205831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8206246Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8206683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8207105Z layer_outputs = layer_module( 2025-08-26T20:32:01.8207481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8207871Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8208285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8208683Z return func(*args, **kwargs) 2025-08-26T20:32:01.8209060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8209455Z return func(*args, **kwargs) 2025-08-26T20:32:01.8209883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8210278Z return func(*args, **kwargs) 2025-08-26T20:32:01.8210718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8211174Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8211589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8211989Z return func(*args, **kwargs) 2025-08-26T20:32:01.8212374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8212773Z return func(*args, **kwargs) 2025-08-26T20:32:01.8213154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8213533Z return func(*args, **kwargs) 2025-08-26T20:32:01.8213926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8214341Z self_outputs = self.self( 2025-08-26T20:32:01.8214725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8215122Z return func(*args, **kwargs) 2025-08-26T20:32:01.8215531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8215908Z return func(*args, **kwargs) 2025-08-26T20:32:01.8216261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8216640Z return func(*args, **kwargs) 2025-08-26T20:32:01.8217058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8217583Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8217804Z 2025-08-26T20:32:01.8217929Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8218331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8218685Z return mod(**inputs) 2025-08-26T20:32:01.8219051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8219452Z return func(*args, **kwargs) 2025-08-26T20:32:01.8219843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8220215Z return func(*args, **kwargs) 2025-08-26T20:32:01.8220572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8220957Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8221413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8221845Z outputs = self.layoutlm( 2025-08-26T20:32:01.8222234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8222634Z return func(*args, **kwargs) 2025-08-26T20:32:01.8223020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8223418Z return func(*args, **kwargs) 2025-08-26T20:32:01.8223776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8224142Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8224549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8225002Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8225408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8225785Z return func(*args, **kwargs) 2025-08-26T20:32:01.8226155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8226538Z return func(*args, **kwargs) 2025-08-26T20:32:01.8226906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8227275Z return func(*args, **kwargs) 2025-08-26T20:32:01.8227477Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8227838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8228202Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8228607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8229022Z layer_outputs = layer_module( 2025-08-26T20:32:01.8229383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8229761Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8230147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8230542Z return func(*args, **kwargs) 2025-08-26T20:32:01.8230910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8231288Z return func(*args, **kwargs) 2025-08-26T20:32:01.8231650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8232025Z return func(*args, **kwargs) 2025-08-26T20:32:01.8232415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8232837Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8233245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8233625Z return func(*args, **kwargs) 2025-08-26T20:32:01.8233992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8234389Z return func(*args, **kwargs) 2025-08-26T20:32:01.8234776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8235173Z return func(*args, **kwargs) 2025-08-26T20:32:01.8235590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8236036Z self_outputs = self.self( 2025-08-26T20:32:01.8236429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8236827Z return func(*args, **kwargs) 2025-08-26T20:32:01.8237222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8237591Z return func(*args, **kwargs) 2025-08-26T20:32:01.8237983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8238382Z return func(*args, **kwargs) 2025-08-26T20:32:01.8238801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8239379Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8239628Z 2025-08-26T20:32:01.8239754Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8240171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8240543Z return mod(**inputs) 2025-08-26T20:32:01.8240956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8241362Z return func(*args, **kwargs) 2025-08-26T20:32:01.8241744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8242148Z return func(*args, **kwargs) 2025-08-26T20:32:01.8242511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8242897Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8243329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8243768Z outputs = self.layoutlm( 2025-08-26T20:32:01.8244162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8244568Z return func(*args, **kwargs) 2025-08-26T20:32:01.8244957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8245352Z return func(*args, **kwargs) 2025-08-26T20:32:01.8245774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8246163Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8246608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8247047Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8247461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8247868Z return func(*args, **kwargs) 2025-08-26T20:32:01.8248266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8248684Z return func(*args, **kwargs) 2025-08-26T20:32:01.8249060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8249422Z return func(*args, **kwargs) 2025-08-26T20:32:01.8249625Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8249991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8250347Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8250762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8251199Z layer_outputs = layer_module( 2025-08-26T20:32:01.8251564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8251950Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8252345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8252732Z return func(*args, **kwargs) 2025-08-26T20:32:01.8253112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8253507Z return func(*args, **kwargs) 2025-08-26T20:32:01.8253868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8254234Z return func(*args, **kwargs) 2025-08-26T20:32:01.8254633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8255059Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8255435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8255791Z return func(*args, **kwargs) 2025-08-26T20:32:01.8256149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8256513Z return func(*args, **kwargs) 2025-08-26T20:32:01.8256864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8257228Z return func(*args, **kwargs) 2025-08-26T20:32:01.8257601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8257996Z self_outputs = self.self( 2025-08-26T20:32:01.8258358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8258733Z return func(*args, **kwargs) 2025-08-26T20:32:01.8259087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8259460Z return func(*args, **kwargs) 2025-08-26T20:32:01.8259821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8260222Z return func(*args, **kwargs) 2025-08-26T20:32:01.8260607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8261072Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8261278Z 2025-08-26T20:32:01.8261362Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8261587Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8261839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8262209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8262534Z return mod(**inputs) 2025-08-26T20:32:01.8262902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8263273Z return func(*args, **kwargs) 2025-08-26T20:32:01.8263628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8263983Z return func(*args, **kwargs) 2025-08-26T20:32:01.8264319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8264678Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8265073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8265506Z outputs = self.layoutlm( 2025-08-26T20:32:01.8265858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8266232Z return func(*args, **kwargs) 2025-08-26T20:32:01.8266604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8266990Z return func(*args, **kwargs) 2025-08-26T20:32:01.8267324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8267684Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8268084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8268496Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8268885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8269246Z return func(*args, **kwargs) 2025-08-26T20:32:01.8269602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8269966Z return func(*args, **kwargs) 2025-08-26T20:32:01.8270317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8270673Z return func(*args, **kwargs) 2025-08-26T20:32:01.8270877Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8271226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8271580Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8271985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8272384Z layer_outputs = layer_module( 2025-08-26T20:32:01.8272740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8273108Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8273497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8273856Z return func(*args, **kwargs) 2025-08-26T20:32:01.8274238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8274617Z return func(*args, **kwargs) 2025-08-26T20:32:01.8274985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8275355Z return func(*args, **kwargs) 2025-08-26T20:32:01.8275746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8276171Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8276577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8276981Z return func(*args, **kwargs) 2025-08-26T20:32:01.8277362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8277733Z return func(*args, **kwargs) 2025-08-26T20:32:01.8278100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8278476Z return func(*args, **kwargs) 2025-08-26T20:32:01.8278870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8279414Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8279919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8280400Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8280554Z 2025-08-26T20:32:01.8280680Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8281079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8281435Z return mod(**inputs) 2025-08-26T20:32:01.8281805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8282186Z return func(*args, **kwargs) 2025-08-26T20:32:01.8282559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8282936Z return func(*args, **kwargs) 2025-08-26T20:32:01.8283285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8283678Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8284096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8284519Z outputs = self.layoutlm( 2025-08-26T20:32:01.8284886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8285269Z return func(*args, **kwargs) 2025-08-26T20:32:01.8285639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8286021Z return func(*args, **kwargs) 2025-08-26T20:32:01.8286371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8286732Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8287151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8287568Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8287955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8288312Z return func(*args, **kwargs) 2025-08-26T20:32:01.8288667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8289053Z return func(*args, **kwargs) 2025-08-26T20:32:01.8289406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8289775Z return func(*args, **kwargs) 2025-08-26T20:32:01.8289960Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8290310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8290659Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8291057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8291456Z layer_outputs = layer_module( 2025-08-26T20:32:01.8291815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8292176Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8292548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8292918Z return func(*args, **kwargs) 2025-08-26T20:32:01.8293263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8293627Z return func(*args, **kwargs) 2025-08-26T20:32:01.8293983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8294368Z return func(*args, **kwargs) 2025-08-26T20:32:01.8294745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8295161Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8295570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8295964Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8296542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8297024Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8297479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8297929Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8298067Z 2025-08-26T20:32:01.8298181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8298539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8298856Z return mod(**inputs) 2025-08-26T20:32:01.8299208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8299582Z return func(*args, **kwargs) 2025-08-26T20:32:01.8299954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8300337Z return func(*args, **kwargs) 2025-08-26T20:32:01.8300664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8301016Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8301427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8301833Z outputs = self.layoutlm( 2025-08-26T20:32:01.8302186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8302556Z return func(*args, **kwargs) 2025-08-26T20:32:01.8302914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8303312Z return func(*args, **kwargs) 2025-08-26T20:32:01.8303658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8304003Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8304401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8304800Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8305171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8305532Z return func(*args, **kwargs) 2025-08-26T20:32:01.8305919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8306291Z return func(*args, **kwargs) 2025-08-26T20:32:01.8306644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8307009Z return func(*args, **kwargs) 2025-08-26T20:32:01.8307195Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8307552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8307904Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8308302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8308733Z layer_outputs = layer_module( 2025-08-26T20:32:01.8309081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8309446Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8309830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8310204Z return func(*args, **kwargs) 2025-08-26T20:32:01.8310564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8310936Z return func(*args, **kwargs) 2025-08-26T20:32:01.8311302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8311683Z return func(*args, **kwargs) 2025-08-26T20:32:01.8312099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8312525Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8312942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8313360Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8313789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8314261Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8314713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8315165Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8315563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8315940Z return self.act(input) 2025-08-26T20:32:01.8316066Z 2025-08-26T20:32:01.8316182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8316578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8316938Z return mod(**inputs) 2025-08-26T20:32:01.8317322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8317754Z return func(*args, **kwargs) 2025-08-26T20:32:01.8318138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8318540Z return func(*args, **kwargs) 2025-08-26T20:32:01.8318911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8319366Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8319819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8320283Z outputs = self.layoutlm( 2025-08-26T20:32:01.8320771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8321155Z return func(*args, **kwargs) 2025-08-26T20:32:01.8321526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8321911Z return func(*args, **kwargs) 2025-08-26T20:32:01.8322294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8322691Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8323127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8323593Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8324002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8324421Z return func(*args, **kwargs) 2025-08-26T20:32:01.8324809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8325210Z return func(*args, **kwargs) 2025-08-26T20:32:01.8325602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8326008Z return func(*args, **kwargs) 2025-08-26T20:32:01.8326219Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8326608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8326989Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8327454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8327893Z layer_outputs = layer_module( 2025-08-26T20:32:01.8328283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8328659Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8329039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8329424Z return func(*args, **kwargs) 2025-08-26T20:32:01.8329811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8330220Z return func(*args, **kwargs) 2025-08-26T20:32:01.8330615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8331024Z return func(*args, **kwargs) 2025-08-26T20:32:01.8331464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8331888Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8332302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8332702Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8333166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8333695Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8334193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8334636Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8334789Z 2025-08-26T20:32:01.8334910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8335316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8335694Z return mod(**inputs) 2025-08-26T20:32:01.8336079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8336464Z return func(*args, **kwargs) 2025-08-26T20:32:01.8336827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8337209Z return func(*args, **kwargs) 2025-08-26T20:32:01.8337562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8337933Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8338345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8338777Z outputs = self.layoutlm( 2025-08-26T20:32:01.8339162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8339568Z return func(*args, **kwargs) 2025-08-26T20:32:01.8339952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8340340Z return func(*args, **kwargs) 2025-08-26T20:32:01.8340708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8341088Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8341523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8341954Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8342357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8342780Z return func(*args, **kwargs) 2025-08-26T20:32:01.8343164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8343564Z return func(*args, **kwargs) 2025-08-26T20:32:01.8343945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8344339Z return func(*args, **kwargs) 2025-08-26T20:32:01.8344552Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8344934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8345303Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8345734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8346169Z layer_outputs = layer_module( 2025-08-26T20:32:01.8346548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8346939Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8347339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8347732Z return func(*args, **kwargs) 2025-08-26T20:32:01.8348138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8348541Z return func(*args, **kwargs) 2025-08-26T20:32:01.8348918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8349309Z return func(*args, **kwargs) 2025-08-26T20:32:01.8349726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8350177Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8350589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8350979Z return func(*args, **kwargs) 2025-08-26T20:32:01.8351385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8351783Z return func(*args, **kwargs) 2025-08-26T20:32:01.8352169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8352568Z return func(*args, **kwargs) 2025-08-26T20:32:01.8352978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8353413Z self_outputs = self.self( 2025-08-26T20:32:01.8353808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8354221Z return func(*args, **kwargs) 2025-08-26T20:32:01.8354602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8355007Z return func(*args, **kwargs) 2025-08-26T20:32:01.8355391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8355794Z return func(*args, **kwargs) 2025-08-26T20:32:01.8356193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8356660Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8356869Z 2025-08-26T20:32:01.8356977Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8357363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8357699Z return mod(**inputs) 2025-08-26T20:32:01.8358058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8358427Z return func(*args, **kwargs) 2025-08-26T20:32:01.8358790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8359162Z return func(*args, **kwargs) 2025-08-26T20:32:01.8359584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8359978Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8360433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8360867Z outputs = self.layoutlm( 2025-08-26T20:32:01.8361264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8361670Z return func(*args, **kwargs) 2025-08-26T20:32:01.8362024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8362402Z return func(*args, **kwargs) 2025-08-26T20:32:01.8362742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8363100Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8363522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8363940Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8364317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8364694Z return func(*args, **kwargs) 2025-08-26T20:32:01.8365061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8365434Z return func(*args, **kwargs) 2025-08-26T20:32:01.8365799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8366190Z return func(*args, **kwargs) 2025-08-26T20:32:01.8366392Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8366745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8367104Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8367513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8367927Z layer_outputs = layer_module( 2025-08-26T20:32:01.8368286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8368707Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8369109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8369519Z return func(*args, **kwargs) 2025-08-26T20:32:01.8369913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8370297Z return func(*args, **kwargs) 2025-08-26T20:32:01.8370667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8371052Z return func(*args, **kwargs) 2025-08-26T20:32:01.8371456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8371889Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8372296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8372676Z return func(*args, **kwargs) 2025-08-26T20:32:01.8373042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8373424Z return func(*args, **kwargs) 2025-08-26T20:32:01.8373791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8374163Z return func(*args, **kwargs) 2025-08-26T20:32:01.8374564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8374975Z self_outputs = self.self( 2025-08-26T20:32:01.8375352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8375729Z return func(*args, **kwargs) 2025-08-26T20:32:01.8376102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8376483Z return func(*args, **kwargs) 2025-08-26T20:32:01.8376855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8377237Z return func(*args, **kwargs) 2025-08-26T20:32:01.8377631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8378146Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8378354Z 2025-08-26T20:32:01.8378468Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8378863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8379215Z return mod(**inputs) 2025-08-26T20:32:01.8379592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8379993Z return func(*args, **kwargs) 2025-08-26T20:32:01.8380374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8380780Z return func(*args, **kwargs) 2025-08-26T20:32:01.8381137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8381512Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8381942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8382365Z outputs = self.layoutlm( 2025-08-26T20:32:01.8382756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8383158Z return func(*args, **kwargs) 2025-08-26T20:32:01.8383565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8383970Z return func(*args, **kwargs) 2025-08-26T20:32:01.8384336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8384712Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8385152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8385591Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8386011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8386410Z return func(*args, **kwargs) 2025-08-26T20:32:01.8386796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8387257Z return func(*args, **kwargs) 2025-08-26T20:32:01.8387653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8388057Z return func(*args, **kwargs) 2025-08-26T20:32:01.8388262Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8388645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8389026Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8389475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8389925Z layer_outputs = layer_module( 2025-08-26T20:32:01.8390298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8390702Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8391119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8391529Z return func(*args, **kwargs) 2025-08-26T20:32:01.8391910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8392316Z return func(*args, **kwargs) 2025-08-26T20:32:01.8392698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8393102Z return func(*args, **kwargs) 2025-08-26T20:32:01.8393534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8393975Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8394385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8394790Z return func(*args, **kwargs) 2025-08-26T20:32:01.8395179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8395588Z return func(*args, **kwargs) 2025-08-26T20:32:01.8395986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8396569Z return func(*args, **kwargs) 2025-08-26T20:32:01.8397023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8397473Z self_outputs = self.self( 2025-08-26T20:32:01.8397872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8398295Z return func(*args, **kwargs) 2025-08-26T20:32:01.8398694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8399152Z return func(*args, **kwargs) 2025-08-26T20:32:01.8399613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8400033Z return func(*args, **kwargs) 2025-08-26T20:32:01.8400474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8400998Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8401220Z 2025-08-26T20:32:01.8401319Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8401553Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8401806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8402204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8402557Z return mod(**inputs) 2025-08-26T20:32:01.8402973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8403370Z return func(*args, **kwargs) 2025-08-26T20:32:01.8403754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8404153Z return func(*args, **kwargs) 2025-08-26T20:32:01.8404519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8404897Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8405334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8405769Z outputs = self.layoutlm( 2025-08-26T20:32:01.8406156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8406559Z return func(*args, **kwargs) 2025-08-26T20:32:01.8406917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8407310Z return func(*args, **kwargs) 2025-08-26T20:32:01.8407679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8408062Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8408500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8408955Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8409358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8409761Z return func(*args, **kwargs) 2025-08-26T20:32:01.8410146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8410537Z return func(*args, **kwargs) 2025-08-26T20:32:01.8410926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8411314Z return func(*args, **kwargs) 2025-08-26T20:32:01.8411516Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8411907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8412266Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8412677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8413083Z layer_outputs = layer_module( 2025-08-26T20:32:01.8413437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8413801Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8414191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8414594Z return func(*args, **kwargs) 2025-08-26T20:32:01.8414961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8415335Z return func(*args, **kwargs) 2025-08-26T20:32:01.8415694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8416067Z return func(*args, **kwargs) 2025-08-26T20:32:01.8416465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8416885Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8417283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8417704Z return func(*args, **kwargs) 2025-08-26T20:32:01.8418080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8418453Z return func(*args, **kwargs) 2025-08-26T20:32:01.8418814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8419180Z return func(*args, **kwargs) 2025-08-26T20:32:01.8419571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8420044Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8420505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8420925Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8421067Z 2025-08-26T20:32:01.8421180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8421554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8421892Z return mod(**inputs) 2025-08-26T20:32:01.8422251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8422628Z return func(*args, **kwargs) 2025-08-26T20:32:01.8422991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8423387Z return func(*args, **kwargs) 2025-08-26T20:32:01.8423738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8424099Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8424503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8424915Z outputs = self.layoutlm( 2025-08-26T20:32:01.8425280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8425653Z return func(*args, **kwargs) 2025-08-26T20:32:01.8426047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8426407Z return func(*args, **kwargs) 2025-08-26T20:32:01.8426751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8427097Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8427493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8427884Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8428261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8428663Z return func(*args, **kwargs) 2025-08-26T20:32:01.8429028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8429404Z return func(*args, **kwargs) 2025-08-26T20:32:01.8429764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8430144Z return func(*args, **kwargs) 2025-08-26T20:32:01.8430337Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8430697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8431031Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8431423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8431816Z layer_outputs = layer_module( 2025-08-26T20:32:01.8432196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8432556Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8432919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8433283Z return func(*args, **kwargs) 2025-08-26T20:32:01.8433638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8434000Z return func(*args, **kwargs) 2025-08-26T20:32:01.8434346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8434708Z return func(*args, **kwargs) 2025-08-26T20:32:01.8435092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8435505Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8435923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8436341Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8436806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8437325Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8437830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8438273Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8438426Z 2025-08-26T20:32:01.8438540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8438936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8439365Z return mod(**inputs) 2025-08-26T20:32:01.8439756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8440154Z return func(*args, **kwargs) 2025-08-26T20:32:01.8440583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8440994Z return func(*args, **kwargs) 2025-08-26T20:32:01.8441374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8441763Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8442151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8442539Z outputs = self.layoutlm( 2025-08-26T20:32:01.8442916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8443354Z return func(*args, **kwargs) 2025-08-26T20:32:01.8443750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8444163Z return func(*args, **kwargs) 2025-08-26T20:32:01.8444539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8444928Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8445378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8445889Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8446308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8446723Z return func(*args, **kwargs) 2025-08-26T20:32:01.8447118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8447554Z return func(*args, **kwargs) 2025-08-26T20:32:01.8447954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8448356Z return func(*args, **kwargs) 2025-08-26T20:32:01.8448576Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8448967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8449408Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8449802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8450222Z layer_outputs = layer_module( 2025-08-26T20:32:01.8450563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8450914Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8451271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8451626Z return func(*args, **kwargs) 2025-08-26T20:32:01.8451973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8452382Z return func(*args, **kwargs) 2025-08-26T20:32:01.8452739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8453097Z return func(*args, **kwargs) 2025-08-26T20:32:01.8453477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8453564Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8453822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8453901Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8454196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8454333Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8454600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8454715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8454921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8454999Z return self.act(input) 2025-08-26T20:32:01.8455003Z 2025-08-26T20:32:01.8455106Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8455307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8455393Z return mod(**inputs) 2025-08-26T20:32:01.8455624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8455698Z return func(*args, **kwargs) 2025-08-26T20:32:01.8455930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8456004Z return func(*args, **kwargs) 2025-08-26T20:32:01.8456218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8456292Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8456559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8456630Z outputs = self.layoutlm( 2025-08-26T20:32:01.8456869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8456962Z return func(*args, **kwargs) 2025-08-26T20:32:01.8457205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8457272Z return func(*args, **kwargs) 2025-08-26T20:32:01.8457487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8457573Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8457849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8457932Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8458172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8458250Z return func(*args, **kwargs) 2025-08-26T20:32:01.8458495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8458564Z return func(*args, **kwargs) 2025-08-26T20:32:01.8458808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8458876Z return func(*args, **kwargs) 2025-08-26T20:32:01.8458953Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8459179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8459269Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8459554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8459627Z layer_outputs = layer_module( 2025-08-26T20:32:01.8459843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8459933Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8460165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8460238Z return func(*args, **kwargs) 2025-08-26T20:32:01.8460486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8460559Z return func(*args, **kwargs) 2025-08-26T20:32:01.8460789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8460854Z return func(*args, **kwargs) 2025-08-26T20:32:01.8461121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8461204Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8461466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8461559Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8461846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8461989Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8462248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8462337Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8462341Z 2025-08-26T20:32:01.8462446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8462647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8462711Z return mod(**inputs) 2025-08-26T20:32:01.8462963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8463041Z return func(*args, **kwargs) 2025-08-26T20:32:01.8463274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8463349Z return func(*args, **kwargs) 2025-08-26T20:32:01.8463563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8463637Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8463909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8463982Z outputs = self.layoutlm( 2025-08-26T20:32:01.8464225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8464303Z return func(*args, **kwargs) 2025-08-26T20:32:01.8464532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8464606Z return func(*args, **kwargs) 2025-08-26T20:32:01.8464815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8464895Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8465152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8465257Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8465491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8465555Z return func(*args, **kwargs) 2025-08-26T20:32:01.8465792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8465860Z return func(*args, **kwargs) 2025-08-26T20:32:01.8466102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8466168Z return func(*args, **kwargs) 2025-08-26T20:32:01.8466245Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8466486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8466560Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8466835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8466906Z layer_outputs = layer_module( 2025-08-26T20:32:01.8467126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8467214Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8467458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8467558Z return func(*args, **kwargs) 2025-08-26T20:32:01.8467794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8467859Z return func(*args, **kwargs) 2025-08-26T20:32:01.8468103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8468171Z return func(*args, **kwargs) 2025-08-26T20:32:01.8468442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8468526Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8468767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8468833Z return func(*args, **kwargs) 2025-08-26T20:32:01.8469083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8469160Z return func(*args, **kwargs) 2025-08-26T20:32:01.8469392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8469466Z return func(*args, **kwargs) 2025-08-26T20:32:01.8469727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8469799Z self_outputs = self.self( 2025-08-26T20:32:01.8470043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8470109Z return func(*args, **kwargs) 2025-08-26T20:32:01.8470349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8470418Z return func(*args, **kwargs) 2025-08-26T20:32:01.8470655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8470730Z return func(*args, **kwargs) 2025-08-26T20:32:01.8470992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8471149Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8471153Z 2025-08-26T20:32:01.8471276Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8471488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8471554Z return mod(**inputs) 2025-08-26T20:32:01.8471791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8471868Z return func(*args, **kwargs) 2025-08-26T20:32:01.8472108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8472184Z return func(*args, **kwargs) 2025-08-26T20:32:01.8472399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8472491Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8472768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8472840Z outputs = self.layoutlm( 2025-08-26T20:32:01.8473082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8473151Z return func(*args, **kwargs) 2025-08-26T20:32:01.8473386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8473468Z return func(*args, **kwargs) 2025-08-26T20:32:01.8473701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8473783Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8474050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8474131Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8474370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8474437Z return func(*args, **kwargs) 2025-08-26T20:32:01.8474679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8474747Z return func(*args, **kwargs) 2025-08-26T20:32:01.8474990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8475076Z return func(*args, **kwargs) 2025-08-26T20:32:01.8475155Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8475378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8475453Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8475730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8475802Z layer_outputs = layer_module( 2025-08-26T20:32:01.8476024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8476112Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8476350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8476426Z return func(*args, **kwargs) 2025-08-26T20:32:01.8476665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8476741Z return func(*args, **kwargs) 2025-08-26T20:32:01.8476986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8477055Z return func(*args, **kwargs) 2025-08-26T20:32:01.8477339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8477441Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8477691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8477760Z return func(*args, **kwargs) 2025-08-26T20:32:01.8478005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8478087Z return func(*args, **kwargs) 2025-08-26T20:32:01.8478342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8478423Z return func(*args, **kwargs) 2025-08-26T20:32:01.8478727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8478808Z self_outputs = self.self( 2025-08-26T20:32:01.8479074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8479146Z return func(*args, **kwargs) 2025-08-26T20:32:01.8479486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8479565Z return func(*args, **kwargs) 2025-08-26T20:32:01.8479828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8479939Z return func(*args, **kwargs) 2025-08-26T20:32:01.8480238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8480406Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8480410Z 2025-08-26T20:32:01.8480541Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8480769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8480843Z return mod(**inputs) 2025-08-26T20:32:01.8481098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8481181Z return func(*args, **kwargs) 2025-08-26T20:32:01.8481436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8481540Z return func(*args, **kwargs) 2025-08-26T20:32:01.8481772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8481853Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8482152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8482226Z outputs = self.layoutlm( 2025-08-26T20:32:01.8482478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8482551Z return func(*args, **kwargs) 2025-08-26T20:32:01.8482803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8482873Z return func(*args, **kwargs) 2025-08-26T20:32:01.8483101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8483193Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8483482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8483567Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8483826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8483898Z return func(*args, **kwargs) 2025-08-26T20:32:01.8484175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8484249Z return func(*args, **kwargs) 2025-08-26T20:32:01.8484513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8484584Z return func(*args, **kwargs) 2025-08-26T20:32:01.8484670Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8484922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8484998Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8485294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8485369Z layer_outputs = layer_module( 2025-08-26T20:32:01.8485597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8485689Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8485929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8486005Z return func(*args, **kwargs) 2025-08-26T20:32:01.8486262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8486345Z return func(*args, **kwargs) 2025-08-26T20:32:01.8486646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8486720Z return func(*args, **kwargs) 2025-08-26T20:32:01.8487016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8487106Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8487369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8487441Z return func(*args, **kwargs) 2025-08-26T20:32:01.8487701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8487782Z return func(*args, **kwargs) 2025-08-26T20:32:01.8488042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8488144Z return func(*args, **kwargs) 2025-08-26T20:32:01.8488451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8488529Z self_outputs = self.self( 2025-08-26T20:32:01.8488805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8488876Z return func(*args, **kwargs) 2025-08-26T20:32:01.8489144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8489216Z return func(*args, **kwargs) 2025-08-26T20:32:01.8489491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8489564Z return func(*args, **kwargs) 2025-08-26T20:32:01.8489868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8490040Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8490044Z 2025-08-26T20:32:01.8490133Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8490231Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8490347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8490563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8490665Z return mod(**inputs) 2025-08-26T20:32:01.8490924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8491003Z return func(*args, **kwargs) 2025-08-26T20:32:01.8491271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8491345Z return func(*args, **kwargs) 2025-08-26T20:32:01.8491587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8491667Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8492002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8492079Z outputs = self.layoutlm( 2025-08-26T20:32:01.8492346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8492427Z return func(*args, **kwargs) 2025-08-26T20:32:01.8492692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8492773Z return func(*args, **kwargs) 2025-08-26T20:32:01.8493008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8493114Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8493417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8493497Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8493760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8493832Z return func(*args, **kwargs) 2025-08-26T20:32:01.8494100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8494173Z return func(*args, **kwargs) 2025-08-26T20:32:01.8494434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8494512Z return func(*args, **kwargs) 2025-08-26T20:32:01.8494593Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8494851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8494932Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8495221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8495306Z layer_outputs = layer_module( 2025-08-26T20:32:01.8495543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8495636Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8495891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8495969Z return func(*args, **kwargs) 2025-08-26T20:32:01.8496383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8496468Z return func(*args, **kwargs) 2025-08-26T20:32:01.8496740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8496813Z return func(*args, **kwargs) 2025-08-26T20:32:01.8497116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8497209Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8497517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8497600Z return func(*args, **kwargs) 2025-08-26T20:32:01.8497857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8497937Z return func(*args, **kwargs) 2025-08-26T20:32:01.8498195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8498270Z return func(*args, **kwargs) 2025-08-26T20:32:01.8498566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8498736Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8499033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8499128Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8499131Z 2025-08-26T20:32:01.8499255Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8499475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8499547Z return mod(**inputs) 2025-08-26T20:32:01.8499809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8499914Z return func(*args, **kwargs) 2025-08-26T20:32:01.8500181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8500257Z return func(*args, **kwargs) 2025-08-26T20:32:01.8500494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8500582Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8500870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8500953Z outputs = self.layoutlm( 2025-08-26T20:32:01.8501207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8501280Z return func(*args, **kwargs) 2025-08-26T20:32:01.8501542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8501656Z return func(*args, **kwargs) 2025-08-26T20:32:01.8501886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8501961Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8502241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8502317Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8502560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8502636Z return func(*args, **kwargs) 2025-08-26T20:32:01.8502883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8502963Z return func(*args, **kwargs) 2025-08-26T20:32:01.8503221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8503296Z return func(*args, **kwargs) 2025-08-26T20:32:01.8503388Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8503624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8503707Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8503998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8504074Z layer_outputs = layer_module( 2025-08-26T20:32:01.8504310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8504391Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8504638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8504714Z return func(*args, **kwargs) 2025-08-26T20:32:01.8504967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8505048Z return func(*args, **kwargs) 2025-08-26T20:32:01.8505326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8505408Z return func(*args, **kwargs) 2025-08-26T20:32:01.8505684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8505780Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8506050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8506128Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8506450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8506600Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8506896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8506986Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8506990Z 2025-08-26T20:32:01.8507103Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8507328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8507399Z return mod(**inputs) 2025-08-26T20:32:01.8507664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8507737Z return func(*args, **kwargs) 2025-08-26T20:32:01.8507998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8508096Z return func(*args, **kwargs) 2025-08-26T20:32:01.8508334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8508423Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8508712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8508795Z outputs = self.layoutlm( 2025-08-26T20:32:01.8509054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8509127Z return func(*args, **kwargs) 2025-08-26T20:32:01.8509392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8509464Z return func(*args, **kwargs) 2025-08-26T20:32:01.8509710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8509791Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8510081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8510170Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8510427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8510523Z return func(*args, **kwargs) 2025-08-26T20:32:01.8510781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8510855Z return func(*args, **kwargs) 2025-08-26T20:32:01.8511122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8511197Z return func(*args, **kwargs) 2025-08-26T20:32:01.8511288Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8511521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8511607Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8511912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8511992Z layer_outputs = layer_module( 2025-08-26T20:32:01.8512239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8512324Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8512585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8512658Z return func(*args, **kwargs) 2025-08-26T20:32:01.8512912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8513013Z return func(*args, **kwargs) 2025-08-26T20:32:01.8513269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8513348Z return func(*args, **kwargs) 2025-08-26T20:32:01.8513637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8513729Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8514020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8514102Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8514431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8514585Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8514880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8515005Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8515236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8515319Z return self.act(input) 2025-08-26T20:32:01.8515324Z 2025-08-26T20:32:01.8515436Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8515657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8515728Z return mod(**inputs) 2025-08-26T20:32:01.8515983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8516065Z return func(*args, **kwargs) 2025-08-26T20:32:01.8516321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8516403Z return func(*args, **kwargs) 2025-08-26T20:32:01.8516639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8516721Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8517016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8517119Z outputs = self.layoutlm( 2025-08-26T20:32:01.8517381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8517454Z return func(*args, **kwargs) 2025-08-26T20:32:01.8517717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8517792Z return func(*args, **kwargs) 2025-08-26T20:32:01.8518026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8518112Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8518448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8518536Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8518791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8518865Z return func(*args, **kwargs) 2025-08-26T20:32:01.8519128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8519256Z return func(*args, **kwargs) 2025-08-26T20:32:01.8519534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8519637Z return func(*args, **kwargs) 2025-08-26T20:32:01.8519723Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8519969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8520051Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8520355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8520437Z layer_outputs = layer_module( 2025-08-26T20:32:01.8520695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8520780Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8521033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8521114Z return func(*args, **kwargs) 2025-08-26T20:32:01.8521386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8521471Z return func(*args, **kwargs) 2025-08-26T20:32:01.8521723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8521798Z return func(*args, **kwargs) 2025-08-26T20:32:01.8522095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8522193Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8522489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8522576Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8522896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8523055Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8523342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8523440Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8523445Z 2025-08-26T20:32:01.8523563Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8523783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8523869Z return mod(**inputs) 2025-08-26T20:32:01.8524127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8524210Z return func(*args, **kwargs) 2025-08-26T20:32:01.8524466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8524549Z return func(*args, **kwargs) 2025-08-26T20:32:01.8524783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8524863Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8525191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8525269Z outputs = self.layoutlm( 2025-08-26T20:32:01.8525534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8525608Z return func(*args, **kwargs) 2025-08-26T20:32:01.8525867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8525948Z return func(*args, **kwargs) 2025-08-26T20:32:01.8526183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8526291Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8526580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8526668Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8526928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8527001Z return func(*args, **kwargs) 2025-08-26T20:32:01.8527268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8527341Z return func(*args, **kwargs) 2025-08-26T20:32:01.8527605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8527678Z return func(*args, **kwargs) 2025-08-26T20:32:01.8527759Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8528021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8528103Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8528392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8528470Z layer_outputs = layer_module( 2025-08-26T20:32:01.8528706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8528794Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8529032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8529106Z return func(*args, **kwargs) 2025-08-26T20:32:01.8529344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8529420Z return func(*args, **kwargs) 2025-08-26T20:32:01.8529658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8529726Z return func(*args, **kwargs) 2025-08-26T20:32:01.8530009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8530090Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8530346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8530413Z return func(*args, **kwargs) 2025-08-26T20:32:01.8530653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8530728Z return func(*args, **kwargs) 2025-08-26T20:32:01.8530963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8531042Z return func(*args, **kwargs) 2025-08-26T20:32:01.8531316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8531390Z self_outputs = self.self( 2025-08-26T20:32:01.8531666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8531735Z return func(*args, **kwargs) 2025-08-26T20:32:01.8531975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8532042Z return func(*args, **kwargs) 2025-08-26T20:32:01.8532277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8532352Z return func(*args, **kwargs) 2025-08-26T20:32:01.8532618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8532795Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8532800Z 2025-08-26T20:32:01.8532913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8533143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8533214Z return mod(**inputs) 2025-08-26T20:32:01.8533478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8533561Z return func(*args, **kwargs) 2025-08-26T20:32:01.8533818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8533899Z return func(*args, **kwargs) 2025-08-26T20:32:01.8534137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8534248Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8534527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8534600Z outputs = self.layoutlm( 2025-08-26T20:32:01.8534860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8534928Z return func(*args, **kwargs) 2025-08-26T20:32:01.8535168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8535235Z return func(*args, **kwargs) 2025-08-26T20:32:01.8535450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8535538Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8535799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8535884Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8536122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8536192Z return func(*args, **kwargs) 2025-08-26T20:32:01.8536439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8536508Z return func(*args, **kwargs) 2025-08-26T20:32:01.8536790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8536864Z return func(*args, **kwargs) 2025-08-26T20:32:01.8536948Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8537192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8537273Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8537570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8537647Z layer_outputs = layer_module( 2025-08-26T20:32:01.8537901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8537993Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8538244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8538319Z return func(*args, **kwargs) 2025-08-26T20:32:01.8538551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8538626Z return func(*args, **kwargs) 2025-08-26T20:32:01.8538859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8538943Z return func(*args, **kwargs) 2025-08-26T20:32:01.8539215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8539298Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8539546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8539616Z return func(*args, **kwargs) 2025-08-26T20:32:01.8539859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8539936Z return func(*args, **kwargs) 2025-08-26T20:32:01.8540177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8540253Z return func(*args, **kwargs) 2025-08-26T20:32:01.8540553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8540627Z self_outputs = self.self( 2025-08-26T20:32:01.8540876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8540945Z return func(*args, **kwargs) 2025-08-26T20:32:01.8541199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8541264Z return func(*args, **kwargs) 2025-08-26T20:32:01.8541508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8541575Z return func(*args, **kwargs) 2025-08-26T20:32:01.8541836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8541988Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8541992Z 2025-08-26T20:32:01.8542097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8542305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8542370Z return mod(**inputs) 2025-08-26T20:32:01.8542606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8542680Z return func(*args, **kwargs) 2025-08-26T20:32:01.8542929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8543007Z return func(*args, **kwargs) 2025-08-26T20:32:01.8543220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8543295Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8543563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8543633Z outputs = self.layoutlm( 2025-08-26T20:32:01.8543875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8543960Z return func(*args, **kwargs) 2025-08-26T20:32:01.8544202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8544269Z return func(*args, **kwargs) 2025-08-26T20:32:01.8544483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8544564Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8544829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8544912Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8545160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8545227Z return func(*args, **kwargs) 2025-08-26T20:32:01.8545470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8545538Z return func(*args, **kwargs) 2025-08-26T20:32:01.8545779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8545847Z return func(*args, **kwargs) 2025-08-26T20:32:01.8545924Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8546151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8546226Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8546510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8546614Z layer_outputs = layer_module( 2025-08-26T20:32:01.8546830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8546915Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8547152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8547226Z return func(*args, **kwargs) 2025-08-26T20:32:01.8547463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8547536Z return func(*args, **kwargs) 2025-08-26T20:32:01.8547771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8547837Z return func(*args, **kwargs) 2025-08-26T20:32:01.8548121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8548207Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8548459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8548529Z return func(*args, **kwargs) 2025-08-26T20:32:01.8548771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8548867Z return func(*args, **kwargs) 2025-08-26T20:32:01.8549112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8549187Z return func(*args, **kwargs) 2025-08-26T20:32:01.8549459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8549533Z self_outputs = self.self( 2025-08-26T20:32:01.8549792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8549860Z return func(*args, **kwargs) 2025-08-26T20:32:01.8550117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8550186Z return func(*args, **kwargs) 2025-08-26T20:32:01.8550432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8550499Z return func(*args, **kwargs) 2025-08-26T20:32:01.8550763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8550916Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8550920Z 2025-08-26T20:32:01.8550999Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8551086Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8551209Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8551407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8551480Z return mod(**inputs) 2025-08-26T20:32:01.8551718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8551794Z return func(*args, **kwargs) 2025-08-26T20:32:01.8552027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8552094Z return func(*args, **kwargs) 2025-08-26T20:32:01.8552314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8552388Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8552678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8552751Z outputs = self.layoutlm( 2025-08-26T20:32:01.8552993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8553063Z return func(*args, **kwargs) 2025-08-26T20:32:01.8553294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8553370Z return func(*args, **kwargs) 2025-08-26T20:32:01.8553587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8553669Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8553937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8554012Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8554276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8554350Z return func(*args, **kwargs) 2025-08-26T20:32:01.8554610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8554683Z return func(*args, **kwargs) 2025-08-26T20:32:01.8554937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8555038Z return func(*args, **kwargs) 2025-08-26T20:32:01.8555122Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8555365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8555444Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8555733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8555821Z layer_outputs = layer_module( 2025-08-26T20:32:01.8556061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8556152Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8556425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8556506Z return func(*args, **kwargs) 2025-08-26T20:32:01.8556764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8556836Z return func(*args, **kwargs) 2025-08-26T20:32:01.8557102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8557174Z return func(*args, **kwargs) 2025-08-26T20:32:01.8557471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8557576Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8557834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8557914Z return func(*args, **kwargs) 2025-08-26T20:32:01.8558171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8558249Z return func(*args, **kwargs) 2025-08-26T20:32:01.8558506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8558578Z return func(*args, **kwargs) 2025-08-26T20:32:01.8558876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8559037Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8559594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8559692Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8559697Z 2025-08-26T20:32:01.8559822Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8560039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8560111Z return mod(**inputs) 2025-08-26T20:32:01.8560382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8560458Z return func(*args, **kwargs) 2025-08-26T20:32:01.8560717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8560791Z return func(*args, **kwargs) 2025-08-26T20:32:01.8561026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8561118Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8561404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8561490Z outputs = self.layoutlm( 2025-08-26T20:32:01.8561744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8561835Z return func(*args, **kwargs) 2025-08-26T20:32:01.8562099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8562171Z return func(*args, **kwargs) 2025-08-26T20:32:01.8562413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8562499Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8562798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8562879Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8563158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8563241Z return func(*args, **kwargs) 2025-08-26T20:32:01.8563496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8563578Z return func(*args, **kwargs) 2025-08-26T20:32:01.8563832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8563903Z return func(*args, **kwargs) 2025-08-26T20:32:01.8563995Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8564226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8564344Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8564629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8564707Z layer_outputs = layer_module( 2025-08-26T20:32:01.8564958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8565042Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8565307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8565381Z return func(*args, **kwargs) 2025-08-26T20:32:01.8565633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8565714Z return func(*args, **kwargs) 2025-08-26T20:32:01.8565990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8566072Z return func(*args, **kwargs) 2025-08-26T20:32:01.8566365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8566467Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8566752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8566838Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8567176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8567309Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8567609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8567701Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8567705Z 2025-08-26T20:32:01.8567819Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8568050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8568120Z return mod(**inputs) 2025-08-26T20:32:01.8568374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8568464Z return func(*args, **kwargs) 2025-08-26T20:32:01.8568715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8568785Z return func(*args, **kwargs) 2025-08-26T20:32:01.8569005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8569090Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8569363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8569440Z outputs = self.layoutlm( 2025-08-26T20:32:01.8569698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8569768Z return func(*args, **kwargs) 2025-08-26T20:32:01.8570017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8570086Z return func(*args, **kwargs) 2025-08-26T20:32:01.8570314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8570388Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8570660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8570762Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8571006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8571082Z return func(*args, **kwargs) 2025-08-26T20:32:01.8571325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8571401Z return func(*args, **kwargs) 2025-08-26T20:32:01.8571647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8571714Z return func(*args, **kwargs) 2025-08-26T20:32:01.8571801Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8572022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8572103Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8572393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8572467Z layer_outputs = layer_module( 2025-08-26T20:32:01.8572700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8572782Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8573030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8573099Z return func(*args, **kwargs) 2025-08-26T20:32:01.8573343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8573423Z return func(*args, **kwargs) 2025-08-26T20:32:01.8573686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8573768Z return func(*args, **kwargs) 2025-08-26T20:32:01.8574053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8574144Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8574429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8574510Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8574863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8574992Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8575270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8575390Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8575619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8575705Z return self.act(input) 2025-08-26T20:32:01.8575709Z 2025-08-26T20:32:01.8575821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8576060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8576132Z return mod(**inputs) 2025-08-26T20:32:01.8576399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8576480Z return func(*args, **kwargs) 2025-08-26T20:32:01.8576748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8576828Z return func(*args, **kwargs) 2025-08-26T20:32:01.8577062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8577171Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8577455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8577526Z outputs = self.layoutlm( 2025-08-26T20:32:01.8577776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8577845Z return func(*args, **kwargs) 2025-08-26T20:32:01.8578098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8578167Z return func(*args, **kwargs) 2025-08-26T20:32:01.8578391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8578481Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8578764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8578873Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8579138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8579211Z return func(*args, **kwargs) 2025-08-26T20:32:01.8579489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8579562Z return func(*args, **kwargs) 2025-08-26T20:32:01.8579828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8579902Z return func(*args, **kwargs) 2025-08-26T20:32:01.8579985Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8580236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8580313Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8580593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8580666Z layer_outputs = layer_module( 2025-08-26T20:32:01.8580904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8580991Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8581275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8581358Z return func(*args, **kwargs) 2025-08-26T20:32:01.8581611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8581692Z return func(*args, **kwargs) 2025-08-26T20:32:01.8581946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8582023Z return func(*args, **kwargs) 2025-08-26T20:32:01.8582318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8582406Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8582696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8582779Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8583101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8583255Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8583540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8583641Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8583943Z 2025-08-26T20:32:01.8584058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8584283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8584355Z return mod(**inputs) 2025-08-26T20:32:01.8584614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8584697Z return func(*args, **kwargs) 2025-08-26T20:32:01.8584956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8585038Z return func(*args, **kwargs) 2025-08-26T20:32:01.8585275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8585357Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8585658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8585754Z outputs = self.layoutlm( 2025-08-26T20:32:01.8586018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8586092Z return func(*args, **kwargs) 2025-08-26T20:32:01.8586358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8586431Z return func(*args, **kwargs) 2025-08-26T20:32:01.8586669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8586757Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8587044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8587130Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8587392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8587466Z return func(*args, **kwargs) 2025-08-26T20:32:01.8587730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8587803Z return func(*args, **kwargs) 2025-08-26T20:32:01.8588073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8588161Z return func(*args, **kwargs) 2025-08-26T20:32:01.8588245Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8588488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8588567Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8588874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8588954Z layer_outputs = layer_module( 2025-08-26T20:32:01.8589195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8589285Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8589571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8589652Z return func(*args, **kwargs) 2025-08-26T20:32:01.8589911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8589989Z return func(*args, **kwargs) 2025-08-26T20:32:01.8590244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8590317Z return func(*args, **kwargs) 2025-08-26T20:32:01.8590627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8590737Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8590999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8591072Z return func(*args, **kwargs) 2025-08-26T20:32:01.8591325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8591405Z return func(*args, **kwargs) 2025-08-26T20:32:01.8591661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8591740Z return func(*args, **kwargs) 2025-08-26T20:32:01.8592028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8592104Z self_outputs = self.self( 2025-08-26T20:32:01.8592382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8592455Z return func(*args, **kwargs) 2025-08-26T20:32:01.8592715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8592790Z return func(*args, **kwargs) 2025-08-26T20:32:01.8593049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8593123Z return func(*args, **kwargs) 2025-08-26T20:32:01.8593411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8593581Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8593585Z 2025-08-26T20:32:01.8593700Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8593928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8594002Z return mod(**inputs) 2025-08-26T20:32:01.8594260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8594341Z return func(*args, **kwargs) 2025-08-26T20:32:01.8594594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8594673Z return func(*args, **kwargs) 2025-08-26T20:32:01.8594925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8595009Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8595305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8595384Z outputs = self.layoutlm( 2025-08-26T20:32:01.8595650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8595724Z return func(*args, **kwargs) 2025-08-26T20:32:01.8595987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8596076Z return func(*args, **kwargs) 2025-08-26T20:32:01.8596480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8596581Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8596893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8596987Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8597256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8597336Z return func(*args, **kwargs) 2025-08-26T20:32:01.8597648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8597722Z return func(*args, **kwargs) 2025-08-26T20:32:01.8597992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8598065Z return func(*args, **kwargs) 2025-08-26T20:32:01.8598150Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8598400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8598483Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8598789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8598867Z layer_outputs = layer_module( 2025-08-26T20:32:01.8599138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8599284Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8599556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8599643Z return func(*args, **kwargs) 2025-08-26T20:32:01.8599903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8599986Z return func(*args, **kwargs) 2025-08-26T20:32:01.8600247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8600322Z return func(*args, **kwargs) 2025-08-26T20:32:01.8600625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8600717Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8600988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8601063Z return func(*args, **kwargs) 2025-08-26T20:32:01.8601321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8601403Z return func(*args, **kwargs) 2025-08-26T20:32:01.8601659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8601773Z return func(*args, **kwargs) 2025-08-26T20:32:01.8602071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8602150Z self_outputs = self.self( 2025-08-26T20:32:01.8602421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8602499Z return func(*args, **kwargs) 2025-08-26T20:32:01.8602775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8602849Z return func(*args, **kwargs) 2025-08-26T20:32:01.8603145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8603221Z return func(*args, **kwargs) 2025-08-26T20:32:01.8603522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8603685Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8603690Z 2025-08-26T20:32:01.8603806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8604038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8604115Z return mod(**inputs) 2025-08-26T20:32:01.8604392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8604473Z return func(*args, **kwargs) 2025-08-26T20:32:01.8604738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8604819Z return func(*args, **kwargs) 2025-08-26T20:32:01.8605054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8605139Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8605445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8605526Z outputs = self.layoutlm( 2025-08-26T20:32:01.8605792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8605887Z return func(*args, **kwargs) 2025-08-26T20:32:01.8606158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8606233Z return func(*args, **kwargs) 2025-08-26T20:32:01.8606473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8606565Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8606867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8606957Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8607222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8607297Z return func(*args, **kwargs) 2025-08-26T20:32:01.8607568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8607646Z return func(*args, **kwargs) 2025-08-26T20:32:01.8607916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8607989Z return func(*args, **kwargs) 2025-08-26T20:32:01.8608077Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8608327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8608409Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8608738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8608819Z layer_outputs = layer_module( 2025-08-26T20:32:01.8609072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8609163Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8609424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8609507Z return func(*args, **kwargs) 2025-08-26T20:32:01.8609794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8609879Z return func(*args, **kwargs) 2025-08-26T20:32:01.8610140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8610216Z return func(*args, **kwargs) 2025-08-26T20:32:01.8610533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8610620Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8610881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8610974Z return func(*args, **kwargs) 2025-08-26T20:32:01.8611234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8611315Z return func(*args, **kwargs) 2025-08-26T20:32:01.8611576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8611656Z return func(*args, **kwargs) 2025-08-26T20:32:01.8611951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8612029Z self_outputs = self.self( 2025-08-26T20:32:01.8612309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8612380Z return func(*args, **kwargs) 2025-08-26T20:32:01.8612650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8612742Z return func(*args, **kwargs) 2025-08-26T20:32:01.8613016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8613088Z return func(*args, **kwargs) 2025-08-26T20:32:01.8613392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8613560Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8613565Z 2025-08-26T20:32:01.8613655Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8613748Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8613870Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8614102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8614182Z return mod(**inputs) 2025-08-26T20:32:01.8614451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8614533Z return func(*args, **kwargs) 2025-08-26T20:32:01.8614802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8614876Z return func(*args, **kwargs) 2025-08-26T20:32:01.8615120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8615224Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8615527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8615604Z outputs = self.layoutlm( 2025-08-26T20:32:01.8615878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8615953Z return func(*args, **kwargs) 2025-08-26T20:32:01.8616207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8616287Z return func(*args, **kwargs) 2025-08-26T20:32:01.8616538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8616628Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8616930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8617014Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8617287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8617360Z return func(*args, **kwargs) 2025-08-26T20:32:01.8617626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8617717Z return func(*args, **kwargs) 2025-08-26T20:32:01.8617981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8618060Z return func(*args, **kwargs) 2025-08-26T20:32:01.8618145Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8618387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8618465Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8618756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8618842Z layer_outputs = layer_module( 2025-08-26T20:32:01.8619079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8619172Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8619463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8619545Z return func(*args, **kwargs) 2025-08-26T20:32:01.8619807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8619880Z return func(*args, **kwargs) 2025-08-26T20:32:01.8620142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8620215Z return func(*args, **kwargs) 2025-08-26T20:32:01.8620521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8620609Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8620875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8620957Z return func(*args, **kwargs) 2025-08-26T20:32:01.8621210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8621290Z return func(*args, **kwargs) 2025-08-26T20:32:01.8621556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8621627Z return func(*args, **kwargs) 2025-08-26T20:32:01.8621974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8622119Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8622485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8622577Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8622586Z 2025-08-26T20:32:01.8622706Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8622924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8622994Z return mod(**inputs) 2025-08-26T20:32:01.8623275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8623349Z return func(*args, **kwargs) 2025-08-26T20:32:01.8623613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8623696Z return func(*args, **kwargs) 2025-08-26T20:32:01.8623915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8624001Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8624270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8624370Z outputs = self.layoutlm( 2025-08-26T20:32:01.8624614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8624682Z return func(*args, **kwargs) 2025-08-26T20:32:01.8624934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8625004Z return func(*args, **kwargs) 2025-08-26T20:32:01.8625234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8625311Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8625594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8625669Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8625912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8626009Z return func(*args, **kwargs) 2025-08-26T20:32:01.8626251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8626326Z return func(*args, **kwargs) 2025-08-26T20:32:01.8626571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8626639Z return func(*args, **kwargs) 2025-08-26T20:32:01.8626729Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8626952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8627036Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8627312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8627389Z layer_outputs = layer_module( 2025-08-26T20:32:01.8627623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8627705Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8627956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8628026Z return func(*args, **kwargs) 2025-08-26T20:32:01.8628291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8628362Z return func(*args, **kwargs) 2025-08-26T20:32:01.8628606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8628681Z return func(*args, **kwargs) 2025-08-26T20:32:01.8628950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8629051Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8629320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8629398Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8629731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8629857Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8630137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8630223Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8630226Z 2025-08-26T20:32:01.8630339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8630542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8630628Z return mod(**inputs) 2025-08-26T20:32:01.8630877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8630947Z return func(*args, **kwargs) 2025-08-26T20:32:01.8631195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8631264Z return func(*args, **kwargs) 2025-08-26T20:32:01.8631484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8631569Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8631838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8631917Z outputs = self.layoutlm( 2025-08-26T20:32:01.8632161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8632252Z return func(*args, **kwargs) 2025-08-26T20:32:01.8632514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8632587Z return func(*args, **kwargs) 2025-08-26T20:32:01.8632825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8632907Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8633200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8633285Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8633540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8633621Z return func(*args, **kwargs) 2025-08-26T20:32:01.8633875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8633957Z return func(*args, **kwargs) 2025-08-26T20:32:01.8634207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8634280Z return func(*args, **kwargs) 2025-08-26T20:32:01.8634372Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8634602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8634704Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8634992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8635068Z layer_outputs = layer_module( 2025-08-26T20:32:01.8635310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8635398Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8635659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8635732Z return func(*args, **kwargs) 2025-08-26T20:32:01.8636006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8636085Z return func(*args, **kwargs) 2025-08-26T20:32:01.8636340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8636419Z return func(*args, **kwargs) 2025-08-26T20:32:01.8636707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8636803Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8637083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8637183Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8637513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8637647Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8637941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8638067Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8638296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8638377Z return self.act(input) 2025-08-26T20:32:01.8638381Z 2025-08-26T20:32:01.8638493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8638740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8638815Z return mod(**inputs) 2025-08-26T20:32:01.8639069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8639151Z return func(*args, **kwargs) 2025-08-26T20:32:01.8639478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8639569Z return func(*args, **kwargs) 2025-08-26T20:32:01.8639810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8639899Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8640192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8640268Z outputs = self.layoutlm( 2025-08-26T20:32:01.8640536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8640611Z return func(*args, **kwargs) 2025-08-26T20:32:01.8640875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8640949Z return func(*args, **kwargs) 2025-08-26T20:32:01.8641179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8641271Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8641585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8641677Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8641935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8642011Z return func(*args, **kwargs) 2025-08-26T20:32:01.8642274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8642349Z return func(*args, **kwargs) 2025-08-26T20:32:01.8642629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8642703Z return func(*args, **kwargs) 2025-08-26T20:32:01.8642794Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8643031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8643112Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8643410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8643488Z layer_outputs = layer_module( 2025-08-26T20:32:01.8643735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8643838Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8644095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8644176Z return func(*args, **kwargs) 2025-08-26T20:32:01.8644437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8644517Z return func(*args, **kwargs) 2025-08-26T20:32:01.8644773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8644845Z return func(*args, **kwargs) 2025-08-26T20:32:01.8645143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8645233Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8645540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8645624Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8645958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8646105Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8646395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8646495Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8646499Z 2025-08-26T20:32:01.8646613Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8646837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8646908Z return mod(**inputs) 2025-08-26T20:32:01.8647165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8647250Z return func(*args, **kwargs) 2025-08-26T20:32:01.8647507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8647589Z return func(*args, **kwargs) 2025-08-26T20:32:01.8647825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8647924Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8648219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8648294Z outputs = self.layoutlm( 2025-08-26T20:32:01.8648553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8648624Z return func(*args, **kwargs) 2025-08-26T20:32:01.8648874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8648941Z return func(*args, **kwargs) 2025-08-26T20:32:01.8649176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8649261Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8649534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8649617Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8649858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8649927Z return func(*args, **kwargs) 2025-08-26T20:32:01.8650175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8650265Z return func(*args, **kwargs) 2025-08-26T20:32:01.8650513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8650581Z return func(*args, **kwargs) 2025-08-26T20:32:01.8650659Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8650886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8650960Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8651240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8651313Z layer_outputs = layer_module( 2025-08-26T20:32:01.8651539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8651624Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8651883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8651961Z return func(*args, **kwargs) 2025-08-26T20:32:01.8652200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8652277Z return func(*args, **kwargs) 2025-08-26T20:32:01.8652515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8652586Z return func(*args, **kwargs) 2025-08-26T20:32:01.8652862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8652945Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8653193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8653261Z return func(*args, **kwargs) 2025-08-26T20:32:01.8653502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8653578Z return func(*args, **kwargs) 2025-08-26T20:32:01.8653818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8653894Z return func(*args, **kwargs) 2025-08-26T20:32:01.8654185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8654260Z self_outputs = self.self( 2025-08-26T20:32:01.8654510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8654581Z return func(*args, **kwargs) 2025-08-26T20:32:01.8654830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8654902Z return func(*args, **kwargs) 2025-08-26T20:32:01.8655154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8655223Z return func(*args, **kwargs) 2025-08-26T20:32:01.8655510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8655670Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8655675Z 2025-08-26T20:32:01.8655784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8655994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8656063Z return mod(**inputs) 2025-08-26T20:32:01.8656306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8656384Z return func(*args, **kwargs) 2025-08-26T20:32:01.8656644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8656719Z return func(*args, **kwargs) 2025-08-26T20:32:01.8656940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8657016Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8657294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8657367Z outputs = self.layoutlm( 2025-08-26T20:32:01.8657616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8657686Z return func(*args, **kwargs) 2025-08-26T20:32:01.8657938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8658025Z return func(*args, **kwargs) 2025-08-26T20:32:01.8658244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8658330Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8658604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8658699Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8658942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8659010Z return func(*args, **kwargs) 2025-08-26T20:32:01.8659256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8659324Z return func(*args, **kwargs) 2025-08-26T20:32:01.8659569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8659640Z return func(*args, **kwargs) 2025-08-26T20:32:01.8659719Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8659948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8660024Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8660305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8660396Z layer_outputs = layer_module( 2025-08-26T20:32:01.8660631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8660710Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8660952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8661031Z return func(*args, **kwargs) 2025-08-26T20:32:01.8661274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8661350Z return func(*args, **kwargs) 2025-08-26T20:32:01.8661605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8661674Z return func(*args, **kwargs) 2025-08-26T20:32:01.8661959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8662043Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8662296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8662362Z return func(*args, **kwargs) 2025-08-26T20:32:01.8662605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8662711Z return func(*args, **kwargs) 2025-08-26T20:32:01.8662951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8663028Z return func(*args, **kwargs) 2025-08-26T20:32:01.8663309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8663392Z self_outputs = self.self( 2025-08-26T20:32:01.8663651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8663724Z return func(*args, **kwargs) 2025-08-26T20:32:01.8663986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8664058Z return func(*args, **kwargs) 2025-08-26T20:32:01.8664337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8664412Z return func(*args, **kwargs) 2025-08-26T20:32:01.8664700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8664859Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8664864Z 2025-08-26T20:32:01.8664979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8665202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8665275Z return mod(**inputs) 2025-08-26T20:32:01.8665530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8665612Z return func(*args, **kwargs) 2025-08-26T20:32:01.8665870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8665949Z return func(*args, **kwargs) 2025-08-26T20:32:01.8666170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8666252Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8666526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8666597Z outputs = self.layoutlm( 2025-08-26T20:32:01.8666865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8666935Z return func(*args, **kwargs) 2025-08-26T20:32:01.8667184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8667251Z return func(*args, **kwargs) 2025-08-26T20:32:01.8667468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8667554Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8667824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8667925Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8668169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8668237Z return func(*args, **kwargs) 2025-08-26T20:32:01.8668486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8668555Z return func(*args, **kwargs) 2025-08-26T20:32:01.8668801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8668867Z return func(*args, **kwargs) 2025-08-26T20:32:01.8668947Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8669191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8669266Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8669546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8669619Z layer_outputs = layer_module( 2025-08-26T20:32:01.8669852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8669933Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8670173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8670249Z return func(*args, **kwargs) 2025-08-26T20:32:01.8670487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8670583Z return func(*args, **kwargs) 2025-08-26T20:32:01.8670824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8670893Z return func(*args, **kwargs) 2025-08-26T20:32:01.8671173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8671259Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8671512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8671580Z return func(*args, **kwargs) 2025-08-26T20:32:01.8671821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8671898Z return func(*args, **kwargs) 2025-08-26T20:32:01.8672142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8672221Z return func(*args, **kwargs) 2025-08-26T20:32:01.8672491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8672571Z self_outputs = self.self( 2025-08-26T20:32:01.8672812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8672881Z return func(*args, **kwargs) 2025-08-26T20:32:01.8673166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8673237Z return func(*args, **kwargs) 2025-08-26T20:32:01.8673486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8673556Z return func(*args, **kwargs) 2025-08-26T20:32:01.8673827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8673987Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8673991Z 2025-08-26T20:32:01.8674088Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8674178Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8674285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8674496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8674577Z return mod(**inputs) 2025-08-26T20:32:01.8674835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8674914Z return func(*args, **kwargs) 2025-08-26T20:32:01.8675168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8675260Z return func(*args, **kwargs) 2025-08-26T20:32:01.8675505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8675586Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8675888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8675964Z outputs = self.layoutlm( 2025-08-26T20:32:01.8676231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8676304Z return func(*args, **kwargs) 2025-08-26T20:32:01.8676564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8676649Z return func(*args, **kwargs) 2025-08-26T20:32:01.8676887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8676996Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8677287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8677367Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8677632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8677706Z return func(*args, **kwargs) 2025-08-26T20:32:01.8677970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8678043Z return func(*args, **kwargs) 2025-08-26T20:32:01.8678297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8678377Z return func(*args, **kwargs) 2025-08-26T20:32:01.8678462Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8678703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8678782Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8679078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8679156Z layer_outputs = layer_module( 2025-08-26T20:32:01.8679495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8679597Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8679853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8679934Z return func(*args, **kwargs) 2025-08-26T20:32:01.8680202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8680283Z return func(*args, **kwargs) 2025-08-26T20:32:01.8680554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8680639Z return func(*args, **kwargs) 2025-08-26T20:32:01.8680956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8681047Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8681307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8681388Z return func(*args, **kwargs) 2025-08-26T20:32:01.8681641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8681722Z return func(*args, **kwargs) 2025-08-26T20:32:01.8681982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8682073Z return func(*args, **kwargs) 2025-08-26T20:32:01.8682374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8682522Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8682818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8682911Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8682915Z 2025-08-26T20:32:01.8683036Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8683262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8683332Z return mod(**inputs) 2025-08-26T20:32:01.8683604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8683695Z return func(*args, **kwargs) 2025-08-26T20:32:01.8683961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8684034Z return func(*args, **kwargs) 2025-08-26T20:32:01.8684266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8684354Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8684642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8684728Z outputs = self.layoutlm( 2025-08-26T20:32:01.8684990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8685071Z return func(*args, **kwargs) 2025-08-26T20:32:01.8685337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8685410Z return func(*args, **kwargs) 2025-08-26T20:32:01.8685651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8685734Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8686028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8686125Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8686394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8686476Z return func(*args, **kwargs) 2025-08-26T20:32:01.8686741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8686823Z return func(*args, **kwargs) 2025-08-26T20:32:01.8687087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8687159Z return func(*args, **kwargs) 2025-08-26T20:32:01.8687251Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8687506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8687595Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8687885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8687963Z layer_outputs = layer_module( 2025-08-26T20:32:01.8688210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8688294Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8688564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8688656Z return func(*args, **kwargs) 2025-08-26T20:32:01.8688930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8689003Z return func(*args, **kwargs) 2025-08-26T20:32:01.8689272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8689352Z return func(*args, **kwargs) 2025-08-26T20:32:01.8689641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8689741Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8690021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8690104Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8690462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8690599Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8690894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8690983Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8690986Z 2025-08-26T20:32:01.8691108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8691334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8691406Z return mod(**inputs) 2025-08-26T20:32:01.8691672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8691744Z return func(*args, **kwargs) 2025-08-26T20:32:01.8692019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8692092Z return func(*args, **kwargs) 2025-08-26T20:32:01.8692324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8692416Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8692704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8692805Z outputs = self.layoutlm( 2025-08-26T20:32:01.8693074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8693146Z return func(*args, **kwargs) 2025-08-26T20:32:01.8693419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8693495Z return func(*args, **kwargs) 2025-08-26T20:32:01.8693736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8693815Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8694127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8694210Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8694479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8694561Z return func(*args, **kwargs) 2025-08-26T20:32:01.8694816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8694894Z return func(*args, **kwargs) 2025-08-26T20:32:01.8695159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8695235Z return func(*args, **kwargs) 2025-08-26T20:32:01.8695345Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8695581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8695666Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8695957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8696034Z layer_outputs = layer_module( 2025-08-26T20:32:01.8696481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8696572Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8696850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8696925Z return func(*args, **kwargs) 2025-08-26T20:32:01.8697243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8697326Z return func(*args, **kwargs) 2025-08-26T20:32:01.8697593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8697675Z return func(*args, **kwargs) 2025-08-26T20:32:01.8697964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8698064Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8698345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8698427Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8698758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8698895Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8699188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8699311Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8699544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8699627Z return self.act(input) 2025-08-26T20:32:01.8699631Z 2025-08-26T20:32:01.8699775Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8700000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8700074Z return mod(**inputs) 2025-08-26T20:32:01.8700344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8700420Z return func(*args, **kwargs) 2025-08-26T20:32:01.8700679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8700758Z return func(*args, **kwargs) 2025-08-26T20:32:01.8701006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8701093Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8701374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8701443Z outputs = self.layoutlm( 2025-08-26T20:32:01.8701684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8701751Z return func(*args, **kwargs) 2025-08-26T20:32:01.8701993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8702061Z return func(*args, **kwargs) 2025-08-26T20:32:01.8702298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8702380Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8702646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8702727Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8702967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8703034Z return func(*args, **kwargs) 2025-08-26T20:32:01.8703276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8703344Z return func(*args, **kwargs) 2025-08-26T20:32:01.8703635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8703727Z return func(*args, **kwargs) 2025-08-26T20:32:01.8703814Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8704036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8704112Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8704395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8704470Z layer_outputs = layer_module( 2025-08-26T20:32:01.8704704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8704787Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8705042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8705120Z return func(*args, **kwargs) 2025-08-26T20:32:01.8705354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8705428Z return func(*args, **kwargs) 2025-08-26T20:32:01.8705663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8705729Z return func(*args, **kwargs) 2025-08-26T20:32:01.8706017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8706104Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8706365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8706440Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8706744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8706885Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8707154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8707265Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8707269Z 2025-08-26T20:32:01.8707381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8707608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8707681Z return mod(**inputs) 2025-08-26T20:32:01.8707936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8708018Z return func(*args, **kwargs) 2025-08-26T20:32:01.8708271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8708375Z return func(*args, **kwargs) 2025-08-26T20:32:01.8708615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8708696Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8708999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8709074Z outputs = self.layoutlm( 2025-08-26T20:32:01.8709341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8709410Z return func(*args, **kwargs) 2025-08-26T20:32:01.8709662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8709730Z return func(*args, **kwargs) 2025-08-26T20:32:01.8709951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8710053Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8710328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8710410Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8710653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8710720Z return func(*args, **kwargs) 2025-08-26T20:32:01.8710969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8711039Z return func(*args, **kwargs) 2025-08-26T20:32:01.8711288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8711358Z return func(*args, **kwargs) 2025-08-26T20:32:01.8711438Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8711667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8711751Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8712021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8712093Z layer_outputs = layer_module( 2025-08-26T20:32:01.8712342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8712424Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8712659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8712734Z return func(*args, **kwargs) 2025-08-26T20:32:01.8712970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8713049Z return func(*args, **kwargs) 2025-08-26T20:32:01.8713289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8713357Z return func(*args, **kwargs) 2025-08-26T20:32:01.8713651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8713736Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8713985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8714053Z return func(*args, **kwargs) 2025-08-26T20:32:01.8714307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8714387Z return func(*args, **kwargs) 2025-08-26T20:32:01.8714643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8714743Z return func(*args, **kwargs) 2025-08-26T20:32:01.8715038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8715120Z self_outputs = self.self( 2025-08-26T20:32:01.8715389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8715457Z return func(*args, **kwargs) 2025-08-26T20:32:01.8715714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8715783Z return func(*args, **kwargs) 2025-08-26T20:32:01.8716034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8716102Z return func(*args, **kwargs) 2025-08-26T20:32:01.8716389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8716550Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8716554Z 2025-08-26T20:32:01.8716661Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8716874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8716942Z return mod(**inputs) 2025-08-26T20:32:01.8717187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8717268Z return func(*args, **kwargs) 2025-08-26T20:32:01.8717519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8717598Z return func(*args, **kwargs) 2025-08-26T20:32:01.8717831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8717920Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8718208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8718286Z outputs = self.layoutlm( 2025-08-26T20:32:01.8718549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8718622Z return func(*args, **kwargs) 2025-08-26T20:32:01.8718900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8718973Z return func(*args, **kwargs) 2025-08-26T20:32:01.8719259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8719357Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8719650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8719744Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8720007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8720109Z return func(*args, **kwargs) 2025-08-26T20:32:01.8720389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8720467Z return func(*args, **kwargs) 2025-08-26T20:32:01.8720729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8720801Z return func(*args, **kwargs) 2025-08-26T20:32:01.8720885Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8721130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8721235Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8721546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8721623Z layer_outputs = layer_module( 2025-08-26T20:32:01.8721857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8721938Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8722181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8722259Z return func(*args, **kwargs) 2025-08-26T20:32:01.8722499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8722573Z return func(*args, **kwargs) 2025-08-26T20:32:01.8722813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8722935Z return func(*args, **kwargs) 2025-08-26T20:32:01.8723218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8723306Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8723561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8723629Z return func(*args, **kwargs) 2025-08-26T20:32:01.8723872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8723949Z return func(*args, **kwargs) 2025-08-26T20:32:01.8724192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8724268Z return func(*args, **kwargs) 2025-08-26T20:32:01.8724543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8724624Z self_outputs = self.self( 2025-08-26T20:32:01.8724868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8724937Z return func(*args, **kwargs) 2025-08-26T20:32:01.8725186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8725269Z return func(*args, **kwargs) 2025-08-26T20:32:01.8725521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8725590Z return func(*args, **kwargs) 2025-08-26T20:32:01.8725860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8726014Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8726019Z 2025-08-26T20:32:01.8726126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8726338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8726421Z return mod(**inputs) 2025-08-26T20:32:01.8726666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8726745Z return func(*args, **kwargs) 2025-08-26T20:32:01.8726985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8727060Z return func(*args, **kwargs) 2025-08-26T20:32:01.8727283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8727366Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8727639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8727727Z outputs = self.layoutlm( 2025-08-26T20:32:01.8727976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8728045Z return func(*args, **kwargs) 2025-08-26T20:32:01.8728294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8728363Z return func(*args, **kwargs) 2025-08-26T20:32:01.8728581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8728666Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8728932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8729032Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8729274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8729342Z return func(*args, **kwargs) 2025-08-26T20:32:01.8729592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8729662Z return func(*args, **kwargs) 2025-08-26T20:32:01.8729910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8729978Z return func(*args, **kwargs) 2025-08-26T20:32:01.8730057Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8730286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8730362Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8730642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8730716Z layer_outputs = layer_module( 2025-08-26T20:32:01.8730947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8731029Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8731271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8731348Z return func(*args, **kwargs) 2025-08-26T20:32:01.8731605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8731686Z return func(*args, **kwargs) 2025-08-26T20:32:01.8731931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8731998Z return func(*args, **kwargs) 2025-08-26T20:32:01.8732281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8732369Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8732637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8732708Z return func(*args, **kwargs) 2025-08-26T20:32:01.8732957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8733039Z return func(*args, **kwargs) 2025-08-26T20:32:01.8733301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8733383Z return func(*args, **kwargs) 2025-08-26T20:32:01.8733675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8733778Z self_outputs = self.self( 2025-08-26T20:32:01.8734035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8734107Z return func(*args, **kwargs) 2025-08-26T20:32:01.8734373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8734446Z return func(*args, **kwargs) 2025-08-26T20:32:01.8734708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8734780Z return func(*args, **kwargs) 2025-08-26T20:32:01.8735067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8735235Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8735265Z 2025-08-26T20:32:01.8735355Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8735459Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8735563Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8735762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8735838Z return mod(**inputs) 2025-08-26T20:32:01.8736082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8736153Z return func(*args, **kwargs) 2025-08-26T20:32:01.8736382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8736454Z return func(*args, **kwargs) 2025-08-26T20:32:01.8736664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8736737Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8737007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8737077Z outputs = self.layoutlm( 2025-08-26T20:32:01.8737318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8737386Z return func(*args, **kwargs) 2025-08-26T20:32:01.8737620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8737709Z return func(*args, **kwargs) 2025-08-26T20:32:01.8737922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8738004Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8738268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8738347Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8738599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8738667Z return func(*args, **kwargs) 2025-08-26T20:32:01.8738932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8739003Z return func(*args, **kwargs) 2025-08-26T20:32:01.8739244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8739319Z return func(*args, **kwargs) 2025-08-26T20:32:01.8739398Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8739627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8739701Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8739977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8740069Z layer_outputs = layer_module( 2025-08-26T20:32:01.8740289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8740379Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8740620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8740696Z return func(*args, **kwargs) 2025-08-26T20:32:01.8740942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8741006Z return func(*args, **kwargs) 2025-08-26T20:32:01.8741239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8741302Z return func(*args, **kwargs) 2025-08-26T20:32:01.8741579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8741660Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8741886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8741956Z return func(*args, **kwargs) 2025-08-26T20:32:01.8742195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8742270Z return func(*args, **kwargs) 2025-08-26T20:32:01.8742508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8742583Z return func(*args, **kwargs) 2025-08-26T20:32:01.8742853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8742987Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8743265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8743352Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8743355Z 2025-08-26T20:32:01.8743468Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8743670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8743753Z return mod(**inputs) 2025-08-26T20:32:01.8744012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8744078Z return func(*args, **kwargs) 2025-08-26T20:32:01.8744318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8744387Z return func(*args, **kwargs) 2025-08-26T20:32:01.8744608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8744689Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8744962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8745040Z outputs = self.layoutlm( 2025-08-26T20:32:01.8745277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8745354Z return func(*args, **kwargs) 2025-08-26T20:32:01.8745587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8745655Z return func(*args, **kwargs) 2025-08-26T20:32:01.8745880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8745959Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8746252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8746327Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8746567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8746645Z return func(*args, **kwargs) 2025-08-26T20:32:01.8746885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8746960Z return func(*args, **kwargs) 2025-08-26T20:32:01.8747200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8747269Z return func(*args, **kwargs) 2025-08-26T20:32:01.8747356Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8747590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8747681Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8747945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8748017Z layer_outputs = layer_module( 2025-08-26T20:32:01.8748243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8748323Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8748566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8748633Z return func(*args, **kwargs) 2025-08-26T20:32:01.8748878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8748950Z return func(*args, **kwargs) 2025-08-26T20:32:01.8749189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8749267Z return func(*args, **kwargs) 2025-08-26T20:32:01.8749539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8749632Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8749911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8749993Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8750307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8750433Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8750710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8750798Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8750802Z 2025-08-26T20:32:01.8750917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8751139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8751209Z return mod(**inputs) 2025-08-26T20:32:01.8751467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8751534Z return func(*args, **kwargs) 2025-08-26T20:32:01.8751775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8751844Z return func(*args, **kwargs) 2025-08-26T20:32:01.8752062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8752147Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8752436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8752515Z outputs = self.layoutlm( 2025-08-26T20:32:01.8752755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8752824Z return func(*args, **kwargs) 2025-08-26T20:32:01.8753073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8753141Z return func(*args, **kwargs) 2025-08-26T20:32:01.8753368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8753445Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8753723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8753819Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8754060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8754140Z return func(*args, **kwargs) 2025-08-26T20:32:01.8754379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8754453Z return func(*args, **kwargs) 2025-08-26T20:32:01.8754698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8754766Z return func(*args, **kwargs) 2025-08-26T20:32:01.8754851Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8755073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8755158Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8755433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8755505Z layer_outputs = layer_module( 2025-08-26T20:32:01.8755752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8755838Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8756115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8756188Z return func(*args, **kwargs) 2025-08-26T20:32:01.8756442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8756522Z return func(*args, **kwargs) 2025-08-26T20:32:01.8756776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8756858Z return func(*args, **kwargs) 2025-08-26T20:32:01.8757145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8757242Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8757537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8757621Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8757951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8758085Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8758380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8758502Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8758760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8758843Z return self.act(input) 2025-08-26T20:32:01.8758847Z 2025-08-26T20:32:01.8758958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8759184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8759329Z return mod(**inputs) 2025-08-26T20:32:01.8759614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8759689Z return func(*args, **kwargs) 2025-08-26T20:32:01.8759959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8760042Z return func(*args, **kwargs) 2025-08-26T20:32:01.8760285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8760401Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8760698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8760777Z outputs = self.layoutlm( 2025-08-26T20:32:01.8761047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8761119Z return func(*args, **kwargs) 2025-08-26T20:32:01.8761372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8761442Z return func(*args, **kwargs) 2025-08-26T20:32:01.8761662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8761746Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8762022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8762113Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8762369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8762452Z return func(*args, **kwargs) 2025-08-26T20:32:01.8762707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8762780Z return func(*args, **kwargs) 2025-08-26T20:32:01.8763062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8763138Z return func(*args, **kwargs) 2025-08-26T20:32:01.8763229Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8763464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8763545Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8763849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8763925Z layer_outputs = layer_module( 2025-08-26T20:32:01.8764188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8764274Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8764542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8764618Z return func(*args, **kwargs) 2025-08-26T20:32:01.8764859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8764936Z return func(*args, **kwargs) 2025-08-26T20:32:01.8765182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8765275Z return func(*args, **kwargs) 2025-08-26T20:32:01.8765583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8765673Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8765961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8766044Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8766376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8766525Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8766811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8766939Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8766945Z 2025-08-26T20:32:01.8767052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8767266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8767334Z return mod(**inputs) 2025-08-26T20:32:01.8767577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8767654Z return func(*args, **kwargs) 2025-08-26T20:32:01.8767895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8767970Z return func(*args, **kwargs) 2025-08-26T20:32:01.8768190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8768272Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8768545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8768619Z outputs = self.layoutlm( 2025-08-26T20:32:01.8768869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8768938Z return func(*args, **kwargs) 2025-08-26T20:32:01.8769188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8769275Z return func(*args, **kwargs) 2025-08-26T20:32:01.8769496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8769580Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8769849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8769933Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8770178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8770246Z return func(*args, **kwargs) 2025-08-26T20:32:01.8770512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8770584Z return func(*args, **kwargs) 2025-08-26T20:32:01.8770829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8770899Z return func(*args, **kwargs) 2025-08-26T20:32:01.8770979Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8771203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8771278Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8771570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8771663Z layer_outputs = layer_module( 2025-08-26T20:32:01.8771917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8771998Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8772239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8772316Z return func(*args, **kwargs) 2025-08-26T20:32:01.8772558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8772638Z return func(*args, **kwargs) 2025-08-26T20:32:01.8772892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8772962Z return func(*args, **kwargs) 2025-08-26T20:32:01.8773276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8773367Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8773630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8773702Z return func(*args, **kwargs) 2025-08-26T20:32:01.8773957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8774037Z return func(*args, **kwargs) 2025-08-26T20:32:01.8774290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8774367Z return func(*args, **kwargs) 2025-08-26T20:32:01.8774658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8774743Z self_outputs = self.self( 2025-08-26T20:32:01.8775002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8775073Z return func(*args, **kwargs) 2025-08-26T20:32:01.8775336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8775408Z return func(*args, **kwargs) 2025-08-26T20:32:01.8775693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8775767Z return func(*args, **kwargs) 2025-08-26T20:32:01.8776053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8776217Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8776222Z 2025-08-26T20:32:01.8776337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8776557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8776629Z return mod(**inputs) 2025-08-26T20:32:01.8776883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8776981Z return func(*args, **kwargs) 2025-08-26T20:32:01.8777236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8777320Z return func(*args, **kwargs) 2025-08-26T20:32:01.8777554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8777641Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8777931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8778009Z outputs = self.layoutlm( 2025-08-26T20:32:01.8778289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8778363Z return func(*args, **kwargs) 2025-08-26T20:32:01.8778624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8778696Z return func(*args, **kwargs) 2025-08-26T20:32:01.8778930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8779017Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8779309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8779397Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8779655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8779747Z return func(*args, **kwargs) 2025-08-26T20:32:01.8780024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8780096Z return func(*args, **kwargs) 2025-08-26T20:32:01.8780366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8780438Z return func(*args, **kwargs) 2025-08-26T20:32:01.8780522Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8780768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8780848Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8781143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8781219Z layer_outputs = layer_module( 2025-08-26T20:32:01.8781465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8781553Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8781809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8781893Z return func(*args, **kwargs) 2025-08-26T20:32:01.8782149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8782247Z return func(*args, **kwargs) 2025-08-26T20:32:01.8782503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8782574Z return func(*args, **kwargs) 2025-08-26T20:32:01.8782870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8782962Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8783226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8783298Z return func(*args, **kwargs) 2025-08-26T20:32:01.8783579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8783661Z return func(*args, **kwargs) 2025-08-26T20:32:01.8783919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8784000Z return func(*args, **kwargs) 2025-08-26T20:32:01.8784289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8784374Z self_outputs = self.self( 2025-08-26T20:32:01.8784629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8784721Z return func(*args, **kwargs) 2025-08-26T20:32:01.8784989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8785061Z return func(*args, **kwargs) 2025-08-26T20:32:01.8785324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8785396Z return func(*args, **kwargs) 2025-08-26T20:32:01.8785680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8785842Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8785846Z 2025-08-26T20:32:01.8785960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8786181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8786273Z return mod(**inputs) 2025-08-26T20:32:01.8786542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8786614Z return func(*args, **kwargs) 2025-08-26T20:32:01.8786873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8786955Z return func(*args, **kwargs) 2025-08-26T20:32:01.8787193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8787279Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8787572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8787648Z outputs = self.layoutlm( 2025-08-26T20:32:01.8787915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8787990Z return func(*args, **kwargs) 2025-08-26T20:32:01.8788256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8788327Z return func(*args, **kwargs) 2025-08-26T20:32:01.8788565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8788653Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8788964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8789052Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8789310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8789382Z return func(*args, **kwargs) 2025-08-26T20:32:01.8789648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8789722Z return func(*args, **kwargs) 2025-08-26T20:32:01.8789985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8790057Z return func(*args, **kwargs) 2025-08-26T20:32:01.8790160Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8790399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8790479Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8790776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8790852Z layer_outputs = layer_module( 2025-08-26T20:32:01.8791098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8791184Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8791460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8791542Z return func(*args, **kwargs) 2025-08-26T20:32:01.8791798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8791879Z return func(*args, **kwargs) 2025-08-26T20:32:01.8792134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8792206Z return func(*args, **kwargs) 2025-08-26T20:32:01.8792499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8792587Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8792848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8792940Z return func(*args, **kwargs) 2025-08-26T20:32:01.8793202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8793274Z return func(*args, **kwargs) 2025-08-26T20:32:01.8793527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8793609Z return func(*args, **kwargs) 2025-08-26T20:32:01.8793896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8793981Z self_outputs = self.self( 2025-08-26T20:32:01.8794239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8794311Z return func(*args, **kwargs) 2025-08-26T20:32:01.8794574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8794649Z return func(*args, **kwargs) 2025-08-26T20:32:01.8794911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8794985Z return func(*args, **kwargs) 2025-08-26T20:32:01.8795274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8795461Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8795466Z 2025-08-26T20:32:01.8795556Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8795650Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8795764Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8795980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8796063Z return mod(**inputs) 2025-08-26T20:32:01.8796524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8796615Z return func(*args, **kwargs) 2025-08-26T20:32:01.8796924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8797012Z return func(*args, **kwargs) 2025-08-26T20:32:01.8797259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8797348Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8797658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8797738Z outputs = self.layoutlm( 2025-08-26T20:32:01.8798010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8798113Z return func(*args, **kwargs) 2025-08-26T20:32:01.8798379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8798461Z return func(*args, **kwargs) 2025-08-26T20:32:01.8798706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8798798Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8799101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8799185Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8799592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8799683Z return func(*args, **kwargs) 2025-08-26T20:32:01.8799955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8800079Z return func(*args, **kwargs) 2025-08-26T20:32:01.8800365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8800439Z return func(*args, **kwargs) 2025-08-26T20:32:01.8800526Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8800775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8800859Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8801166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8801244Z layer_outputs = layer_module( 2025-08-26T20:32:01.8801493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8801588Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8801873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8801955Z return func(*args, **kwargs) 2025-08-26T20:32:01.8802232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8802308Z return func(*args, **kwargs) 2025-08-26T20:32:01.8802609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8802686Z return func(*args, **kwargs) 2025-08-26T20:32:01.8802994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8803085Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8803359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8803442Z return func(*args, **kwargs) 2025-08-26T20:32:01.8803711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8803791Z return func(*args, **kwargs) 2025-08-26T20:32:01.8804074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8804156Z return func(*args, **kwargs) 2025-08-26T20:32:01.8804453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8804599Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8804905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8804997Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8805003Z 2025-08-26T20:32:01.8805145Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8805368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8805442Z return mod(**inputs) 2025-08-26T20:32:01.8805725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8805801Z return func(*args, **kwargs) 2025-08-26T20:32:01.8806077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8806152Z return func(*args, **kwargs) 2025-08-26T20:32:01.8806395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8806482Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8806778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8806893Z outputs = self.layoutlm( 2025-08-26T20:32:01.8807132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8807206Z return func(*args, **kwargs) 2025-08-26T20:32:01.8807504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8807573Z return func(*args, **kwargs) 2025-08-26T20:32:01.8807801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8807876Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8808154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8808229Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8808469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8808548Z return func(*args, **kwargs) 2025-08-26T20:32:01.8808788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8808865Z return func(*args, **kwargs) 2025-08-26T20:32:01.8809104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8809173Z return func(*args, **kwargs) 2025-08-26T20:32:01.8809286Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8809504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8809585Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8809853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8809936Z layer_outputs = layer_module( 2025-08-26T20:32:01.8810161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8810241Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8810504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8810574Z return func(*args, **kwargs) 2025-08-26T20:32:01.8810826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8810895Z return func(*args, **kwargs) 2025-08-26T20:32:01.8811136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8811212Z return func(*args, **kwargs) 2025-08-26T20:32:01.8811486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8811599Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8811861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8811942Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8812256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8812382Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8812658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8812742Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8812746Z 2025-08-26T20:32:01.8812858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8813078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8813149Z return mod(**inputs) 2025-08-26T20:32:01.8813398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8813465Z return func(*args, **kwargs) 2025-08-26T20:32:01.8813710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8813779Z return func(*args, **kwargs) 2025-08-26T20:32:01.8814001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8814085Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8814354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8814431Z outputs = self.layoutlm( 2025-08-26T20:32:01.8814670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8814739Z return func(*args, **kwargs) 2025-08-26T20:32:01.8814986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8815056Z return func(*args, **kwargs) 2025-08-26T20:32:01.8815282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8815357Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8815648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8815724Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8815963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8816040Z return func(*args, **kwargs) 2025-08-26T20:32:01.8816281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8816357Z return func(*args, **kwargs) 2025-08-26T20:32:01.8816614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8816684Z return func(*args, **kwargs) 2025-08-26T20:32:01.8816770Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8816989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8817069Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8817339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8817410Z layer_outputs = layer_module( 2025-08-26T20:32:01.8817641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8817738Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8817985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8818053Z return func(*args, **kwargs) 2025-08-26T20:32:01.8818298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8818367Z return func(*args, **kwargs) 2025-08-26T20:32:01.8818607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8818682Z return func(*args, **kwargs) 2025-08-26T20:32:01.8818950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8819043Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8819329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8819407Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8819719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8819841Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8820121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8820237Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8820451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8820529Z return self.act(input) 2025-08-26T20:32:01.8820533Z 2025-08-26T20:32:01.8820638Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8820850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8820920Z return mod(**inputs) 2025-08-26T20:32:01.8821168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8821239Z return func(*args, **kwargs) 2025-08-26T20:32:01.8821480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8821573Z return func(*args, **kwargs) 2025-08-26T20:32:01.8821796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8821879Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8822150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8822224Z outputs = self.layoutlm( 2025-08-26T20:32:01.8822476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8822544Z return func(*args, **kwargs) 2025-08-26T20:32:01.8822810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8822880Z return func(*args, **kwargs) 2025-08-26T20:32:01.8823098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8823183Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8823449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8823531Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8823769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8823847Z return func(*args, **kwargs) 2025-08-26T20:32:01.8824130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8824203Z return func(*args, **kwargs) 2025-08-26T20:32:01.8824467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8824540Z return func(*args, **kwargs) 2025-08-26T20:32:01.8824630Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8824866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8824943Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8825238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8825315Z layer_outputs = layer_module( 2025-08-26T20:32:01.8825576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8825659Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8825900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8825988Z return func(*args, **kwargs) 2025-08-26T20:32:01.8826222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8826300Z return func(*args, **kwargs) 2025-08-26T20:32:01.8826537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8826605Z return func(*args, **kwargs) 2025-08-26T20:32:01.8826878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8826962Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8827229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8827303Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8827603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8827737Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8828015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8828108Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8828112Z 2025-08-26T20:32:01.8828217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8828419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8828488Z return mod(**inputs) 2025-08-26T20:32:01.8828731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8828806Z return func(*args, **kwargs) 2025-08-26T20:32:01.8829059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8829136Z return func(*args, **kwargs) 2025-08-26T20:32:01.8829355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8829441Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8829725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8829796Z outputs = self.layoutlm( 2025-08-26T20:32:01.8830038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8830122Z return func(*args, **kwargs) 2025-08-26T20:32:01.8830363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8830430Z return func(*args, **kwargs) 2025-08-26T20:32:01.8830647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8830725Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8830992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8831072Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8831306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8831373Z return func(*args, **kwargs) 2025-08-26T20:32:01.8831611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8831703Z return func(*args, **kwargs) 2025-08-26T20:32:01.8831947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8832014Z return func(*args, **kwargs) 2025-08-26T20:32:01.8832091Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8832313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8832387Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8832665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8832738Z layer_outputs = layer_module( 2025-08-26T20:32:01.8832969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8833052Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8833296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8833376Z return func(*args, **kwargs) 2025-08-26T20:32:01.8833636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8833715Z return func(*args, **kwargs) 2025-08-26T20:32:01.8833970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8834059Z return func(*args, **kwargs) 2025-08-26T20:32:01.8834352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8834442Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8834705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8834781Z return func(*args, **kwargs) 2025-08-26T20:32:01.8835040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8835121Z return func(*args, **kwargs) 2025-08-26T20:32:01.8835844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8835934Z return func(*args, **kwargs) 2025-08-26T20:32:01.8836237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8836324Z self_outputs = self.self( 2025-08-26T20:32:01.8836589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8836663Z return func(*args, **kwargs) 2025-08-26T20:32:01.8836934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8837028Z return func(*args, **kwargs) 2025-08-26T20:32:01.8837311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8837383Z return func(*args, **kwargs) 2025-08-26T20:32:01.8837674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8837847Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8837851Z 2025-08-26T20:32:01.8837966Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8838192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8838266Z return mod(**inputs) 2025-08-26T20:32:01.8838538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8838639Z return func(*args, **kwargs) 2025-08-26T20:32:01.8838894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8838975Z return func(*args, **kwargs) 2025-08-26T20:32:01.8839275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8839376Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8839668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8839748Z outputs = self.layoutlm( 2025-08-26T20:32:01.8840014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8840087Z return func(*args, **kwargs) 2025-08-26T20:32:01.8840356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8840428Z return func(*args, **kwargs) 2025-08-26T20:32:01.8840647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8840735Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8841014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8841096Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8841352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8841422Z return func(*args, **kwargs) 2025-08-26T20:32:01.8841668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8841737Z return func(*args, **kwargs) 2025-08-26T20:32:01.8842002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8842072Z return func(*args, **kwargs) 2025-08-26T20:32:01.8842159Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8842393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8842470Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8842750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8842825Z layer_outputs = layer_module( 2025-08-26T20:32:01.8843069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8843153Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8843419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8843518Z return func(*args, **kwargs) 2025-08-26T20:32:01.8843785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8843864Z return func(*args, **kwargs) 2025-08-26T20:32:01.8844203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8844275Z return func(*args, **kwargs) 2025-08-26T20:32:01.8844576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8844664Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8844935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8845006Z return func(*args, **kwargs) 2025-08-26T20:32:01.8845293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8845368Z return func(*args, **kwargs) 2025-08-26T20:32:01.8845635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8845716Z return func(*args, **kwargs) 2025-08-26T20:32:01.8846005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8846087Z self_outputs = self.self( 2025-08-26T20:32:01.8846351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8846422Z return func(*args, **kwargs) 2025-08-26T20:32:01.8846685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8846755Z return func(*args, **kwargs) 2025-08-26T20:32:01.8847019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8847092Z return func(*args, **kwargs) 2025-08-26T20:32:01.8847377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8847537Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8847542Z 2025-08-26T20:32:01.8847656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8847900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8847973Z return mod(**inputs) 2025-08-26T20:32:01.8848241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8848313Z return func(*args, **kwargs) 2025-08-26T20:32:01.8848565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8848647Z return func(*args, **kwargs) 2025-08-26T20:32:01.8848883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8848987Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8849273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8849349Z outputs = self.layoutlm( 2025-08-26T20:32:01.8849659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8849729Z return func(*args, **kwargs) 2025-08-26T20:32:01.8849968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8871963Z return func(*args, **kwargs) 2025-08-26T20:32:01.8872455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8872640Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8872967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8873059Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8873349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8873437Z return func(*args, **kwargs) 2025-08-26T20:32:01.8873703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8873786Z return func(*args, **kwargs) 2025-08-26T20:32:01.8874044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8874149Z return func(*args, **kwargs) 2025-08-26T20:32:01.8874239Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8874484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8874574Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8874881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8874969Z layer_outputs = layer_module( 2025-08-26T20:32:01.8875219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8875311Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8875583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8875657Z return func(*args, **kwargs) 2025-08-26T20:32:01.8875928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8876003Z return func(*args, **kwargs) 2025-08-26T20:32:01.8876270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8876343Z return func(*args, **kwargs) 2025-08-26T20:32:01.8876643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8876743Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8877037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8877119Z return func(*args, **kwargs) 2025-08-26T20:32:01.8877375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8877449Z return func(*args, **kwargs) 2025-08-26T20:32:01.8877716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8877787Z return func(*args, **kwargs) 2025-08-26T20:32:01.8878115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8878196Z self_outputs = self.self( 2025-08-26T20:32:01.8878448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8878530Z return func(*args, **kwargs) 2025-08-26T20:32:01.8878783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8878864Z return func(*args, **kwargs) 2025-08-26T20:32:01.8879119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8879277Z return func(*args, **kwargs) 2025-08-26T20:32:01.8879599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8879767Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8879774Z 2025-08-26T20:32:01.8879877Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8879964Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8880093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8880323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8880400Z return mod(**inputs) 2025-08-26T20:32:01.8880676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8880752Z return func(*args, **kwargs) 2025-08-26T20:32:01.8881021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8881117Z return func(*args, **kwargs) 2025-08-26T20:32:01.8881361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8881456Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8881759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8881847Z outputs = self.layoutlm( 2025-08-26T20:32:01.8882103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8882178Z return func(*args, **kwargs) 2025-08-26T20:32:01.8882441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8882519Z return func(*args, **kwargs) 2025-08-26T20:32:01.8882762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8882845Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8883141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8883226Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8883487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8883588Z return func(*args, **kwargs) 2025-08-26T20:32:01.8883850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8883929Z return func(*args, **kwargs) 2025-08-26T20:32:01.8884193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8884268Z return func(*args, **kwargs) 2025-08-26T20:32:01.8884362Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8884594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8884681Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8884986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8885074Z layer_outputs = layer_module( 2025-08-26T20:32:01.8885316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8885403Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8885668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8885739Z return func(*args, **kwargs) 2025-08-26T20:32:01.8885998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8886090Z return func(*args, **kwargs) 2025-08-26T20:32:01.8886347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8886428Z return func(*args, **kwargs) 2025-08-26T20:32:01.8886725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8886824Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8887085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8887159Z return func(*args, **kwargs) 2025-08-26T20:32:01.8887433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8887505Z return func(*args, **kwargs) 2025-08-26T20:32:01.8887792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8887869Z return func(*args, **kwargs) 2025-08-26T20:32:01.8888164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8888316Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8888615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8888718Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8888723Z 2025-08-26T20:32:01.8888843Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8889075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8889150Z return mod(**inputs) 2025-08-26T20:32:01.8889418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8889502Z return func(*args, **kwargs) 2025-08-26T20:32:01.8889763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8889847Z return func(*args, **kwargs) 2025-08-26T20:32:01.8890088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8890171Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8890492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8890576Z outputs = self.layoutlm( 2025-08-26T20:32:01.8890845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8890921Z return func(*args, **kwargs) 2025-08-26T20:32:01.8891189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8891262Z return func(*args, **kwargs) 2025-08-26T20:32:01.8891499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8891607Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8891903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8891995Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8892257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8892331Z return func(*args, **kwargs) 2025-08-26T20:32:01.8892603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8892677Z return func(*args, **kwargs) 2025-08-26T20:32:01.8892963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8893038Z return func(*args, **kwargs) 2025-08-26T20:32:01.8893126Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8893374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8893455Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8893757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8893835Z layer_outputs = layer_module( 2025-08-26T20:32:01.8894081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8894177Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8894458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8894543Z return func(*args, **kwargs) 2025-08-26T20:32:01.8894806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8894890Z return func(*args, **kwargs) 2025-08-26T20:32:01.8895153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8895227Z return func(*args, **kwargs) 2025-08-26T20:32:01.8895535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8895632Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8895929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8896019Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8896537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8896692Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8896993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8897098Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8897104Z 2025-08-26T20:32:01.8897289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8897526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8897603Z return mod(**inputs) 2025-08-26T20:32:01.8897873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8897957Z return func(*args, **kwargs) 2025-08-26T20:32:01.8898222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8898303Z return func(*args, **kwargs) 2025-08-26T20:32:01.8898580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8898665Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8898972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8899052Z outputs = self.layoutlm( 2025-08-26T20:32:01.8899324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8899397Z return func(*args, **kwargs) 2025-08-26T20:32:01.8899656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8899738Z return func(*args, **kwargs) 2025-08-26T20:32:01.8900001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8900091Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8900388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8900477Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8900742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8900827Z return func(*args, **kwargs) 2025-08-26T20:32:01.8901090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8901162Z return func(*args, **kwargs) 2025-08-26T20:32:01.8901423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8901527Z return func(*args, **kwargs) 2025-08-26T20:32:01.8901611Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8901851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8901931Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8902225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8902302Z layer_outputs = layer_module( 2025-08-26T20:32:01.8902539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8902630Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8902884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8902964Z return func(*args, **kwargs) 2025-08-26T20:32:01.8903218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8903289Z return func(*args, **kwargs) 2025-08-26T20:32:01.8903551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8903622Z return func(*args, **kwargs) 2025-08-26T20:32:01.8903913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8904031Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8904316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8904397Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8904721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8904862Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8905146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8905289Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8905517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8905593Z return self.act(input) 2025-08-26T20:32:01.8905606Z 2025-08-26T20:32:01.8905722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8905939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8906017Z return mod(**inputs) 2025-08-26T20:32:01.8906276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8906376Z return func(*args, **kwargs) 2025-08-26T20:32:01.8906636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8906708Z return func(*args, **kwargs) 2025-08-26T20:32:01.8906952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8907032Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8907339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8907418Z outputs = self.layoutlm( 2025-08-26T20:32:01.8907677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8907756Z return func(*args, **kwargs) 2025-08-26T20:32:01.8908015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8908115Z return func(*args, **kwargs) 2025-08-26T20:32:01.8908351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8908431Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8908739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8908821Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8909091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8909165Z return func(*args, **kwargs) 2025-08-26T20:32:01.8909423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8909504Z return func(*args, **kwargs) 2025-08-26T20:32:01.8909766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8909851Z return func(*args, **kwargs) 2025-08-26T20:32:01.8909936Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8910180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8910260Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8910555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8910657Z layer_outputs = layer_module( 2025-08-26T20:32:01.8910900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8910992Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8911245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8911322Z return func(*args, **kwargs) 2025-08-26T20:32:01.8911585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8911656Z return func(*args, **kwargs) 2025-08-26T20:32:01.8911931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8912004Z return func(*args, **kwargs) 2025-08-26T20:32:01.8912291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8912389Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8912665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8912755Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8913089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8913263Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8913550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8913642Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8913646Z 2025-08-26T20:32:01.8913769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8913989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8914067Z return mod(**inputs) 2025-08-26T20:32:01.8914325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8914398Z return func(*args, **kwargs) 2025-08-26T20:32:01.8914663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8914754Z return func(*args, **kwargs) 2025-08-26T20:32:01.8914993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8915075Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8915372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8915450Z outputs = self.layoutlm( 2025-08-26T20:32:01.8915707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8915789Z return func(*args, **kwargs) 2025-08-26T20:32:01.8916040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8916119Z return func(*args, **kwargs) 2025-08-26T20:32:01.8916352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8916433Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8916731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8916814Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8917089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8917182Z return func(*args, **kwargs) 2025-08-26T20:32:01.8917446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8917528Z return func(*args, **kwargs) 2025-08-26T20:32:01.8917792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8917875Z return func(*args, **kwargs) 2025-08-26T20:32:01.8917962Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8918204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8918290Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8918614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8918699Z layer_outputs = layer_module( 2025-08-26T20:32:01.8918937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8919029Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8919349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8919425Z return func(*args, **kwargs) 2025-08-26T20:32:01.8919687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8919790Z return func(*args, **kwargs) 2025-08-26T20:32:01.8920057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8920131Z return func(*args, **kwargs) 2025-08-26T20:32:01.8920437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8920536Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8920791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8920870Z return func(*args, **kwargs) 2025-08-26T20:32:01.8921123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8921194Z return func(*args, **kwargs) 2025-08-26T20:32:01.8921488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8921563Z return func(*args, **kwargs) 2025-08-26T20:32:01.8921854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8921935Z self_outputs = self.self( 2025-08-26T20:32:01.8922198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8922271Z return func(*args, **kwargs) 2025-08-26T20:32:01.8922524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8922603Z return func(*args, **kwargs) 2025-08-26T20:32:01.8922857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8922939Z return func(*args, **kwargs) 2025-08-26T20:32:01.8923224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8923385Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8923390Z 2025-08-26T20:32:01.8923513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8923730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8923808Z return mod(**inputs) 2025-08-26T20:32:01.8924083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8924158Z return func(*args, **kwargs) 2025-08-26T20:32:01.8924420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8924492Z return func(*args, **kwargs) 2025-08-26T20:32:01.8924732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8924815Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8925109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8925200Z outputs = self.layoutlm( 2025-08-26T20:32:01.8925456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8925537Z return func(*args, **kwargs) 2025-08-26T20:32:01.8925791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8925872Z return func(*args, **kwargs) 2025-08-26T20:32:01.8926105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8926186Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8926511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8926589Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8926849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8926920Z return func(*args, **kwargs) 2025-08-26T20:32:01.8927176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8927252Z return func(*args, **kwargs) 2025-08-26T20:32:01.8927503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8927579Z return func(*args, **kwargs) 2025-08-26T20:32:01.8927660Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8927890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8927995Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8928278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8928361Z layer_outputs = layer_module( 2025-08-26T20:32:01.8928598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8928691Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8928946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8929019Z return func(*args, **kwargs) 2025-08-26T20:32:01.8929281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8929351Z return func(*args, **kwargs) 2025-08-26T20:32:01.8929610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8929684Z return func(*args, **kwargs) 2025-08-26T20:32:01.8929969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8930061Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8930301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8930393Z return func(*args, **kwargs) 2025-08-26T20:32:01.8930628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8930695Z return func(*args, **kwargs) 2025-08-26T20:32:01.8930934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8930999Z return func(*args, **kwargs) 2025-08-26T20:32:01.8931271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8931339Z self_outputs = self.self( 2025-08-26T20:32:01.8931597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8931664Z return func(*args, **kwargs) 2025-08-26T20:32:01.8931905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8931976Z return func(*args, **kwargs) 2025-08-26T20:32:01.8932216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8932289Z return func(*args, **kwargs) 2025-08-26T20:32:01.8932558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8932718Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8932722Z 2025-08-26T20:32:01.8932833Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8933036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8933106Z return mod(**inputs) 2025-08-26T20:32:01.8933342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8933409Z return func(*args, **kwargs) 2025-08-26T20:32:01.8933652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8933719Z return func(*args, **kwargs) 2025-08-26T20:32:01.8933940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8934029Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8934323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8934396Z outputs = self.layoutlm( 2025-08-26T20:32:01.8934650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8934726Z return func(*args, **kwargs) 2025-08-26T20:32:01.8934978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8935053Z return func(*args, **kwargs) 2025-08-26T20:32:01.8935281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8935359Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8935647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8935728Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8935988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8936066Z return func(*args, **kwargs) 2025-08-26T20:32:01.8936306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8936379Z return func(*args, **kwargs) 2025-08-26T20:32:01.8936632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8936703Z return func(*args, **kwargs) 2025-08-26T20:32:01.8936778Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8936997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8937074Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8937354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8937435Z layer_outputs = layer_module( 2025-08-26T20:32:01.8937673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8937777Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8938029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8938099Z return func(*args, **kwargs) 2025-08-26T20:32:01.8938355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8938425Z return func(*args, **kwargs) 2025-08-26T20:32:01.8938680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8938751Z return func(*args, **kwargs) 2025-08-26T20:32:01.8939052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8939146Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8939397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8939470Z return func(*args, **kwargs) 2025-08-26T20:32:01.8939711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8939778Z return func(*args, **kwargs) 2025-08-26T20:32:01.8940018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8940082Z return func(*args, **kwargs) 2025-08-26T20:32:01.8940355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8940441Z self_outputs = self.self( 2025-08-26T20:32:01.8940698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8940768Z return func(*args, **kwargs) 2025-08-26T20:32:01.8941020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8941095Z return func(*args, **kwargs) 2025-08-26T20:32:01.8941349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8941421Z return func(*args, **kwargs) 2025-08-26T20:32:01.8941702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.8941857Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8941863Z 2025-08-26T20:32:01.8942110Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8942193Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.8942307Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8942522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8942600Z return mod(**inputs) 2025-08-26T20:32:01.8942855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8942980Z return func(*args, **kwargs) 2025-08-26T20:32:01.8943251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8943321Z return func(*args, **kwargs) 2025-08-26T20:32:01.8943555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8943631Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8943920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8944001Z outputs = self.layoutlm( 2025-08-26T20:32:01.8944269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8944350Z return func(*args, **kwargs) 2025-08-26T20:32:01.8944610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8944688Z return func(*args, **kwargs) 2025-08-26T20:32:01.8944919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8944999Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8945293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8945403Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8945672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8945741Z return func(*args, **kwargs) 2025-08-26T20:32:01.8945982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8946058Z return func(*args, **kwargs) 2025-08-26T20:32:01.8946299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8946378Z return func(*args, **kwargs) 2025-08-26T20:32:01.8946460Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8946686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8946772Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8947080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8947165Z layer_outputs = layer_module( 2025-08-26T20:32:01.8947402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8947489Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8947752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8947826Z return func(*args, **kwargs) 2025-08-26T20:32:01.8948087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8948159Z return func(*args, **kwargs) 2025-08-26T20:32:01.8948420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8948490Z return func(*args, **kwargs) 2025-08-26T20:32:01.8948765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8948857Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8949099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8949174Z return func(*args, **kwargs) 2025-08-26T20:32:01.8949431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8949499Z return func(*args, **kwargs) 2025-08-26T20:32:01.8949750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8949818Z return func(*args, **kwargs) 2025-08-26T20:32:01.8950097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.8950245Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.8950532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.8950643Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8950647Z 2025-08-26T20:32:01.8950758Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8950979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8951049Z return mod(**inputs) 2025-08-26T20:32:01.8951309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8951380Z return func(*args, **kwargs) 2025-08-26T20:32:01.8951628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8951709Z return func(*args, **kwargs) 2025-08-26T20:32:01.8951953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8952038Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8952323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8952397Z outputs = self.layoutlm( 2025-08-26T20:32:01.8952659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8952727Z return func(*args, **kwargs) 2025-08-26T20:32:01.8952987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8953058Z return func(*args, **kwargs) 2025-08-26T20:32:01.8953289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8953396Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8953685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8953771Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8954031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8954112Z return func(*args, **kwargs) 2025-08-26T20:32:01.8954366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8954438Z return func(*args, **kwargs) 2025-08-26T20:32:01.8954698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8954769Z return func(*args, **kwargs) 2025-08-26T20:32:01.8954859Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8955090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8955168Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8955464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8955541Z layer_outputs = layer_module( 2025-08-26T20:32:01.8955789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8955889Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8956144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8956221Z return func(*args, **kwargs) 2025-08-26T20:32:01.8956475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8956557Z return func(*args, **kwargs) 2025-08-26T20:32:01.8956808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8956879Z return func(*args, **kwargs) 2025-08-26T20:32:01.8957189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8957284Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8957575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8957658Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8957990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8958124Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8958413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.8958529Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8958533Z 2025-08-26T20:32:01.8958644Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8958868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8958939Z return mod(**inputs) 2025-08-26T20:32:01.8959278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8959374Z return func(*args, **kwargs) 2025-08-26T20:32:01.8959635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8959717Z return func(*args, **kwargs) 2025-08-26T20:32:01.8959949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8960064Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8960358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8960437Z outputs = self.layoutlm( 2025-08-26T20:32:01.8960713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8960788Z return func(*args, **kwargs) 2025-08-26T20:32:01.8961065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8961137Z return func(*args, **kwargs) 2025-08-26T20:32:01.8961368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8961457Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8961746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8961833Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8962074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8962146Z return func(*args, **kwargs) 2025-08-26T20:32:01.8962419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8962494Z return func(*args, **kwargs) 2025-08-26T20:32:01.8962778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8962854Z return func(*args, **kwargs) 2025-08-26T20:32:01.8962944Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8963179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8963259Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8963559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8963637Z layer_outputs = layer_module( 2025-08-26T20:32:01.8963904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8963990Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8964252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8964332Z return func(*args, **kwargs) 2025-08-26T20:32:01.8964595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8964675Z return func(*args, **kwargs) 2025-08-26T20:32:01.8964940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8965035Z return func(*args, **kwargs) 2025-08-26T20:32:01.8965341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8965434Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8965731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8965817Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8966157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.8966290Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.8966585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.8966738Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.8966978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.8967063Z return self.act(input) 2025-08-26T20:32:01.8967067Z 2025-08-26T20:32:01.8967186Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8967406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8967487Z return mod(**inputs) 2025-08-26T20:32:01.8967761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8967842Z return func(*args, **kwargs) 2025-08-26T20:32:01.8968116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8968191Z return func(*args, **kwargs) 2025-08-26T20:32:01.8968442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8968527Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8968831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8968910Z outputs = self.layoutlm( 2025-08-26T20:32:01.8969196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8969269Z return func(*args, **kwargs) 2025-08-26T20:32:01.8969559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8969645Z return func(*args, **kwargs) 2025-08-26T20:32:01.8969883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8969973Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8970268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8970351Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8970667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8970744Z return func(*args, **kwargs) 2025-08-26T20:32:01.8971013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8971083Z return func(*args, **kwargs) 2025-08-26T20:32:01.8971323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8971400Z return func(*args, **kwargs) 2025-08-26T20:32:01.8971477Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8971718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8971818Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8972105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8972191Z layer_outputs = layer_module( 2025-08-26T20:32:01.8972431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8972525Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8972787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8972867Z return func(*args, **kwargs) 2025-08-26T20:32:01.8973126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8973198Z return func(*args, **kwargs) 2025-08-26T20:32:01.8973478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8973550Z return func(*args, **kwargs) 2025-08-26T20:32:01.8973846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.8973939Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.8974219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.8974312Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.8974639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.8974798Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.8975066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.8975159Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.8975163Z 2025-08-26T20:32:01.8975271Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8975476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8975553Z return mod(**inputs) 2025-08-26T20:32:01.8975792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8975883Z return func(*args, **kwargs) 2025-08-26T20:32:01.8976122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8976189Z return func(*args, **kwargs) 2025-08-26T20:32:01.8976416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8976494Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8976772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8976845Z outputs = self.layoutlm( 2025-08-26T20:32:01.8977098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8977176Z return func(*args, **kwargs) 2025-08-26T20:32:01.8977415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8977493Z return func(*args, **kwargs) 2025-08-26T20:32:01.8977713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8977795Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8978067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8978160Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8978410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8978478Z return func(*args, **kwargs) 2025-08-26T20:32:01.8978730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8978799Z return func(*args, **kwargs) 2025-08-26T20:32:01.8979043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8979119Z return func(*args, **kwargs) 2025-08-26T20:32:01.8979198Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8979425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8979501Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8979788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8979870Z layer_outputs = layer_module( 2025-08-26T20:32:01.8980096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8980186Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8980424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8980494Z return func(*args, **kwargs) 2025-08-26T20:32:01.8980741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8980811Z return func(*args, **kwargs) 2025-08-26T20:32:01.8981056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8981125Z return func(*args, **kwargs) 2025-08-26T20:32:01.8981404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8981490Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8981730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8981806Z return func(*args, **kwargs) 2025-08-26T20:32:01.8982058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8982134Z return func(*args, **kwargs) 2025-08-26T20:32:01.8982370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8982437Z return func(*args, **kwargs) 2025-08-26T20:32:01.8982728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8982809Z self_outputs = self.self( 2025-08-26T20:32:01.8983072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8983143Z return func(*args, **kwargs) 2025-08-26T20:32:01.8983412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8983493Z return func(*args, **kwargs) 2025-08-26T20:32:01.8983746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8983836Z return func(*args, **kwargs) 2025-08-26T20:32:01.8984103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:01.8984261Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8984266Z 2025-08-26T20:32:01.8984374Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8984594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8984669Z return mod(**inputs) 2025-08-26T20:32:01.8984914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8984989Z return func(*args, **kwargs) 2025-08-26T20:32:01.8985225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8985294Z return func(*args, **kwargs) 2025-08-26T20:32:01.8985519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8985596Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8985871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8985962Z outputs = self.layoutlm( 2025-08-26T20:32:01.8986206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8986283Z return func(*args, **kwargs) 2025-08-26T20:32:01.8986530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8986606Z return func(*args, **kwargs) 2025-08-26T20:32:01.8986832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8986913Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8987188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8987262Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8987515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8987587Z return func(*args, **kwargs) 2025-08-26T20:32:01.8987837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8987904Z return func(*args, **kwargs) 2025-08-26T20:32:01.8988160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8988239Z return func(*args, **kwargs) 2025-08-26T20:32:01.8988339Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8988576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8988663Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8988933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8989014Z layer_outputs = layer_module( 2025-08-26T20:32:01.8989241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8989333Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8989599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8989678Z return func(*args, **kwargs) 2025-08-26T20:32:01.8989941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8990011Z return func(*args, **kwargs) 2025-08-26T20:32:01.8990274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8990345Z return func(*args, **kwargs) 2025-08-26T20:32:01.8990639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.8990745Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.8990995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8991073Z return func(*args, **kwargs) 2025-08-26T20:32:01.8991322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8991399Z return func(*args, **kwargs) 2025-08-26T20:32:01.8991652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8991722Z return func(*args, **kwargs) 2025-08-26T20:32:01.8992014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.8992088Z self_outputs = self.self( 2025-08-26T20:32:01.8992344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8992433Z return func(*args, **kwargs) 2025-08-26T20:32:01.8992689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8992769Z return func(*args, **kwargs) 2025-08-26T20:32:01.8993025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8993105Z return func(*args, **kwargs) 2025-08-26T20:32:01.8993397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:01.8993554Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.8993558Z 2025-08-26T20:32:01.8993669Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.8993887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.8993970Z return mod(**inputs) 2025-08-26T20:32:01.8994230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8994309Z return func(*args, **kwargs) 2025-08-26T20:32:01.8994567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8994638Z return func(*args, **kwargs) 2025-08-26T20:32:01.8994909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8994991Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8995283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.8995358Z outputs = self.layoutlm( 2025-08-26T20:32:01.8995611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8995693Z return func(*args, **kwargs) 2025-08-26T20:32:01.8995946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8996043Z return func(*args, **kwargs) 2025-08-26T20:32:01.8996500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8996597Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8996888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.8996967Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.8997232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8997305Z return func(*args, **kwargs) 2025-08-26T20:32:01.8997573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8997669Z return func(*args, **kwargs) 2025-08-26T20:32:01.8997926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8998009Z return func(*args, **kwargs) 2025-08-26T20:32:01.8998094Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.8998347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.8998428Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.8998730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.8998818Z layer_outputs = layer_module( 2025-08-26T20:32:01.8999067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.8999240Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.8999514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8999590Z return func(*args, **kwargs) 2025-08-26T20:32:01.8999859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.8999943Z return func(*args, **kwargs) 2025-08-26T20:32:01.9000206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9000279Z return func(*args, **kwargs) 2025-08-26T20:32:01.9000576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.9000665Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.9000922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9001005Z return func(*args, **kwargs) 2025-08-26T20:32:01.9001262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9001343Z return func(*args, **kwargs) 2025-08-26T20:32:01.9001598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9001669Z return func(*args, **kwargs) 2025-08-26T20:32:01.9001991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:01.9002070Z self_outputs = self.self( 2025-08-26T20:32:01.9002330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9002400Z return func(*args, **kwargs) 2025-08-26T20:32:01.9002655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9002735Z return func(*args, **kwargs) 2025-08-26T20:32:01.9002985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9003079Z return func(*args, **kwargs) 2025-08-26T20:32:01.9003364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:01.9003529Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:01.9003534Z 2025-08-26T20:32:01.9003621Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.9003706Z cudagraph partition due to non gpu ops 2025-08-26T20:32:01.9003825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.9004039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.9004135Z return mod(**inputs) 2025-08-26T20:32:01.9004390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9004462Z return func(*args, **kwargs) 2025-08-26T20:32:01.9004722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9004795Z return func(*args, **kwargs) 2025-08-26T20:32:01.9005033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9005112Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9005396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.9005475Z outputs = self.layoutlm( 2025-08-26T20:32:01.9005715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9005809Z return func(*args, **kwargs) 2025-08-26T20:32:01.9006050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9006122Z return func(*args, **kwargs) 2025-08-26T20:32:01.9006352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9006427Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9006707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.9006781Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.9007029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9007098Z return func(*args, **kwargs) 2025-08-26T20:32:01.9007339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9007419Z return func(*args, **kwargs) 2025-08-26T20:32:01.9007661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9007736Z return func(*args, **kwargs) 2025-08-26T20:32:01.9007814Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.9008057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9008141Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9008412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.9008492Z layer_outputs = layer_module( 2025-08-26T20:32:01.9008715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.9008800Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.9009050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9009119Z return func(*args, **kwargs) 2025-08-26T20:32:01.9009375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9009446Z return func(*args, **kwargs) 2025-08-26T20:32:01.9009694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9009762Z return func(*args, **kwargs) 2025-08-26T20:32:01.9010035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:01.9010127Z self_attention_outputs = self.attention( 2025-08-26T20:32:01.9010368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9010459Z return func(*args, **kwargs) 2025-08-26T20:32:01.9010703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9010773Z return func(*args, **kwargs) 2025-08-26T20:32:01.9011026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9011094Z return func(*args, **kwargs) 2025-08-26T20:32:01.9011378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:01.9011513Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:01.9011788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:01.9011903Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.9011908Z 2025-08-26T20:32:01.9012014Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.9012223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.9012291Z return mod(**inputs) 2025-08-26T20:32:01.9012547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9012616Z return func(*args, **kwargs) 2025-08-26T20:32:01.9012861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9012938Z return func(*args, **kwargs) 2025-08-26T20:32:01.9013158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9013240Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9013514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.9013589Z outputs = self.layoutlm( 2025-08-26T20:32:01.9013857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9013930Z return func(*args, **kwargs) 2025-08-26T20:32:01.9014196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9014269Z return func(*args, **kwargs) 2025-08-26T20:32:01.9014520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9014610Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9014895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.9014982Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.9015241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9015321Z return func(*args, **kwargs) 2025-08-26T20:32:01.9015596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9015669Z return func(*args, **kwargs) 2025-08-26T20:32:01.9015932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9016005Z return func(*args, **kwargs) 2025-08-26T20:32:01.9016104Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.9016325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9016397Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9016679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.9016771Z layer_outputs = layer_module( 2025-08-26T20:32:01.9016999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.9017079Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.9017321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9017396Z return func(*args, **kwargs) 2025-08-26T20:32:01.9017639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9017714Z return func(*args, **kwargs) 2025-08-26T20:32:01.9017953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9018020Z return func(*args, **kwargs) 2025-08-26T20:32:01.9018322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.9018410Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.9018680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.9018761Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.9019075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.9019204Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.9019473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:01.9019567Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.9019570Z 2025-08-26T20:32:01.9019676Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.9019885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.9019955Z return mod(**inputs) 2025-08-26T20:32:01.9020197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9020276Z return func(*args, **kwargs) 2025-08-26T20:32:01.9020513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9020588Z return func(*args, **kwargs) 2025-08-26T20:32:01.9020821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9020905Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9021186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.9021264Z outputs = self.layoutlm( 2025-08-26T20:32:01.9021530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9021603Z return func(*args, **kwargs) 2025-08-26T20:32:01.9021882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9021954Z return func(*args, **kwargs) 2025-08-26T20:32:01.9022189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9022281Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9022577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.9022665Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.9022936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9023007Z return func(*args, **kwargs) 2025-08-26T20:32:01.9023277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9023347Z return func(*args, **kwargs) 2025-08-26T20:32:01.9023610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9023684Z return func(*args, **kwargs) 2025-08-26T20:32:01.9023767Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.9024016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9024097Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9024406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.9024485Z layer_outputs = layer_module( 2025-08-26T20:32:01.9024756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.9024846Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.9025110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9025195Z return func(*args, **kwargs) 2025-08-26T20:32:01.9025458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9025540Z return func(*args, **kwargs) 2025-08-26T20:32:01.9025798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9025872Z return func(*args, **kwargs) 2025-08-26T20:32:01.9026173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.9026268Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.9026565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.9026651Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.9026985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:01.9027130Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:01.9027437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:01.9027575Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:01.9027812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:01.9027898Z return self.act(input) 2025-08-26T20:32:01.9027903Z 2025-08-26T20:32:01.9028020Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.9028246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.9028327Z return mod(**inputs) 2025-08-26T20:32:01.9028607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9028690Z return func(*args, **kwargs) 2025-08-26T20:32:01.9028957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9029033Z return func(*args, **kwargs) 2025-08-26T20:32:01.9029312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9029395Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9029700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-26T20:32:01.9029784Z outputs = self.layoutlm( 2025-08-26T20:32:01.9030039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9030106Z return func(*args, **kwargs) 2025-08-26T20:32:01.9030341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9030415Z return func(*args, **kwargs) 2025-08-26T20:32:01.9030627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9030709Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9030970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:01.9031042Z encoder_outputs = self.encoder( 2025-08-26T20:32:01.9031282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9031366Z return func(*args, **kwargs) 2025-08-26T20:32:01.9031610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9031677Z return func(*args, **kwargs) 2025-08-26T20:32:01.9031914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9031987Z return func(*args, **kwargs) 2025-08-26T20:32:01.9032064Z [Previous line repeated 1 more time] 2025-08-26T20:32:01.9032287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9032359Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9032625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:01.9032707Z layer_outputs = layer_module( 2025-08-26T20:32:01.9032932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:01.9033019Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:01.9033266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9033340Z return func(*args, **kwargs) 2025-08-26T20:32:01.9033584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9033668Z return func(*args, **kwargs) 2025-08-26T20:32:01.9033915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9033986Z return func(*args, **kwargs) 2025-08-26T20:32:01.9034274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:01.9034367Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:01.9034646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:01.9034745Z return forward_fn(*input_tensors) 2025-08-26T20:32:01.9035060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:01.9035205Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:01.9035479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:01.9035572Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.9035576Z 2025-08-26T20:32:01.9035682Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.9035884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.9035977Z return mod(**inputs) 2025-08-26T20:32:01.9036219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9036296Z return func(*args, **kwargs) 2025-08-26T20:32:01.9036540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9036609Z return func(*args, **kwargs) 2025-08-26T20:32:01.9036836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9036913Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9037191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-08-26T20:32:01.9037290Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:32:01.9037561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-08-26T20:32:01.9037705Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:32:01.9037978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 472, in forward 2025-08-26T20:32:01.9038086Z hidden_states = self.transform(hidden_states) 2025-08-26T20:32:01.9038359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 447, in forward 2025-08-26T20:32:01.9038456Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:01.9038460Z 2025-08-26T20:32:01.9038571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.9038776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.9038855Z return mod(**inputs) 2025-08-26T20:32:01.9039099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9039182Z return func(*args, **kwargs) 2025-08-26T20:32:01.9039509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9039588Z return func(*args, **kwargs) 2025-08-26T20:32:01.9039830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9039911Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9040226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-08-26T20:32:01.9040328Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:32:01.9040634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-08-26T20:32:01.9040747Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:32:01.9041012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 473, in forward 2025-08-26T20:32:01.9041111Z hidden_states = self.decoder(hidden_states) 2025-08-26T20:32:01.9041115Z 2025-08-26T20:32:01.9041234Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:01.9041440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:01.9041509Z return mod(**inputs) 2025-08-26T20:32:01.9041746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9041824Z return func(*args, **kwargs) 2025-08-26T20:32:01.9042058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:01.9042135Z return func(*args, **kwargs) 2025-08-26T20:32:01.9042353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:01.9042451Z output = func(self, *args, **kwargs) 2025-08-26T20:32:01.9042723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 776, in forward 2025-08-26T20:32:01.9042796Z masked_lm_loss = loss_fct( 2025-08-26T20:32:01.9042799Z 2025-08-26T20:32:10.6757406Z Compilation time (from dynamo_timed): 16.326097639 2025-08-26T20:32:10.6796704Z pass 2025-08-26T20:32:10.6798075Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:10.6799088Z TIMING: _recursive_pre_grad_passes:0.0086 _recursive_joint_graph_passes:0.48603 _recursive_post_grad_passes:0.08013 async_compile.wait:0.64198 code_gen:7.90757 inductor_compile:9.24506 backend_compile:13.0292 gc:0.00039 entire_frame_compile:16.3261 total_wall_time:16.3261 2025-08-26T20:32:10.6800396Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:15436 | FakeTensor.__torch_dispatch__:4457 | ProxyTorchDispatchMode.__torch_dispatch__:5848 2025-08-26T20:32:10.6801246Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-08-26T20:32:16.2712674Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:32:16.2713697Z from pkg_resources import resource_filename 2025-08-26T20:32:16.9017968Z 2025-08-26T20:32:18.2031960Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:32:18.2032497Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:32:18.2047132Z cpu eval LayoutLMForSequenceClassification 2025-08-26T20:32:18.7248342Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:18.9251564Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:19.1286643Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:27.9732550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9733101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9733509Z return mod(**inputs) 2025-08-26T20:32:27.9734232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9734650Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9735099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9735565Z outputs = self.layoutlm( 2025-08-26T20:32:27.9735976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9736440Z return func(*args, **kwargs) 2025-08-26T20:32:27.9736845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9737254Z return func(*args, **kwargs) 2025-08-26T20:32:27.9737767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9738169Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9738615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9739043Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9739458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9739860Z return func(*args, **kwargs) 2025-08-26T20:32:27.9740262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9740723Z return func(*args, **kwargs) 2025-08-26T20:32:27.9741111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9741523Z return func(*args, **kwargs) 2025-08-26T20:32:27.9741743Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9742136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9742532Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9742963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9743401Z layer_outputs = layer_module( 2025-08-26T20:32:27.9743783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9744234Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9744637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9745048Z return func(*args, **kwargs) 2025-08-26T20:32:27.9745432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9745838Z return func(*args, **kwargs) 2025-08-26T20:32:27.9746231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9746627Z return func(*args, **kwargs) 2025-08-26T20:32:27.9747043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9747503Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9747910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9748310Z return func(*args, **kwargs) 2025-08-26T20:32:27.9748699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9749094Z return func(*args, **kwargs) 2025-08-26T20:32:27.9749486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9749894Z return func(*args, **kwargs) 2025-08-26T20:32:27.9750329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:27.9750760Z self_outputs = self.self( 2025-08-26T20:32:27.9751161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9751597Z return func(*args, **kwargs) 2025-08-26T20:32:27.9751988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9752385Z return func(*args, **kwargs) 2025-08-26T20:32:27.9752778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9753211Z return func(*args, **kwargs) 2025-08-26T20:32:27.9753634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:27.9754160Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:27.9754392Z 2025-08-26T20:32:27.9754690Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9755088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9755462Z return mod(**inputs) 2025-08-26T20:32:27.9755822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9756230Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9756675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9757120Z outputs = self.layoutlm( 2025-08-26T20:32:27.9757510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9757927Z return func(*args, **kwargs) 2025-08-26T20:32:27.9758335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9758759Z return func(*args, **kwargs) 2025-08-26T20:32:27.9759143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9759744Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9760229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9760681Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9761086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9761495Z return func(*args, **kwargs) 2025-08-26T20:32:27.9761887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9762344Z return func(*args, **kwargs) 2025-08-26T20:32:27.9762738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9763148Z return func(*args, **kwargs) 2025-08-26T20:32:27.9763362Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9763747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9764139Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9764595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9765046Z layer_outputs = layer_module( 2025-08-26T20:32:27.9765432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9765829Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9766264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9766680Z return func(*args, **kwargs) 2025-08-26T20:32:27.9767072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9767485Z return func(*args, **kwargs) 2025-08-26T20:32:27.9767876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9768339Z return func(*args, **kwargs) 2025-08-26T20:32:27.9768799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9769293Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9769713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9770129Z return func(*args, **kwargs) 2025-08-26T20:32:27.9770526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9770931Z return func(*args, **kwargs) 2025-08-26T20:32:27.9771341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9771733Z return func(*args, **kwargs) 2025-08-26T20:32:27.9772163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:27.9772629Z self_outputs = self.self( 2025-08-26T20:32:27.9772991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9773363Z return func(*args, **kwargs) 2025-08-26T20:32:27.9773724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9774103Z return func(*args, **kwargs) 2025-08-26T20:32:27.9774454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9774822Z return func(*args, **kwargs) 2025-08-26T20:32:27.9775198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:27.9775724Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:27.9775947Z 2025-08-26T20:32:27.9776059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9776442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9776797Z return mod(**inputs) 2025-08-26T20:32:27.9777149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9777513Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9777919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9778325Z outputs = self.layoutlm( 2025-08-26T20:32:27.9778678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9779063Z return func(*args, **kwargs) 2025-08-26T20:32:27.9779443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9779838Z return func(*args, **kwargs) 2025-08-26T20:32:27.9780179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9780528Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9780930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9781359Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9781738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9782109Z return func(*args, **kwargs) 2025-08-26T20:32:27.9782463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9782836Z return func(*args, **kwargs) 2025-08-26T20:32:27.9783198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9783582Z return func(*args, **kwargs) 2025-08-26T20:32:27.9783783Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9784180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9784562Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9784994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9785426Z layer_outputs = layer_module( 2025-08-26T20:32:27.9785798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9786193Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9786611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9787056Z return func(*args, **kwargs) 2025-08-26T20:32:27.9787433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9787811Z return func(*args, **kwargs) 2025-08-26T20:32:27.9788198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9788573Z return func(*args, **kwargs) 2025-08-26T20:32:27.9788987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9789422Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9789827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9790241Z return func(*args, **kwargs) 2025-08-26T20:32:27.9790636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9791001Z return func(*args, **kwargs) 2025-08-26T20:32:27.9791365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9791739Z return func(*args, **kwargs) 2025-08-26T20:32:27.9792148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:27.9792580Z self_outputs = self.self( 2025-08-26T20:32:27.9792963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9793357Z return func(*args, **kwargs) 2025-08-26T20:32:27.9793740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9794142Z return func(*args, **kwargs) 2025-08-26T20:32:27.9794532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9794933Z return func(*args, **kwargs) 2025-08-26T20:32:27.9795361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:27.9795885Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:27.9796110Z 2025-08-26T20:32:27.9796466Z cudagraph partition due to non gpu ops 2025-08-26T20:32:27.9796713Z cudagraph partition due to non gpu ops 2025-08-26T20:32:27.9796981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9797387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9797758Z return mod(**inputs) 2025-08-26T20:32:27.9798128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9798516Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9798962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9799508Z outputs = self.layoutlm( 2025-08-26T20:32:27.9799921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9800327Z return func(*args, **kwargs) 2025-08-26T20:32:27.9800737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9801117Z return func(*args, **kwargs) 2025-08-26T20:32:27.9801461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9801824Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9802228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9802687Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9803093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9803492Z return func(*args, **kwargs) 2025-08-26T20:32:27.9803877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9804275Z return func(*args, **kwargs) 2025-08-26T20:32:27.9804656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9805059Z return func(*args, **kwargs) 2025-08-26T20:32:27.9805268Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9805639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9806060Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9806490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9806919Z layer_outputs = layer_module( 2025-08-26T20:32:27.9807297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9807675Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9808080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9808475Z return func(*args, **kwargs) 2025-08-26T20:32:27.9808858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9809252Z return func(*args, **kwargs) 2025-08-26T20:32:27.9809649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9810039Z return func(*args, **kwargs) 2025-08-26T20:32:27.9810455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9810921Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9811330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9811747Z return func(*args, **kwargs) 2025-08-26T20:32:27.9812135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9812540Z return func(*args, **kwargs) 2025-08-26T20:32:27.9812955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9813366Z return func(*args, **kwargs) 2025-08-26T20:32:27.9813783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:27.9814301Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:27.9814809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:27.9815245Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:27.9815404Z 2025-08-26T20:32:27.9815522Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9815914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9816272Z return mod(**inputs) 2025-08-26T20:32:27.9816626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9817003Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9817439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9817891Z outputs = self.layoutlm( 2025-08-26T20:32:27.9818281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9818682Z return func(*args, **kwargs) 2025-08-26T20:32:27.9819087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9819488Z return func(*args, **kwargs) 2025-08-26T20:32:27.9819850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9820231Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9820654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9821116Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9821531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9821932Z return func(*args, **kwargs) 2025-08-26T20:32:27.9822323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9822714Z return func(*args, **kwargs) 2025-08-26T20:32:27.9823100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9823514Z return func(*args, **kwargs) 2025-08-26T20:32:27.9823822Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9824201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9824584Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9825014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9825450Z layer_outputs = layer_module( 2025-08-26T20:32:27.9825829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9826214Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9826619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9827012Z return func(*args, **kwargs) 2025-08-26T20:32:27.9827418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9827823Z return func(*args, **kwargs) 2025-08-26T20:32:27.9828206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9828606Z return func(*args, **kwargs) 2025-08-26T20:32:27.9829024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:27.9829488Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:27.9829943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:27.9830368Z return forward_fn(*input_tensors) 2025-08-26T20:32:27.9830836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:27.9831353Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:27.9831835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:27.9832273Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:27.9832430Z 2025-08-26T20:32:27.9832546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9832968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9833329Z return mod(**inputs) 2025-08-26T20:32:27.9833686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9834083Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9834534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9834984Z outputs = self.layoutlm( 2025-08-26T20:32:27.9835381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9835778Z return func(*args, **kwargs) 2025-08-26T20:32:27.9836173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9836604Z return func(*args, **kwargs) 2025-08-26T20:32:27.9836978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9837370Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9837812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9838262Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9838681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9839092Z return func(*args, **kwargs) 2025-08-26T20:32:27.9839560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9839969Z return func(*args, **kwargs) 2025-08-26T20:32:27.9840362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9840773Z return func(*args, **kwargs) 2025-08-26T20:32:27.9840990Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9841373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9841770Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9842215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9842656Z layer_outputs = layer_module( 2025-08-26T20:32:27.9843056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9843459Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9843872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9844277Z return func(*args, **kwargs) 2025-08-26T20:32:27.9844670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9845068Z return func(*args, **kwargs) 2025-08-26T20:32:27.9845458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9845895Z return func(*args, **kwargs) 2025-08-26T20:32:27.9846312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:27.9846753Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:27.9847188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:27.9847611Z return forward_fn(*input_tensors) 2025-08-26T20:32:27.9848075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:27.9848603Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:27.9849053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:27.9849516Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:27.9849939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:27.9850318Z return self.act(input) 2025-08-26T20:32:27.9850444Z 2025-08-26T20:32:27.9850575Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9850967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9851305Z return mod(**inputs) 2025-08-26T20:32:27.9851647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9852028Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9852431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9852863Z outputs = self.layoutlm( 2025-08-26T20:32:27.9853255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9853652Z return func(*args, **kwargs) 2025-08-26T20:32:27.9854043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9854443Z return func(*args, **kwargs) 2025-08-26T20:32:27.9854804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9855223Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9855652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9856093Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9856478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9856871Z return func(*args, **kwargs) 2025-08-26T20:32:27.9857263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9857671Z return func(*args, **kwargs) 2025-08-26T20:32:27.9858079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9858481Z return func(*args, **kwargs) 2025-08-26T20:32:27.9858694Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9859074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9859460Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9859884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9860314Z layer_outputs = layer_module( 2025-08-26T20:32:27.9860687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9861101Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9861508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9861912Z return func(*args, **kwargs) 2025-08-26T20:32:27.9862303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9862704Z return func(*args, **kwargs) 2025-08-26T20:32:27.9863093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9863483Z return func(*args, **kwargs) 2025-08-26T20:32:27.9863918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:27.9864363Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:27.9864802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:27.9865228Z return forward_fn(*input_tensors) 2025-08-26T20:32:27.9865687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:27.9866218Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:27.9866712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:27.9867153Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:27.9867326Z 2025-08-26T20:32:27.9867444Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9867835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9868189Z return mod(**inputs) 2025-08-26T20:32:27.9868549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9868930Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9869357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9869787Z outputs = self.layoutlm( 2025-08-26T20:32:27.9870173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9870581Z return func(*args, **kwargs) 2025-08-26T20:32:27.9870959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9871359Z return func(*args, **kwargs) 2025-08-26T20:32:27.9871721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9872102Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9872533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9872962Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9873378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9873779Z return func(*args, **kwargs) 2025-08-26T20:32:27.9874166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9874559Z return func(*args, **kwargs) 2025-08-26T20:32:27.9874939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9875333Z return func(*args, **kwargs) 2025-08-26T20:32:27.9875544Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9875942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9876317Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9876747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9877178Z layer_outputs = layer_module( 2025-08-26T20:32:27.9877561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9877964Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9878371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9878801Z return func(*args, **kwargs) 2025-08-26T20:32:27.9879193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9879685Z return func(*args, **kwargs) 2025-08-26T20:32:27.9880080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9880487Z return func(*args, **kwargs) 2025-08-26T20:32:27.9880917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9881379Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9881801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9882197Z return func(*args, **kwargs) 2025-08-26T20:32:27.9882593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9883025Z return func(*args, **kwargs) 2025-08-26T20:32:27.9883416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9883808Z return func(*args, **kwargs) 2025-08-26T20:32:27.9884236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:27.9884674Z self_outputs = self.self( 2025-08-26T20:32:27.9885077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9885477Z return func(*args, **kwargs) 2025-08-26T20:32:27.9885866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9886271Z return func(*args, **kwargs) 2025-08-26T20:32:27.9886665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9887079Z return func(*args, **kwargs) 2025-08-26T20:32:27.9887505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:27.9888026Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:27.9888258Z 2025-08-26T20:32:27.9888378Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9888863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9889202Z return mod(**inputs) 2025-08-26T20:32:27.9889538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9889924Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9890355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9890786Z outputs = self.layoutlm( 2025-08-26T20:32:27.9891167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9891557Z return func(*args, **kwargs) 2025-08-26T20:32:27.9891939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9892319Z return func(*args, **kwargs) 2025-08-26T20:32:27.9892665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9893025Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9893422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9893830Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9894225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9894643Z return func(*args, **kwargs) 2025-08-26T20:32:27.9895019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9895413Z return func(*args, **kwargs) 2025-08-26T20:32:27.9895798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9896387Z return func(*args, **kwargs) 2025-08-26T20:32:27.9896600Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9896960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9897325Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9897734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9898249Z layer_outputs = layer_module( 2025-08-26T20:32:27.9898614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9899006Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9899417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9899816Z return func(*args, **kwargs) 2025-08-26T20:32:27.9900200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9900588Z return func(*args, **kwargs) 2025-08-26T20:32:27.9900978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9901348Z return func(*args, **kwargs) 2025-08-26T20:32:27.9901746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9902166Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9902558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9902931Z return func(*args, **kwargs) 2025-08-26T20:32:27.9903293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9903666Z return func(*args, **kwargs) 2025-08-26T20:32:27.9904068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9904446Z return func(*args, **kwargs) 2025-08-26T20:32:27.9904836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:27.9905241Z self_outputs = self.self( 2025-08-26T20:32:27.9905607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9905973Z return func(*args, **kwargs) 2025-08-26T20:32:27.9906332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9906732Z return func(*args, **kwargs) 2025-08-26T20:32:27.9907102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9907461Z return func(*args, **kwargs) 2025-08-26T20:32:27.9907871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:27.9908372Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:27.9908578Z 2025-08-26T20:32:27.9908702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9909089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9909469Z return mod(**inputs) 2025-08-26T20:32:27.9909822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9910203Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9910606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9911009Z outputs = self.layoutlm( 2025-08-26T20:32:27.9911381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9911758Z return func(*args, **kwargs) 2025-08-26T20:32:27.9912125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9912506Z return func(*args, **kwargs) 2025-08-26T20:32:27.9912860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9913228Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9913657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9914095Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9914490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9914887Z return func(*args, **kwargs) 2025-08-26T20:32:27.9915270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9915673Z return func(*args, **kwargs) 2025-08-26T20:32:27.9916055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9916446Z return func(*args, **kwargs) 2025-08-26T20:32:27.9916664Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9917061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9917440Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9917871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9918292Z layer_outputs = layer_module( 2025-08-26T20:32:27.9918692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9919084Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9919572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9919992Z return func(*args, **kwargs) 2025-08-26T20:32:27.9920395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9920814Z return func(*args, **kwargs) 2025-08-26T20:32:27.9921202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9921623Z return func(*args, **kwargs) 2025-08-26T20:32:27.9922016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9922441Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9922837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9923214Z return func(*args, **kwargs) 2025-08-26T20:32:27.9923570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9923950Z return func(*args, **kwargs) 2025-08-26T20:32:27.9924339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9924715Z return func(*args, **kwargs) 2025-08-26T20:32:27.9925145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:27.9925568Z self_outputs = self.self( 2025-08-26T20:32:27.9925967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9926372Z return func(*args, **kwargs) 2025-08-26T20:32:27.9926755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9927151Z return func(*args, **kwargs) 2025-08-26T20:32:27.9927504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9927913Z return func(*args, **kwargs) 2025-08-26T20:32:27.9928314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:27.9928824Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:27.9929042Z 2025-08-26T20:32:27.9929131Z cudagraph partition due to non gpu ops 2025-08-26T20:32:27.9929363Z cudagraph partition due to non gpu ops 2025-08-26T20:32:27.9929623Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9930018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9930368Z return mod(**inputs) 2025-08-26T20:32:27.9930715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9931099Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9931535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9931962Z outputs = self.layoutlm( 2025-08-26T20:32:27.9932340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9932746Z return func(*args, **kwargs) 2025-08-26T20:32:27.9933112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9933486Z return func(*args, **kwargs) 2025-08-26T20:32:27.9933848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9934220Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9934648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9935077Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9935472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9935874Z return func(*args, **kwargs) 2025-08-26T20:32:27.9936235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9936624Z return func(*args, **kwargs) 2025-08-26T20:32:27.9936984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9937357Z return func(*args, **kwargs) 2025-08-26T20:32:27.9937549Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9937907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9938274Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9938682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9939106Z layer_outputs = layer_module( 2025-08-26T20:32:27.9939468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9939857Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9940275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9940683Z return func(*args, **kwargs) 2025-08-26T20:32:27.9941076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9941501Z return func(*args, **kwargs) 2025-08-26T20:32:27.9941889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9942285Z return func(*args, **kwargs) 2025-08-26T20:32:27.9942728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:27.9943182Z self_attention_outputs = self.attention( 2025-08-26T20:32:27.9943604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9944011Z return func(*args, **kwargs) 2025-08-26T20:32:27.9944409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9944811Z return func(*args, **kwargs) 2025-08-26T20:32:27.9945208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9945613Z return func(*args, **kwargs) 2025-08-26T20:32:27.9946036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:27.9946546Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:27.9947044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:27.9947508Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:27.9947671Z 2025-08-26T20:32:27.9947789Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9948193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9948556Z return mod(**inputs) 2025-08-26T20:32:27.9948943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9949344Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9949790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9950240Z outputs = self.layoutlm( 2025-08-26T20:32:27.9950630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9951043Z return func(*args, **kwargs) 2025-08-26T20:32:27.9951435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9951868Z return func(*args, **kwargs) 2025-08-26T20:32:27.9952243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9952627Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9953074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9953538Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9953947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9954348Z return func(*args, **kwargs) 2025-08-26T20:32:27.9954771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9955169Z return func(*args, **kwargs) 2025-08-26T20:32:27.9955556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9955950Z return func(*args, **kwargs) 2025-08-26T20:32:27.9956153Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9956537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9956915Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9957345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9957769Z layer_outputs = layer_module( 2025-08-26T20:32:27.9958146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9958574Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9958980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9959478Z return func(*args, **kwargs) 2025-08-26T20:32:27.9959873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9960281Z return func(*args, **kwargs) 2025-08-26T20:32:27.9960676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9961092Z return func(*args, **kwargs) 2025-08-26T20:32:27.9961502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:27.9961955Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:27.9962395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:27.9962824Z return forward_fn(*input_tensors) 2025-08-26T20:32:27.9963289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:27.9963799Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:27.9964315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:27.9964764Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:27.9964914Z 2025-08-26T20:32:27.9965038Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9965432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9965779Z return mod(**inputs) 2025-08-26T20:32:27.9966136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9966522Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9966954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9967400Z outputs = self.layoutlm( 2025-08-26T20:32:27.9967780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9968176Z return func(*args, **kwargs) 2025-08-26T20:32:27.9968568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9968967Z return func(*args, **kwargs) 2025-08-26T20:32:27.9969321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9969702Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9970165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9970607Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9971015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9971412Z return func(*args, **kwargs) 2025-08-26T20:32:27.9971806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9972214Z return func(*args, **kwargs) 2025-08-26T20:32:27.9972584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9972981Z return func(*args, **kwargs) 2025-08-26T20:32:27.9973195Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9973578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9973958Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9974366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9974771Z layer_outputs = layer_module( 2025-08-26T20:32:27.9975128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9975504Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9975893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9976258Z return func(*args, **kwargs) 2025-08-26T20:32:27.9976623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9976995Z return func(*args, **kwargs) 2025-08-26T20:32:27.9977363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9977741Z return func(*args, **kwargs) 2025-08-26T20:32:27.9978173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:27.9978633Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:27.9979073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:27.9979520Z return forward_fn(*input_tensors) 2025-08-26T20:32:27.9979981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:27.9980503Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:27.9980958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:27.9981412Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:27.9981800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:27.9982150Z return self.act(input) 2025-08-26T20:32:27.9982283Z 2025-08-26T20:32:27.9982390Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:27.9982766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:27.9983106Z return mod(**inputs) 2025-08-26T20:32:27.9983447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9983809Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9984228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:27.9984641Z outputs = self.layoutlm( 2025-08-26T20:32:27.9985024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9985399Z return func(*args, **kwargs) 2025-08-26T20:32:27.9985759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9986134Z return func(*args, **kwargs) 2025-08-26T20:32:27.9986477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9986839Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9987240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:27.9987654Z encoder_outputs = self.encoder( 2025-08-26T20:32:27.9988031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9988432Z return func(*args, **kwargs) 2025-08-26T20:32:27.9988829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9989230Z return func(*args, **kwargs) 2025-08-26T20:32:27.9989617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9989991Z return func(*args, **kwargs) 2025-08-26T20:32:27.9990188Z [Previous line repeated 1 more time] 2025-08-26T20:32:27.9990543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:27.9990904Z output = func(self, *args, **kwargs) 2025-08-26T20:32:27.9991304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:27.9991708Z layer_outputs = layer_module( 2025-08-26T20:32:27.9992063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:27.9992426Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:27.9992810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9993188Z return func(*args, **kwargs) 2025-08-26T20:32:27.9993551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9994788Z return func(*args, **kwargs) 2025-08-26T20:32:27.9995189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:27.9995584Z return func(*args, **kwargs) 2025-08-26T20:32:27.9996001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:27.9996617Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:27.9997049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:27.9997471Z return forward_fn(*input_tensors) 2025-08-26T20:32:27.9997977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:27.9998510Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:27.9999004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:27.9999491Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:27.9999654Z 2025-08-26T20:32:27.9999769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0000167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0000576Z return mod(**inputs) 2025-08-26T20:32:28.0000939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0001345Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0001794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0002225Z outputs = self.layoutlm( 2025-08-26T20:32:28.0002613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0003002Z return func(*args, **kwargs) 2025-08-26T20:32:28.0003389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0003789Z return func(*args, **kwargs) 2025-08-26T20:32:28.0004153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0004563Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0004980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0005413Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0005808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0006202Z return func(*args, **kwargs) 2025-08-26T20:32:28.0006576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0006973Z return func(*args, **kwargs) 2025-08-26T20:32:28.0007354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0007753Z return func(*args, **kwargs) 2025-08-26T20:32:28.0007965Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0008339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0008717Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0009144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0009570Z layer_outputs = layer_module( 2025-08-26T20:32:28.0009917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0010320Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0010703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0011072Z return func(*args, **kwargs) 2025-08-26T20:32:28.0011436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0011802Z return func(*args, **kwargs) 2025-08-26T20:32:28.0012166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0012537Z return func(*args, **kwargs) 2025-08-26T20:32:28.0012945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0013374Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0013772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0014145Z return func(*args, **kwargs) 2025-08-26T20:32:28.0014505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0014876Z return func(*args, **kwargs) 2025-08-26T20:32:28.0015238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0015654Z return func(*args, **kwargs) 2025-08-26T20:32:28.0016051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0016488Z self_outputs = self.self( 2025-08-26T20:32:28.0016878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0017267Z return func(*args, **kwargs) 2025-08-26T20:32:28.0017655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0018032Z return func(*args, **kwargs) 2025-08-26T20:32:28.0018398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0018753Z return func(*args, **kwargs) 2025-08-26T20:32:28.0019168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0019649Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0019850Z 2025-08-26T20:32:28.0019965Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0020337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0020668Z return mod(**inputs) 2025-08-26T20:32:28.0021024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0021407Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0021812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0022223Z outputs = self.layoutlm( 2025-08-26T20:32:28.0022581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0022958Z return func(*args, **kwargs) 2025-08-26T20:32:28.0023325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0023702Z return func(*args, **kwargs) 2025-08-26T20:32:28.0024037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0024393Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0024854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0025300Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0025709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0026108Z return func(*args, **kwargs) 2025-08-26T20:32:28.0026496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0026872Z return func(*args, **kwargs) 2025-08-26T20:32:28.0027232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0027618Z return func(*args, **kwargs) 2025-08-26T20:32:28.0027817Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0028173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0028539Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0028956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0029374Z layer_outputs = layer_module( 2025-08-26T20:32:28.0029747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0030168Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0030574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0030968Z return func(*args, **kwargs) 2025-08-26T20:32:28.0031362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0031755Z return func(*args, **kwargs) 2025-08-26T20:32:28.0032121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0032491Z return func(*args, **kwargs) 2025-08-26T20:32:28.0032896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0033348Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0033776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0034174Z return func(*args, **kwargs) 2025-08-26T20:32:28.0034555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0034950Z return func(*args, **kwargs) 2025-08-26T20:32:28.0035336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0035730Z return func(*args, **kwargs) 2025-08-26T20:32:28.0036155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0036575Z self_outputs = self.self( 2025-08-26T20:32:28.0036960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0037353Z return func(*args, **kwargs) 2025-08-26T20:32:28.0037738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0038142Z return func(*args, **kwargs) 2025-08-26T20:32:28.0038525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0038927Z return func(*args, **kwargs) 2025-08-26T20:32:28.0039437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0040008Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0040227Z 2025-08-26T20:32:28.0040360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0040747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0041101Z return mod(**inputs) 2025-08-26T20:32:28.0041459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0041847Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0042256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0042705Z outputs = self.layoutlm( 2025-08-26T20:32:28.0043090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0043483Z return func(*args, **kwargs) 2025-08-26T20:32:28.0043868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0044255Z return func(*args, **kwargs) 2025-08-26T20:32:28.0044619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0045002Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0045441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0045895Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0046297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0046698Z return func(*args, **kwargs) 2025-08-26T20:32:28.0047080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0047478Z return func(*args, **kwargs) 2025-08-26T20:32:28.0047858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0048261Z return func(*args, **kwargs) 2025-08-26T20:32:28.0048471Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0048860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0049266Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0049707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0050198Z layer_outputs = layer_module( 2025-08-26T20:32:28.0050584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0050974Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0051378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0051776Z return func(*args, **kwargs) 2025-08-26T20:32:28.0052157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0052553Z return func(*args, **kwargs) 2025-08-26T20:32:28.0052939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0053325Z return func(*args, **kwargs) 2025-08-26T20:32:28.0053740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0054189Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0054599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0054997Z return func(*args, **kwargs) 2025-08-26T20:32:28.0055405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0055801Z return func(*args, **kwargs) 2025-08-26T20:32:28.0056186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0056602Z return func(*args, **kwargs) 2025-08-26T20:32:28.0057009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0057442Z self_outputs = self.self( 2025-08-26T20:32:28.0057828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0058217Z return func(*args, **kwargs) 2025-08-26T20:32:28.0058594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0059008Z return func(*args, **kwargs) 2025-08-26T20:32:28.0059394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0059804Z return func(*args, **kwargs) 2025-08-26T20:32:28.0060219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0060732Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0060976Z 2025-08-26T20:32:28.0061077Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0061299Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0061545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0061917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0062245Z return mod(**inputs) 2025-08-26T20:32:28.0062594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0062977Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0063409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0063833Z outputs = self.layoutlm( 2025-08-26T20:32:28.0064201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0064600Z return func(*args, **kwargs) 2025-08-26T20:32:28.0064964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0065341Z return func(*args, **kwargs) 2025-08-26T20:32:28.0065679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0066053Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0066486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0066918Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0067328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0067715Z return func(*args, **kwargs) 2025-08-26T20:32:28.0068083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0068462Z return func(*args, **kwargs) 2025-08-26T20:32:28.0068844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0069250Z return func(*args, **kwargs) 2025-08-26T20:32:28.0069455Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0069860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0070244Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0070675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0071107Z layer_outputs = layer_module( 2025-08-26T20:32:28.0071500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0071899Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0072306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0072719Z return func(*args, **kwargs) 2025-08-26T20:32:28.0073145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0073546Z return func(*args, **kwargs) 2025-08-26T20:32:28.0073934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0074326Z return func(*args, **kwargs) 2025-08-26T20:32:28.0074745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0075201Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0075629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0076067Z return func(*args, **kwargs) 2025-08-26T20:32:28.0076463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0076868Z return func(*args, **kwargs) 2025-08-26T20:32:28.0077268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0077680Z return func(*args, **kwargs) 2025-08-26T20:32:28.0078113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0078626Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0079129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0079738Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0079912Z 2025-08-26T20:32:28.0080031Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0080441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0080816Z return mod(**inputs) 2025-08-26T20:32:28.0081180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0081573Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0082021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0082470Z outputs = self.layoutlm( 2025-08-26T20:32:28.0082876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0083298Z return func(*args, **kwargs) 2025-08-26T20:32:28.0083700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0084115Z return func(*args, **kwargs) 2025-08-26T20:32:28.0084483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0084896Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0085333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0085816Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0086239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0086665Z return func(*args, **kwargs) 2025-08-26T20:32:28.0087061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0087462Z return func(*args, **kwargs) 2025-08-26T20:32:28.0087866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0088275Z return func(*args, **kwargs) 2025-08-26T20:32:28.0088493Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0088907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0089311Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0089767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0090213Z layer_outputs = layer_module( 2025-08-26T20:32:28.0090601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0091006Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0091435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0091879Z return func(*args, **kwargs) 2025-08-26T20:32:28.0092271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0092679Z return func(*args, **kwargs) 2025-08-26T20:32:28.0093083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0093499Z return func(*args, **kwargs) 2025-08-26T20:32:28.0093930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0094406Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0094853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0095290Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0095787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0096480Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0097019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0097495Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0097652Z 2025-08-26T20:32:28.0097791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0098212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0098578Z return mod(**inputs) 2025-08-26T20:32:28.0098950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0099350Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0099812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0100258Z outputs = self.layoutlm( 2025-08-26T20:32:28.0100667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0101089Z return func(*args, **kwargs) 2025-08-26T20:32:28.0101493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0101985Z return func(*args, **kwargs) 2025-08-26T20:32:28.0102366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0102766Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0103205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0103664Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0104095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0104499Z return func(*args, **kwargs) 2025-08-26T20:32:28.0104913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0105321Z return func(*args, **kwargs) 2025-08-26T20:32:28.0105706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0106092Z return func(*args, **kwargs) 2025-08-26T20:32:28.0106304Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0106682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0107067Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0107502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0107964Z layer_outputs = layer_module( 2025-08-26T20:32:28.0108336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0108734Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0109136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0109525Z return func(*args, **kwargs) 2025-08-26T20:32:28.0109917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0110313Z return func(*args, **kwargs) 2025-08-26T20:32:28.0110698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0111087Z return func(*args, **kwargs) 2025-08-26T20:32:28.0111528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0111977Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0112413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0112831Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0113288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0113804Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0114287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0114775Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0115201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0115568Z return self.act(input) 2025-08-26T20:32:28.0115699Z 2025-08-26T20:32:28.0115811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0116208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0116565Z return mod(**inputs) 2025-08-26T20:32:28.0116918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0117326Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0117781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0118229Z outputs = self.layoutlm( 2025-08-26T20:32:28.0118634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0119038Z return func(*args, **kwargs) 2025-08-26T20:32:28.0119513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0119946Z return func(*args, **kwargs) 2025-08-26T20:32:28.0120356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0120764Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0121196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0121652Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0122066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0122496Z return func(*args, **kwargs) 2025-08-26T20:32:28.0122892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0123320Z return func(*args, **kwargs) 2025-08-26T20:32:28.0123714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0124131Z return func(*args, **kwargs) 2025-08-26T20:32:28.0124344Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0124732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0125130Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0125577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0126021Z layer_outputs = layer_module( 2025-08-26T20:32:28.0126407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0126800Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0127251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0127675Z return func(*args, **kwargs) 2025-08-26T20:32:28.0128081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0128495Z return func(*args, **kwargs) 2025-08-26T20:32:28.0128903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0129319Z return func(*args, **kwargs) 2025-08-26T20:32:28.0129745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0130222Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0130666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0131110Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0131578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0132130Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0132638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0133085Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0133243Z 2025-08-26T20:32:28.0133399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0133796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0134177Z return mod(**inputs) 2025-08-26T20:32:28.0134532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0134929Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0135368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0135803Z outputs = self.layoutlm( 2025-08-26T20:32:28.0136225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0136628Z return func(*args, **kwargs) 2025-08-26T20:32:28.0137030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0137425Z return func(*args, **kwargs) 2025-08-26T20:32:28.0137789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0138163Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0138594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0139081Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0139480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0139874Z return func(*args, **kwargs) 2025-08-26T20:32:28.0140259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0140659Z return func(*args, **kwargs) 2025-08-26T20:32:28.0141046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0141438Z return func(*args, **kwargs) 2025-08-26T20:32:28.0141652Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0142032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0142419Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0142851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0143258Z layer_outputs = layer_module( 2025-08-26T20:32:28.0143607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0143979Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0144360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0144738Z return func(*args, **kwargs) 2025-08-26T20:32:28.0145097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0145465Z return func(*args, **kwargs) 2025-08-26T20:32:28.0145828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0146202Z return func(*args, **kwargs) 2025-08-26T20:32:28.0146601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0147014Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0147406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0147774Z return func(*args, **kwargs) 2025-08-26T20:32:28.0148154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0148538Z return func(*args, **kwargs) 2025-08-26T20:32:28.0148914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0149301Z return func(*args, **kwargs) 2025-08-26T20:32:28.0149695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0150110Z self_outputs = self.self( 2025-08-26T20:32:28.0150493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0150892Z return func(*args, **kwargs) 2025-08-26T20:32:28.0151308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0151718Z return func(*args, **kwargs) 2025-08-26T20:32:28.0152110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0152504Z return func(*args, **kwargs) 2025-08-26T20:32:28.0152935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0153448Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0153668Z 2025-08-26T20:32:28.0153820Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0154212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0154558Z return mod(**inputs) 2025-08-26T20:32:28.0154918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0155306Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0155741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0156167Z outputs = self.layoutlm( 2025-08-26T20:32:28.0156556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0156951Z return func(*args, **kwargs) 2025-08-26T20:32:28.0157862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0158291Z return func(*args, **kwargs) 2025-08-26T20:32:28.0158657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0159056Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0159596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0160054Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0160467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0160870Z return func(*args, **kwargs) 2025-08-26T20:32:28.0161270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0161679Z return func(*args, **kwargs) 2025-08-26T20:32:28.0162072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0162461Z return func(*args, **kwargs) 2025-08-26T20:32:28.0162674Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0163056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0163436Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0163891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0164323Z layer_outputs = layer_module( 2025-08-26T20:32:28.0164709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0165104Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0165511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0165908Z return func(*args, **kwargs) 2025-08-26T20:32:28.0166289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0166688Z return func(*args, **kwargs) 2025-08-26T20:32:28.0167089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0167487Z return func(*args, **kwargs) 2025-08-26T20:32:28.0167912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0168363Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0168780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0169183Z return func(*args, **kwargs) 2025-08-26T20:32:28.0169561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0169991Z return func(*args, **kwargs) 2025-08-26T20:32:28.0170413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0170822Z return func(*args, **kwargs) 2025-08-26T20:32:28.0171247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0171685Z self_outputs = self.self( 2025-08-26T20:32:28.0172091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0172492Z return func(*args, **kwargs) 2025-08-26T20:32:28.0172880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0173285Z return func(*args, **kwargs) 2025-08-26T20:32:28.0173697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0174159Z return func(*args, **kwargs) 2025-08-26T20:32:28.0174588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0175107Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0175318Z 2025-08-26T20:32:28.0175444Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0175830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0176193Z return mod(**inputs) 2025-08-26T20:32:28.0176548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0176936Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0177371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0177812Z outputs = self.layoutlm( 2025-08-26T20:32:28.0178200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0178601Z return func(*args, **kwargs) 2025-08-26T20:32:28.0178979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0179364Z return func(*args, **kwargs) 2025-08-26T20:32:28.0179741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0180133Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0180569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0180996Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0181402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0181778Z return func(*args, **kwargs) 2025-08-26T20:32:28.0182143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0182535Z return func(*args, **kwargs) 2025-08-26T20:32:28.0182895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0183284Z return func(*args, **kwargs) 2025-08-26T20:32:28.0183494Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0183874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0184255Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0184684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0185132Z layer_outputs = layer_module( 2025-08-26T20:32:28.0185512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0185907Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0186315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0186720Z return func(*args, **kwargs) 2025-08-26T20:32:28.0187113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0187514Z return func(*args, **kwargs) 2025-08-26T20:32:28.0187896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0188305Z return func(*args, **kwargs) 2025-08-26T20:32:28.0188744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0189222Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0189639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0190036Z return func(*args, **kwargs) 2025-08-26T20:32:28.0190424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0190829Z return func(*args, **kwargs) 2025-08-26T20:32:28.0191218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0191623Z return func(*args, **kwargs) 2025-08-26T20:32:28.0192039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0192476Z self_outputs = self.self( 2025-08-26T20:32:28.0192868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0193274Z return func(*args, **kwargs) 2025-08-26T20:32:28.0193667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0194082Z return func(*args, **kwargs) 2025-08-26T20:32:28.0194483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0194918Z return func(*args, **kwargs) 2025-08-26T20:32:28.0195351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0195872Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0196102Z 2025-08-26T20:32:28.0196308Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0196565Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0196834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0197234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0197597Z return mod(**inputs) 2025-08-26T20:32:28.0198014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0198416Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0198863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0199364Z outputs = self.layoutlm( 2025-08-26T20:32:28.0199795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0200235Z return func(*args, **kwargs) 2025-08-26T20:32:28.0200631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0201062Z return func(*args, **kwargs) 2025-08-26T20:32:28.0201429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0201816Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0202249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0202686Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0203079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0203474Z return func(*args, **kwargs) 2025-08-26T20:32:28.0203855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0204246Z return func(*args, **kwargs) 2025-08-26T20:32:28.0204664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0205037Z return func(*args, **kwargs) 2025-08-26T20:32:28.0205234Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0205596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0205957Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0206358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0206766Z layer_outputs = layer_module( 2025-08-26T20:32:28.0207124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0207518Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0207922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0208315Z return func(*args, **kwargs) 2025-08-26T20:32:28.0208686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0209060Z return func(*args, **kwargs) 2025-08-26T20:32:28.0209425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0209793Z return func(*args, **kwargs) 2025-08-26T20:32:28.0210222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0210647Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0211031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0211404Z return func(*args, **kwargs) 2025-08-26T20:32:28.0211761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0212141Z return func(*args, **kwargs) 2025-08-26T20:32:28.0212533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0212960Z return func(*args, **kwargs) 2025-08-26T20:32:28.0213385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0213917Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0214421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0214870Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0215014Z 2025-08-26T20:32:28.0215130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0215497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0215855Z return mod(**inputs) 2025-08-26T20:32:28.0216193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0216559Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0216977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0217380Z outputs = self.layoutlm( 2025-08-26T20:32:28.0217749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0218125Z return func(*args, **kwargs) 2025-08-26T20:32:28.0218490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0218859Z return func(*args, **kwargs) 2025-08-26T20:32:28.0219224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0219586Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0220020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0220460Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0220853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0221264Z return func(*args, **kwargs) 2025-08-26T20:32:28.0221655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0222029Z return func(*args, **kwargs) 2025-08-26T20:32:28.0222385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0222762Z return func(*args, **kwargs) 2025-08-26T20:32:28.0222961Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0223323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0223683Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0224087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0224501Z layer_outputs = layer_module( 2025-08-26T20:32:28.0224892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0225281Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0225693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0226097Z return func(*args, **kwargs) 2025-08-26T20:32:28.0226478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0226877Z return func(*args, **kwargs) 2025-08-26T20:32:28.0227237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0227606Z return func(*args, **kwargs) 2025-08-26T20:32:28.0228025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0228456Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0228873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0229276Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0229721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0230237Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0230742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0231183Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0231326Z 2025-08-26T20:32:28.0231443Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0231804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0232146Z return mod(**inputs) 2025-08-26T20:32:28.0232503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0232885Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0233313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0233746Z outputs = self.layoutlm( 2025-08-26T20:32:28.0234153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0234554Z return func(*args, **kwargs) 2025-08-26T20:32:28.0234939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0235333Z return func(*args, **kwargs) 2025-08-26T20:32:28.0235696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0236079Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0236516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0236940Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0237337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0237751Z return func(*args, **kwargs) 2025-08-26T20:32:28.0238153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0238560Z return func(*args, **kwargs) 2025-08-26T20:32:28.0238948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0239433Z return func(*args, **kwargs) 2025-08-26T20:32:28.0239653Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0240079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0240473Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0240925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0241453Z layer_outputs = layer_module( 2025-08-26T20:32:28.0241835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0242232Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0242639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0243049Z return func(*args, **kwargs) 2025-08-26T20:32:28.0243457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0243855Z return func(*args, **kwargs) 2025-08-26T20:32:28.0244236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0244623Z return func(*args, **kwargs) 2025-08-26T20:32:28.0245046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0245490Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0245936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0246393Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0246865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0247379Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0247864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0248346Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0248752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0249125Z return self.act(input) 2025-08-26T20:32:28.0249256Z 2025-08-26T20:32:28.0249370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0249790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0250136Z return mod(**inputs) 2025-08-26T20:32:28.0250464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0250830Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0251241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0251650Z outputs = self.layoutlm( 2025-08-26T20:32:28.0252010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0252391Z return func(*args, **kwargs) 2025-08-26T20:32:28.0252777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0253176Z return func(*args, **kwargs) 2025-08-26T20:32:28.0253542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0253914Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0254358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0254768Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0255185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0255575Z return func(*args, **kwargs) 2025-08-26T20:32:28.0255961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0256358Z return func(*args, **kwargs) 2025-08-26T20:32:28.0256748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0257149Z return func(*args, **kwargs) 2025-08-26T20:32:28.0257351Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0257731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0258094Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0258549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0258979Z layer_outputs = layer_module( 2025-08-26T20:32:28.0259357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0259748Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0260160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0260554Z return func(*args, **kwargs) 2025-08-26T20:32:28.0260937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0261362Z return func(*args, **kwargs) 2025-08-26T20:32:28.0261748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0262155Z return func(*args, **kwargs) 2025-08-26T20:32:28.0262573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0263031Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0263462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0263865Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0264308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0264855Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0265355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0265806Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0265956Z 2025-08-26T20:32:28.0266081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0266470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0266801Z return mod(**inputs) 2025-08-26T20:32:28.0267155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0267536Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0267970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0268404Z outputs = self.layoutlm( 2025-08-26T20:32:28.0268786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0269183Z return func(*args, **kwargs) 2025-08-26T20:32:28.0269567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0269962Z return func(*args, **kwargs) 2025-08-26T20:32:28.0270339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0270724Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0271154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0271595Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0272005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0272408Z return func(*args, **kwargs) 2025-08-26T20:32:28.0272803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0273210Z return func(*args, **kwargs) 2025-08-26T20:32:28.0273620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0274027Z return func(*args, **kwargs) 2025-08-26T20:32:28.0274238Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0274622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0275014Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0275451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0275877Z layer_outputs = layer_module( 2025-08-26T20:32:28.0276256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0276667Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0277094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0277508Z return func(*args, **kwargs) 2025-08-26T20:32:28.0277898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0278300Z return func(*args, **kwargs) 2025-08-26T20:32:28.0278698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0279116Z return func(*args, **kwargs) 2025-08-26T20:32:28.0279623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0280165Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0280612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0281035Z return func(*args, **kwargs) 2025-08-26T20:32:28.0281437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0281844Z return func(*args, **kwargs) 2025-08-26T20:32:28.0282241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0282641Z return func(*args, **kwargs) 2025-08-26T20:32:28.0283077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0283522Z self_outputs = self.self( 2025-08-26T20:32:28.0283922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0284331Z return func(*args, **kwargs) 2025-08-26T20:32:28.0284722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0285134Z return func(*args, **kwargs) 2025-08-26T20:32:28.0285522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0285922Z return func(*args, **kwargs) 2025-08-26T20:32:28.0286347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0286860Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0287078Z 2025-08-26T20:32:28.0287210Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0287595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0287947Z return mod(**inputs) 2025-08-26T20:32:28.0288300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0288682Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0289126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0289556Z outputs = self.layoutlm( 2025-08-26T20:32:28.0289944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0290337Z return func(*args, **kwargs) 2025-08-26T20:32:28.0290724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0291122Z return func(*args, **kwargs) 2025-08-26T20:32:28.0291483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0291889Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0292326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0292760Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0293166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0293562Z return func(*args, **kwargs) 2025-08-26T20:32:28.0293949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0294344Z return func(*args, **kwargs) 2025-08-26T20:32:28.0294724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0295121Z return func(*args, **kwargs) 2025-08-26T20:32:28.0295350Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0295730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0296105Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0296661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0297094Z layer_outputs = layer_module( 2025-08-26T20:32:28.0297472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0297872Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0298272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0298670Z return func(*args, **kwargs) 2025-08-26T20:32:28.0299059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0299472Z return func(*args, **kwargs) 2025-08-26T20:32:28.0299831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0300206Z return func(*args, **kwargs) 2025-08-26T20:32:28.0300608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0301058Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0301527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0301922Z return func(*args, **kwargs) 2025-08-26T20:32:28.0302311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0302706Z return func(*args, **kwargs) 2025-08-26T20:32:28.0303091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0303492Z return func(*args, **kwargs) 2025-08-26T20:32:28.0303908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0304338Z self_outputs = self.self( 2025-08-26T20:32:28.0304770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0305172Z return func(*args, **kwargs) 2025-08-26T20:32:28.0305549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0305943Z return func(*args, **kwargs) 2025-08-26T20:32:28.0306326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0306733Z return func(*args, **kwargs) 2025-08-26T20:32:28.0307154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0307681Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0307902Z 2025-08-26T20:32:28.0308015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0308416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0308772Z return mod(**inputs) 2025-08-26T20:32:28.0309128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0309509Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0309944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0310368Z outputs = self.layoutlm( 2025-08-26T20:32:28.0310751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0311175Z return func(*args, **kwargs) 2025-08-26T20:32:28.0311561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0311956Z return func(*args, **kwargs) 2025-08-26T20:32:28.0312320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0312702Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0313134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0313569Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0313975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0314423Z return func(*args, **kwargs) 2025-08-26T20:32:28.0314810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0315221Z return func(*args, **kwargs) 2025-08-26T20:32:28.0315620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0316023Z return func(*args, **kwargs) 2025-08-26T20:32:28.0316236Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0316633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0317024Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0317455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0317886Z layer_outputs = layer_module( 2025-08-26T20:32:28.0318262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0318669Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0319099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0319580Z return func(*args, **kwargs) 2025-08-26T20:32:28.0320019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0320433Z return func(*args, **kwargs) 2025-08-26T20:32:28.0320876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0321284Z return func(*args, **kwargs) 2025-08-26T20:32:28.0321704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0322158Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0322577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0322994Z return func(*args, **kwargs) 2025-08-26T20:32:28.0323363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0323748Z return func(*args, **kwargs) 2025-08-26T20:32:28.0324119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0324519Z return func(*args, **kwargs) 2025-08-26T20:32:28.0324932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0325341Z self_outputs = self.self( 2025-08-26T20:32:28.0325710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0326075Z return func(*args, **kwargs) 2025-08-26T20:32:28.0326458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0326831Z return func(*args, **kwargs) 2025-08-26T20:32:28.0327189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0327554Z return func(*args, **kwargs) 2025-08-26T20:32:28.0327951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0328441Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0328645Z 2025-08-26T20:32:28.0328736Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0328957Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0329194Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0329565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0329897Z return mod(**inputs) 2025-08-26T20:32:28.0330234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0330589Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0330999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0331400Z outputs = self.layoutlm( 2025-08-26T20:32:28.0331786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0332167Z return func(*args, **kwargs) 2025-08-26T20:32:28.0332521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0332897Z return func(*args, **kwargs) 2025-08-26T20:32:28.0333243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0333611Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0334021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0334428Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0334821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0335197Z return func(*args, **kwargs) 2025-08-26T20:32:28.0335566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0335932Z return func(*args, **kwargs) 2025-08-26T20:32:28.0336296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0336669Z return func(*args, **kwargs) 2025-08-26T20:32:28.0336873Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0337256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0337618Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0338024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0338425Z layer_outputs = layer_module( 2025-08-26T20:32:28.0338784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0339148Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0339533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0339910Z return func(*args, **kwargs) 2025-08-26T20:32:28.0340277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0340671Z return func(*args, **kwargs) 2025-08-26T20:32:28.0341027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0341425Z return func(*args, **kwargs) 2025-08-26T20:32:28.0341841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0342291Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0342696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0343093Z return func(*args, **kwargs) 2025-08-26T20:32:28.0343457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0343831Z return func(*args, **kwargs) 2025-08-26T20:32:28.0344072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0344149Z return func(*args, **kwargs) 2025-08-26T20:32:28.0344420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0344553Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0344832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0344938Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0344943Z 2025-08-26T20:32:28.0345060Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0345265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0345340Z return mod(**inputs) 2025-08-26T20:32:28.0345569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0345651Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0345933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0346006Z outputs = self.layoutlm( 2025-08-26T20:32:28.0346279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0346350Z return func(*args, **kwargs) 2025-08-26T20:32:28.0346592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0346670Z return func(*args, **kwargs) 2025-08-26T20:32:28.0346904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0346991Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0347283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0347384Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0347650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0347724Z return func(*args, **kwargs) 2025-08-26T20:32:28.0347993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0348065Z return func(*args, **kwargs) 2025-08-26T20:32:28.0348330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0348401Z return func(*args, **kwargs) 2025-08-26T20:32:28.0348483Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0348728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0348825Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0349118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0349194Z layer_outputs = layer_module( 2025-08-26T20:32:28.0349437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0349530Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0349787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0349867Z return func(*args, **kwargs) 2025-08-26T20:32:28.0350123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0350194Z return func(*args, **kwargs) 2025-08-26T20:32:28.0350456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0350531Z return func(*args, **kwargs) 2025-08-26T20:32:28.0350821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0350915Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0351207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0351290Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0351639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0351782Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0352070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0352171Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0352176Z 2025-08-26T20:32:28.0352291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0352507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0352585Z return mod(**inputs) 2025-08-26T20:32:28.0352842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0352931Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0353218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0353303Z outputs = self.layoutlm( 2025-08-26T20:32:28.0353560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0353633Z return func(*args, **kwargs) 2025-08-26T20:32:28.0353899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0353992Z return func(*args, **kwargs) 2025-08-26T20:32:28.0354238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0354322Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0354613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0354704Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0354968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0355049Z return func(*args, **kwargs) 2025-08-26T20:32:28.0355310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0355402Z return func(*args, **kwargs) 2025-08-26T20:32:28.0355666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0355737Z return func(*args, **kwargs) 2025-08-26T20:32:28.0355829Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0356067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0356147Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0356447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0356523Z layer_outputs = layer_module( 2025-08-26T20:32:28.0356772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0356870Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0357135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0357212Z return func(*args, **kwargs) 2025-08-26T20:32:28.0357467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0357548Z return func(*args, **kwargs) 2025-08-26T20:32:28.0357806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0357886Z return func(*args, **kwargs) 2025-08-26T20:32:28.0358191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0358284Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0358576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0358661Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0358993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0359126Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0359743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0359881Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0360118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0360207Z return self.act(input) 2025-08-26T20:32:28.0360212Z 2025-08-26T20:32:28.0360330Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0360571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0360644Z return mod(**inputs) 2025-08-26T20:32:28.0360879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0360992Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0361279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0361365Z outputs = self.layoutlm( 2025-08-26T20:32:28.0361625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0361699Z return func(*args, **kwargs) 2025-08-26T20:32:28.0361966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0362041Z return func(*args, **kwargs) 2025-08-26T20:32:28.0362283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0362384Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0362675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0362762Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0363022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0363104Z return func(*args, **kwargs) 2025-08-26T20:32:28.0363364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0363445Z return func(*args, **kwargs) 2025-08-26T20:32:28.0363701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0363774Z return func(*args, **kwargs) 2025-08-26T20:32:28.0363863Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0364096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0364184Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0364476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0364552Z layer_outputs = layer_module( 2025-08-26T20:32:28.0364802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0364887Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0365173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0365248Z return func(*args, **kwargs) 2025-08-26T20:32:28.0365513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0365593Z return func(*args, **kwargs) 2025-08-26T20:32:28.0365856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0365936Z return func(*args, **kwargs) 2025-08-26T20:32:28.0366228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0366341Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0366624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0366707Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0367036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0367182Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0367477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0367586Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0367590Z 2025-08-26T20:32:28.0367704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0367931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0368001Z return mod(**inputs) 2025-08-26T20:32:28.0368244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0368326Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0368621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0368697Z outputs = self.layoutlm( 2025-08-26T20:32:28.0368954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0369056Z return func(*args, **kwargs) 2025-08-26T20:32:28.0369315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0369396Z return func(*args, **kwargs) 2025-08-26T20:32:28.0369630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0369712Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0370010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0370091Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0370354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0370426Z return func(*args, **kwargs) 2025-08-26T20:32:28.0370680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0370764Z return func(*args, **kwargs) 2025-08-26T20:32:28.0371021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0371100Z return func(*args, **kwargs) 2025-08-26T20:32:28.0371186Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0371418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0371505Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0371813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0371900Z layer_outputs = layer_module( 2025-08-26T20:32:28.0372143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0372236Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0372499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0372574Z return func(*args, **kwargs) 2025-08-26T20:32:28.0372841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0372929Z return func(*args, **kwargs) 2025-08-26T20:32:28.0373194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0373268Z return func(*args, **kwargs) 2025-08-26T20:32:28.0373554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0373651Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0373910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0373991Z return func(*args, **kwargs) 2025-08-26T20:32:28.0374267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0374338Z return func(*args, **kwargs) 2025-08-26T20:32:28.0374608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0374680Z return func(*args, **kwargs) 2025-08-26T20:32:28.0374979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0375058Z self_outputs = self.self( 2025-08-26T20:32:28.0375323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0375396Z return func(*args, **kwargs) 2025-08-26T20:32:28.0375653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0375763Z return func(*args, **kwargs) 2025-08-26T20:32:28.0376031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0376112Z return func(*args, **kwargs) 2025-08-26T20:32:28.0376420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0376581Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0376587Z 2025-08-26T20:32:28.0376709Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0376926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0377005Z return mod(**inputs) 2025-08-26T20:32:28.0377241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0377325Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0377622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0377699Z outputs = self.layoutlm( 2025-08-26T20:32:28.0377967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0378040Z return func(*args, **kwargs) 2025-08-26T20:32:28.0378333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0378406Z return func(*args, **kwargs) 2025-08-26T20:32:28.0378638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0378726Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0379013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0379103Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0379391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0379463Z return func(*args, **kwargs) 2025-08-26T20:32:28.0379752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0379828Z return func(*args, **kwargs) 2025-08-26T20:32:28.0380098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0380169Z return func(*args, **kwargs) 2025-08-26T20:32:28.0380252Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0380491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0380572Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0380906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0380984Z layer_outputs = layer_module( 2025-08-26T20:32:28.0381230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0381323Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0381592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0381674Z return func(*args, **kwargs) 2025-08-26T20:32:28.0381934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0382012Z return func(*args, **kwargs) 2025-08-26T20:32:28.0382281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0382371Z return func(*args, **kwargs) 2025-08-26T20:32:28.0382668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0382756Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0383034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0383105Z return func(*args, **kwargs) 2025-08-26T20:32:28.0383373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0383452Z return func(*args, **kwargs) 2025-08-26T20:32:28.0383719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0383797Z return func(*args, **kwargs) 2025-08-26T20:32:28.0384089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0384173Z self_outputs = self.self( 2025-08-26T20:32:28.0384455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0384529Z return func(*args, **kwargs) 2025-08-26T20:32:28.0384804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0384877Z return func(*args, **kwargs) 2025-08-26T20:32:28.0385175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0385252Z return func(*args, **kwargs) 2025-08-26T20:32:28.0385548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0385712Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0385719Z 2025-08-26T20:32:28.0385837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0386066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0386141Z return mod(**inputs) 2025-08-26T20:32:28.0386406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0386501Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0386803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0386891Z outputs = self.layoutlm( 2025-08-26T20:32:28.0387161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0387237Z return func(*args, **kwargs) 2025-08-26T20:32:28.0387508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0387606Z return func(*args, **kwargs) 2025-08-26T20:32:28.0387857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0387939Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0388251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0388332Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0388601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0388685Z return func(*args, **kwargs) 2025-08-26T20:32:28.0388949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0389032Z return func(*args, **kwargs) 2025-08-26T20:32:28.0389315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0389392Z return func(*args, **kwargs) 2025-08-26T20:32:28.0389485Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0389726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0389813Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0390114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0390195Z layer_outputs = layer_module( 2025-08-26T20:32:28.0390448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0390534Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0390804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0390883Z return func(*args, **kwargs) 2025-08-26T20:32:28.0391154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0391228Z return func(*args, **kwargs) 2025-08-26T20:32:28.0391493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0391575Z return func(*args, **kwargs) 2025-08-26T20:32:28.0391891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0391992Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0392258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0392333Z return func(*args, **kwargs) 2025-08-26T20:32:28.0392600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0392677Z return func(*args, **kwargs) 2025-08-26T20:32:28.0392946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0393019Z return func(*args, **kwargs) 2025-08-26T20:32:28.0393331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0393420Z self_outputs = self.self( 2025-08-26T20:32:28.0393687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0393771Z return func(*args, **kwargs) 2025-08-26T20:32:28.0394035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0394109Z return func(*args, **kwargs) 2025-08-26T20:32:28.0394382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0394480Z return func(*args, **kwargs) 2025-08-26T20:32:28.0394787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0394953Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0394957Z 2025-08-26T20:32:28.0395054Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0395145Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0395263Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0395491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0395563Z return mod(**inputs) 2025-08-26T20:32:28.0395816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0395918Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0396390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0396486Z outputs = self.layoutlm( 2025-08-26T20:32:28.0396755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0396838Z return func(*args, **kwargs) 2025-08-26T20:32:28.0397104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0397178Z return func(*args, **kwargs) 2025-08-26T20:32:28.0397426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0397508Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0397813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0397898Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0398170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0398244Z return func(*args, **kwargs) 2025-08-26T20:32:28.0398510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0398594Z return func(*args, **kwargs) 2025-08-26T20:32:28.0398906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0398991Z return func(*args, **kwargs) 2025-08-26T20:32:28.0399077Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0399376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0399481Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0399780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0399869Z layer_outputs = layer_module( 2025-08-26T20:32:28.0400144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0400234Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0400512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0400586Z return func(*args, **kwargs) 2025-08-26T20:32:28.0400856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0400932Z return func(*args, **kwargs) 2025-08-26T20:32:28.0401195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0401323Z return func(*args, **kwargs) 2025-08-26T20:32:28.0401621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0401723Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0401991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0402075Z return func(*args, **kwargs) 2025-08-26T20:32:28.0402341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0402416Z return func(*args, **kwargs) 2025-08-26T20:32:28.0402689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0402762Z return func(*args, **kwargs) 2025-08-26T20:32:28.0403068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0403257Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0403550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0403654Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0403659Z 2025-08-26T20:32:28.0403785Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0404009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0404081Z return mod(**inputs) 2025-08-26T20:32:28.0404325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0404404Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0404694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0404782Z outputs = self.layoutlm( 2025-08-26T20:32:28.0405041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0405118Z return func(*args, **kwargs) 2025-08-26T20:32:28.0405378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0405449Z return func(*args, **kwargs) 2025-08-26T20:32:28.0405707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0405789Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0406084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0406163Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0406420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0406502Z return func(*args, **kwargs) 2025-08-26T20:32:28.0406762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0406861Z return func(*args, **kwargs) 2025-08-26T20:32:28.0407119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0407197Z return func(*args, **kwargs) 2025-08-26T20:32:28.0407283Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0407518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0407603Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0407893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0407981Z layer_outputs = layer_module( 2025-08-26T20:32:28.0408236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0408320Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0408584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0408656Z return func(*args, **kwargs) 2025-08-26T20:32:28.0408919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0408992Z return func(*args, **kwargs) 2025-08-26T20:32:28.0409247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0409328Z return func(*args, **kwargs) 2025-08-26T20:32:28.0409615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0409735Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0410042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0410124Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0410458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0410592Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0410889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0410977Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0410980Z 2025-08-26T20:32:28.0411099Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0411318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0411392Z return mod(**inputs) 2025-08-26T20:32:28.0411638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0411718Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0412014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0412090Z outputs = self.layoutlm( 2025-08-26T20:32:28.0412365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0412446Z return func(*args, **kwargs) 2025-08-26T20:32:28.0412700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0412779Z return func(*args, **kwargs) 2025-08-26T20:32:28.0413015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0413102Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0413385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0413486Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0413748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0413821Z return func(*args, **kwargs) 2025-08-26T20:32:28.0414082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0414154Z return func(*args, **kwargs) 2025-08-26T20:32:28.0414409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0414491Z return func(*args, **kwargs) 2025-08-26T20:32:28.0414574Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0414834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0414917Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0415210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0415297Z layer_outputs = layer_module( 2025-08-26T20:32:28.0415544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0415638Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0415898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0415973Z return func(*args, **kwargs) 2025-08-26T20:32:28.0416239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0416331Z return func(*args, **kwargs) 2025-08-26T20:32:28.0416596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0416668Z return func(*args, **kwargs) 2025-08-26T20:32:28.0416966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0417059Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0417344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0417436Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0417760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0417901Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0418190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0418314Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0418555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0418632Z return self.act(input) 2025-08-26T20:32:28.0418637Z 2025-08-26T20:32:28.0418756Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0418990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0419069Z return mod(**inputs) 2025-08-26T20:32:28.0419306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0419388Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0419688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0419765Z outputs = self.layoutlm( 2025-08-26T20:32:28.0420034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0420128Z return func(*args, **kwargs) 2025-08-26T20:32:28.0420387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0420469Z return func(*args, **kwargs) 2025-08-26T20:32:28.0420705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0420805Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0421092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0421174Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0421459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0421531Z return func(*args, **kwargs) 2025-08-26T20:32:28.0421799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0421872Z return func(*args, **kwargs) 2025-08-26T20:32:28.0422136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0422212Z return func(*args, **kwargs) 2025-08-26T20:32:28.0422296Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0422537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0422612Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0422894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0422989Z layer_outputs = layer_module( 2025-08-26T20:32:28.0423213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0423301Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0423547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0423623Z return func(*args, **kwargs) 2025-08-26T20:32:28.0423867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0423933Z return func(*args, **kwargs) 2025-08-26T20:32:28.0424183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0424252Z return func(*args, **kwargs) 2025-08-26T20:32:28.0424537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0424625Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0424892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0424980Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0425292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0425456Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0425729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0425821Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0425825Z 2025-08-26T20:32:28.0425932Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0426137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0426213Z return mod(**inputs) 2025-08-26T20:32:28.0426435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0426543Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0426814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0426890Z outputs = self.layoutlm( 2025-08-26T20:32:28.0427141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0427210Z return func(*args, **kwargs) 2025-08-26T20:32:28.0427461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0427530Z return func(*args, **kwargs) 2025-08-26T20:32:28.0427781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0427857Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0428132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0428216Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0428461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0452579Z return func(*args, **kwargs) 2025-08-26T20:32:28.0453107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0453196Z return func(*args, **kwargs) 2025-08-26T20:32:28.0453491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0453713Z return func(*args, **kwargs) 2025-08-26T20:32:28.0453817Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0454071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0454163Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0454482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0454565Z layer_outputs = layer_module( 2025-08-26T20:32:28.0454823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0454916Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0455178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0455267Z return func(*args, **kwargs) 2025-08-26T20:32:28.0455524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0455611Z return func(*args, **kwargs) 2025-08-26T20:32:28.0455866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0455949Z return func(*args, **kwargs) 2025-08-26T20:32:28.0456245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0456372Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0456647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0456720Z return func(*args, **kwargs) 2025-08-26T20:32:28.0456985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0457060Z return func(*args, **kwargs) 2025-08-26T20:32:28.0457317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0457398Z return func(*args, **kwargs) 2025-08-26T20:32:28.0457726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0457817Z self_outputs = self.self( 2025-08-26T20:32:28.0458077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0458149Z return func(*args, **kwargs) 2025-08-26T20:32:28.0458413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0458486Z return func(*args, **kwargs) 2025-08-26T20:32:28.0458752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0458860Z return func(*args, **kwargs) 2025-08-26T20:32:28.0459147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0459319Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0459328Z 2025-08-26T20:32:28.0459455Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0459682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0459760Z return mod(**inputs) 2025-08-26T20:32:28.0460011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0460099Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0460386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0460497Z outputs = self.layoutlm( 2025-08-26T20:32:28.0460755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0460833Z return func(*args, **kwargs) 2025-08-26T20:32:28.0461089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0461161Z return func(*args, **kwargs) 2025-08-26T20:32:28.0461406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0461489Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0461786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0461868Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0462133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0462211Z return func(*args, **kwargs) 2025-08-26T20:32:28.0462468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0462548Z return func(*args, **kwargs) 2025-08-26T20:32:28.0462804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0462882Z return func(*args, **kwargs) 2025-08-26T20:32:28.0462972Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0463224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0463316Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0463602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0463691Z layer_outputs = layer_module( 2025-08-26T20:32:28.0463932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0464020Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0464306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0464382Z return func(*args, **kwargs) 2025-08-26T20:32:28.0464642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0464716Z return func(*args, **kwargs) 2025-08-26T20:32:28.0464978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0465050Z return func(*args, **kwargs) 2025-08-26T20:32:28.0465336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0465436Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0465712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0465791Z return func(*args, **kwargs) 2025-08-26T20:32:28.0466051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0466125Z return func(*args, **kwargs) 2025-08-26T20:32:28.0466399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0466472Z return func(*args, **kwargs) 2025-08-26T20:32:28.0466782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0466859Z self_outputs = self.self( 2025-08-26T20:32:28.0467115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0467219Z return func(*args, **kwargs) 2025-08-26T20:32:28.0467472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0467553Z return func(*args, **kwargs) 2025-08-26T20:32:28.0467807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0467888Z return func(*args, **kwargs) 2025-08-26T20:32:28.0468190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0468350Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0468356Z 2025-08-26T20:32:28.0468486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0468713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0468799Z return mod(**inputs) 2025-08-26T20:32:28.0469040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0469121Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0469427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0469506Z outputs = self.layoutlm( 2025-08-26T20:32:28.0469797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0469873Z return func(*args, **kwargs) 2025-08-26T20:32:28.0470135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0470218Z return func(*args, **kwargs) 2025-08-26T20:32:28.0470457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0470550Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0470847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0470936Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0471220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0471298Z return func(*args, **kwargs) 2025-08-26T20:32:28.0471570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0471643Z return func(*args, **kwargs) 2025-08-26T20:32:28.0471911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0471987Z return func(*args, **kwargs) 2025-08-26T20:32:28.0472076Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0472343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0472427Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0472736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0472821Z layer_outputs = layer_module( 2025-08-26T20:32:28.0473068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0473169Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0473437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0473523Z return func(*args, **kwargs) 2025-08-26T20:32:28.0473790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0474293Z return func(*args, **kwargs) 2025-08-26T20:32:28.0474564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0474640Z return func(*args, **kwargs) 2025-08-26T20:32:28.0474950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0475040Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0475314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0475389Z return func(*args, **kwargs) 2025-08-26T20:32:28.0475652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0475736Z return func(*args, **kwargs) 2025-08-26T20:32:28.0475999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0476086Z return func(*args, **kwargs) 2025-08-26T20:32:28.0476382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0476460Z self_outputs = self.self( 2025-08-26T20:32:28.0476733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0476806Z return func(*args, **kwargs) 2025-08-26T20:32:28.0477097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0477175Z return func(*args, **kwargs) 2025-08-26T20:32:28.0477439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0477523Z return func(*args, **kwargs) 2025-08-26T20:32:28.0477823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0478001Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0478006Z 2025-08-26T20:32:28.0478098Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0478228Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0478351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0478577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0478661Z return mod(**inputs) 2025-08-26T20:32:28.0478903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0478992Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0479389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0479506Z outputs = self.layoutlm( 2025-08-26T20:32:28.0479784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0479861Z return func(*args, **kwargs) 2025-08-26T20:32:28.0480138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0480212Z return func(*args, **kwargs) 2025-08-26T20:32:28.0480456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0480547Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0480847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0480941Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0481206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0481315Z return func(*args, **kwargs) 2025-08-26T20:32:28.0481582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0481656Z return func(*args, **kwargs) 2025-08-26T20:32:28.0481932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0482009Z return func(*args, **kwargs) 2025-08-26T20:32:28.0482106Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0482350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0482434Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0482742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0482825Z layer_outputs = layer_module( 2025-08-26T20:32:28.0483080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0483169Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0483437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0483521Z return func(*args, **kwargs) 2025-08-26T20:32:28.0483783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0483884Z return func(*args, **kwargs) 2025-08-26T20:32:28.0484148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0484222Z return func(*args, **kwargs) 2025-08-26T20:32:28.0484529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0484625Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0484899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0484973Z return func(*args, **kwargs) 2025-08-26T20:32:28.0485258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0485334Z return func(*args, **kwargs) 2025-08-26T20:32:28.0485598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0485681Z return func(*args, **kwargs) 2025-08-26T20:32:28.0485978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0486131Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0486430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0486543Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0486547Z 2025-08-26T20:32:28.0486673Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0486899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0486977Z return mod(**inputs) 2025-08-26T20:32:28.0487220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0487309Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0487609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0487689Z outputs = self.layoutlm( 2025-08-26T20:32:28.0487960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0488057Z return func(*args, **kwargs) 2025-08-26T20:32:28.0488325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0488401Z return func(*args, **kwargs) 2025-08-26T20:32:28.0488641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0488732Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0489026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0489114Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0489379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0489453Z return func(*args, **kwargs) 2025-08-26T20:32:28.0489724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0489799Z return func(*args, **kwargs) 2025-08-26T20:32:28.0490072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0490147Z return func(*args, **kwargs) 2025-08-26T20:32:28.0490233Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0490479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0490627Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0490931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0491012Z layer_outputs = layer_module( 2025-08-26T20:32:28.0491264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0491351Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0491616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0491701Z return func(*args, **kwargs) 2025-08-26T20:32:28.0491984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0492065Z return func(*args, **kwargs) 2025-08-26T20:32:28.0492333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0492406Z return func(*args, **kwargs) 2025-08-26T20:32:28.0492714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0492812Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0493111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0493225Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0493560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0493710Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0494005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0494110Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0494114Z 2025-08-26T20:32:28.0494232Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0494468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0494541Z return mod(**inputs) 2025-08-26T20:32:28.0494782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0494893Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0495190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0495278Z outputs = self.layoutlm( 2025-08-26T20:32:28.0495546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0495621Z return func(*args, **kwargs) 2025-08-26T20:32:28.0495891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0495963Z return func(*args, **kwargs) 2025-08-26T20:32:28.0496377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0496469Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0496769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0496851Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0497116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0497199Z return func(*args, **kwargs) 2025-08-26T20:32:28.0497461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0497539Z return func(*args, **kwargs) 2025-08-26T20:32:28.0497892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0497966Z return func(*args, **kwargs) 2025-08-26T20:32:28.0498056Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0499460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0499753Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0500284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0500371Z layer_outputs = layer_module( 2025-08-26T20:32:28.0500907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0501007Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0501312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0501396Z return func(*args, **kwargs) 2025-08-26T20:32:28.0501677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0501766Z return func(*args, **kwargs) 2025-08-26T20:32:28.0502048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0502185Z return func(*args, **kwargs) 2025-08-26T20:32:28.0502497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0502604Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0502905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0502995Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0503356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0503499Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0503827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0504002Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0504249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0504336Z return self.act(input) 2025-08-26T20:32:28.0504343Z 2025-08-26T20:32:28.0504471Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0504732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0504809Z return mod(**inputs) 2025-08-26T20:32:28.0505065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0505153Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0505465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0505558Z outputs = self.layoutlm( 2025-08-26T20:32:28.0505841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0505929Z return func(*args, **kwargs) 2025-08-26T20:32:28.0506204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0506285Z return func(*args, **kwargs) 2025-08-26T20:32:28.0506533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0506617Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0506994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0507081Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0507354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0507439Z return func(*args, **kwargs) 2025-08-26T20:32:28.0507708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0507787Z return func(*args, **kwargs) 2025-08-26T20:32:28.0508075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0508150Z return func(*args, **kwargs) 2025-08-26T20:32:28.0508245Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0508484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0508572Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0508879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0508970Z layer_outputs = layer_module( 2025-08-26T20:32:28.0509226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0509355Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0509636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0509708Z return func(*args, **kwargs) 2025-08-26T20:32:28.0509986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0510058Z return func(*args, **kwargs) 2025-08-26T20:32:28.0510322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0510403Z return func(*args, **kwargs) 2025-08-26T20:32:28.0510697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0510798Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0511121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0511208Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0511546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0511695Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0512001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0512095Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0512101Z 2025-08-26T20:32:28.0512229Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0512467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0512544Z return mod(**inputs) 2025-08-26T20:32:28.0512794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0512883Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0513185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0513271Z outputs = self.layoutlm( 2025-08-26T20:32:28.0513548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0513651Z return func(*args, **kwargs) 2025-08-26T20:32:28.0513925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0514009Z return func(*args, **kwargs) 2025-08-26T20:32:28.0514250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0514341Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0514639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0514723Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0515016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0515092Z return func(*args, **kwargs) 2025-08-26T20:32:28.0515372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0515449Z return func(*args, **kwargs) 2025-08-26T20:32:28.0515719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0515801Z return func(*args, **kwargs) 2025-08-26T20:32:28.0515888Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0516139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0516247Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0516546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0516634Z layer_outputs = layer_module( 2025-08-26T20:32:28.0516883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0516977Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0517257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0517333Z return func(*args, **kwargs) 2025-08-26T20:32:28.0517605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0517688Z return func(*args, **kwargs) 2025-08-26T20:32:28.0517986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0518069Z return func(*args, **kwargs) 2025-08-26T20:32:28.0518377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0518480Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0518752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0518828Z return func(*args, **kwargs) 2025-08-26T20:32:28.0519110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0519186Z return func(*args, **kwargs) 2025-08-26T20:32:28.0519827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0519926Z return func(*args, **kwargs) 2025-08-26T20:32:28.0520248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0520339Z self_outputs = self.self( 2025-08-26T20:32:28.0520619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0520701Z return func(*args, **kwargs) 2025-08-26T20:32:28.0521060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0521149Z return func(*args, **kwargs) 2025-08-26T20:32:28.0521430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0521510Z return func(*args, **kwargs) 2025-08-26T20:32:28.0521820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0521997Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0522002Z 2025-08-26T20:32:28.0522132Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0522383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0522461Z return mod(**inputs) 2025-08-26T20:32:28.0522710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0522797Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0523101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0523180Z outputs = self.layoutlm( 2025-08-26T20:32:28.0523445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0523528Z return func(*args, **kwargs) 2025-08-26T20:32:28.0523814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0523897Z return func(*args, **kwargs) 2025-08-26T20:32:28.0524143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0524227Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0524532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0524619Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0524896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0524969Z return func(*args, **kwargs) 2025-08-26T20:32:28.0525225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0525319Z return func(*args, **kwargs) 2025-08-26T20:32:28.0525576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0525652Z return func(*args, **kwargs) 2025-08-26T20:32:28.0525737Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0525965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0526042Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0526323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0526413Z layer_outputs = layer_module( 2025-08-26T20:32:28.0526639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0526730Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0526982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0527053Z return func(*args, **kwargs) 2025-08-26T20:32:28.0527310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0527380Z return func(*args, **kwargs) 2025-08-26T20:32:28.0527631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0527729Z return func(*args, **kwargs) 2025-08-26T20:32:28.0528013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0528099Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0528343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0528422Z return func(*args, **kwargs) 2025-08-26T20:32:28.0528662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0528741Z return func(*args, **kwargs) 2025-08-26T20:32:28.0529010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0529085Z return func(*args, **kwargs) 2025-08-26T20:32:28.0529404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0529483Z self_outputs = self.self( 2025-08-26T20:32:28.0529792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0529874Z return func(*args, **kwargs) 2025-08-26T20:32:28.0530115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0530221Z return func(*args, **kwargs) 2025-08-26T20:32:28.0530480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0530559Z return func(*args, **kwargs) 2025-08-26T20:32:28.0530836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0530991Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0530998Z 2025-08-26T20:32:28.0531110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0531315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0531393Z return mod(**inputs) 2025-08-26T20:32:28.0531616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0531731Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0532005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0532076Z outputs = self.layoutlm( 2025-08-26T20:32:28.0532335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0532405Z return func(*args, **kwargs) 2025-08-26T20:32:28.0532669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0532746Z return func(*args, **kwargs) 2025-08-26T20:32:28.0532975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0533062Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0533336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0533421Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0533667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0533742Z return func(*args, **kwargs) 2025-08-26T20:32:28.0533987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0534054Z return func(*args, **kwargs) 2025-08-26T20:32:28.0534338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0534413Z return func(*args, **kwargs) 2025-08-26T20:32:28.0534501Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0534722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0534801Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0535091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0535166Z layer_outputs = layer_module( 2025-08-26T20:32:28.0535443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0535528Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0535772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0535849Z return func(*args, **kwargs) 2025-08-26T20:32:28.0536094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0536183Z return func(*args, **kwargs) 2025-08-26T20:32:28.0536415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0536485Z return func(*args, **kwargs) 2025-08-26T20:32:28.0536780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0536863Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0537107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0537175Z return func(*args, **kwargs) 2025-08-26T20:32:28.0537420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0537487Z return func(*args, **kwargs) 2025-08-26T20:32:28.0537722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0537799Z return func(*args, **kwargs) 2025-08-26T20:32:28.0538065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0538167Z self_outputs = self.self( 2025-08-26T20:32:28.0538410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0538477Z return func(*args, **kwargs) 2025-08-26T20:32:28.0538728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0538794Z return func(*args, **kwargs) 2025-08-26T20:32:28.0539040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0539106Z return func(*args, **kwargs) 2025-08-26T20:32:28.0539376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0539532Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0539538Z 2025-08-26T20:32:28.0539621Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0539709Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0539822Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0540035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0540106Z return mod(**inputs) 2025-08-26T20:32:28.0540336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0540451Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0540725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0540804Z outputs = self.layoutlm( 2025-08-26T20:32:28.0541047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0541121Z return func(*args, **kwargs) 2025-08-26T20:32:28.0541372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0541439Z return func(*args, **kwargs) 2025-08-26T20:32:28.0541690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0541777Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0542050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0542130Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0542365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0542437Z return func(*args, **kwargs) 2025-08-26T20:32:28.0542674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0542774Z return func(*args, **kwargs) 2025-08-26T20:32:28.0543015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0543082Z return func(*args, **kwargs) 2025-08-26T20:32:28.0543169Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0543388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0543468Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0543737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0543807Z layer_outputs = layer_module( 2025-08-26T20:32:28.0544033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0544135Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0544381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0544449Z return func(*args, **kwargs) 2025-08-26T20:32:28.0544686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0544763Z return func(*args, **kwargs) 2025-08-26T20:32:28.0544998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0545073Z return func(*args, **kwargs) 2025-08-26T20:32:28.0545340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0545421Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0545663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0545732Z return func(*args, **kwargs) 2025-08-26T20:32:28.0545970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0546035Z return func(*args, **kwargs) 2025-08-26T20:32:28.0546277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0546342Z return func(*args, **kwargs) 2025-08-26T20:32:28.0546635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0546777Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0547042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0547134Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0547140Z 2025-08-26T20:32:28.0547248Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0547454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0547531Z return mod(**inputs) 2025-08-26T20:32:28.0547779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0547868Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0548152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0548227Z outputs = self.layoutlm( 2025-08-26T20:32:28.0548479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0548549Z return func(*args, **kwargs) 2025-08-26T20:32:28.0548797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0548894Z return func(*args, **kwargs) 2025-08-26T20:32:28.0549122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0549201Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0549481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0549576Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0549819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0549895Z return func(*args, **kwargs) 2025-08-26T20:32:28.0550130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0550203Z return func(*args, **kwargs) 2025-08-26T20:32:28.0550450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0550549Z return func(*args, **kwargs) 2025-08-26T20:32:28.0550634Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0550853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0550928Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0551200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0551273Z layer_outputs = layer_module( 2025-08-26T20:32:28.0551500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0551581Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0551827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0551897Z return func(*args, **kwargs) 2025-08-26T20:32:28.0552142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0552219Z return func(*args, **kwargs) 2025-08-26T20:32:28.0552463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0552537Z return func(*args, **kwargs) 2025-08-26T20:32:28.0552840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0552929Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0553197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0553279Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0553593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0553731Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0553995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0554108Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0554113Z 2025-08-26T20:32:28.0554220Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0554428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0554495Z return mod(**inputs) 2025-08-26T20:32:28.0554719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0554795Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0555060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0555166Z outputs = self.layoutlm( 2025-08-26T20:32:28.0555412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0555485Z return func(*args, **kwargs) 2025-08-26T20:32:28.0555729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0555800Z return func(*args, **kwargs) 2025-08-26T20:32:28.0556034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0556112Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0556393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0556473Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0556743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0556821Z return func(*args, **kwargs) 2025-08-26T20:32:28.0557067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0557147Z return func(*args, **kwargs) 2025-08-26T20:32:28.0557389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0557468Z return func(*args, **kwargs) 2025-08-26T20:32:28.0557549Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0557772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0557854Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0558124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0558207Z layer_outputs = layer_module( 2025-08-26T20:32:28.0558436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0558517Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0558769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0558842Z return func(*args, **kwargs) 2025-08-26T20:32:28.0559127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0559204Z return func(*args, **kwargs) 2025-08-26T20:32:28.0559680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0559778Z return func(*args, **kwargs) 2025-08-26T20:32:28.0560068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0560190Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0560492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0560590Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0560972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0561116Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0561423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0561559Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0561785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0561865Z return self.act(input) 2025-08-26T20:32:28.0561895Z 2025-08-26T20:32:28.0562003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0562219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0562289Z return mod(**inputs) 2025-08-26T20:32:28.0562529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0562620Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0562924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0563016Z outputs = self.layoutlm( 2025-08-26T20:32:28.0563287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0563377Z return func(*args, **kwargs) 2025-08-26T20:32:28.0563667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0563747Z return func(*args, **kwargs) 2025-08-26T20:32:28.0563965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0564044Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0564322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0564396Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0564651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0564722Z return func(*args, **kwargs) 2025-08-26T20:32:28.0564991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0565071Z return func(*args, **kwargs) 2025-08-26T20:32:28.0565336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0565421Z return func(*args, **kwargs) 2025-08-26T20:32:28.0565507Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0565743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0565829Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0566138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0566223Z layer_outputs = layer_module( 2025-08-26T20:32:28.0566463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0566564Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0566826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0566897Z return func(*args, **kwargs) 2025-08-26T20:32:28.0567153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0567221Z return func(*args, **kwargs) 2025-08-26T20:32:28.0567514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0567590Z return func(*args, **kwargs) 2025-08-26T20:32:28.0567887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0567989Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0568270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0568362Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0568688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0568858Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0569157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0569267Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0569271Z 2025-08-26T20:32:28.0569389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0569608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0569687Z return mod(**inputs) 2025-08-26T20:32:28.0569920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0570003Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0570323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0570403Z outputs = self.layoutlm( 2025-08-26T20:32:28.0570667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0570742Z return func(*args, **kwargs) 2025-08-26T20:32:28.0570999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0571082Z return func(*args, **kwargs) 2025-08-26T20:32:28.0571319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0571406Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0571689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0571772Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0572042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0572118Z return func(*args, **kwargs) 2025-08-26T20:32:28.0572380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0572453Z return func(*args, **kwargs) 2025-08-26T20:32:28.0572706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0572808Z return func(*args, **kwargs) 2025-08-26T20:32:28.0572895Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0573137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0573219Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0573518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0573599Z layer_outputs = layer_module( 2025-08-26T20:32:28.0573869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0573964Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0574247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0574329Z return func(*args, **kwargs) 2025-08-26T20:32:28.0574593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0574664Z return func(*args, **kwargs) 2025-08-26T20:32:28.0574928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0574999Z return func(*args, **kwargs) 2025-08-26T20:32:28.0575298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0575412Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0575667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0575748Z return func(*args, **kwargs) 2025-08-26T20:32:28.0576005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0576085Z return func(*args, **kwargs) 2025-08-26T20:32:28.0576339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0576416Z return func(*args, **kwargs) 2025-08-26T20:32:28.0576703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0576805Z self_outputs = self.self( 2025-08-26T20:32:28.0577077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0577150Z return func(*args, **kwargs) 2025-08-26T20:32:28.0577422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0577497Z return func(*args, **kwargs) 2025-08-26T20:32:28.0577762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0577842Z return func(*args, **kwargs) 2025-08-26T20:32:28.0578137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0578309Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0578314Z 2025-08-26T20:32:28.0578430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0578648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0578724Z return mod(**inputs) 2025-08-26T20:32:28.0579100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0579202Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0579491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0579599Z outputs = self.layoutlm( 2025-08-26T20:32:28.0579859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0579933Z return func(*args, **kwargs) 2025-08-26T20:32:28.0580194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0580271Z return func(*args, **kwargs) 2025-08-26T20:32:28.0580514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0580593Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0580941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0581034Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0581293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0581374Z return func(*args, **kwargs) 2025-08-26T20:32:28.0581633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0581708Z return func(*args, **kwargs) 2025-08-26T20:32:28.0581972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0582064Z return func(*args, **kwargs) 2025-08-26T20:32:28.0582155Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0582387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0582467Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0582761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0582838Z layer_outputs = layer_module( 2025-08-26T20:32:28.0583087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0583173Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0583445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0583520Z return func(*args, **kwargs) 2025-08-26T20:32:28.0583829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0583912Z return func(*args, **kwargs) 2025-08-26T20:32:28.0584187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0584268Z return func(*args, **kwargs) 2025-08-26T20:32:28.0584579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0584671Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0584931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0585003Z return func(*args, **kwargs) 2025-08-26T20:32:28.0585286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0585359Z return func(*args, **kwargs) 2025-08-26T20:32:28.0585622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0585704Z return func(*args, **kwargs) 2025-08-26T20:32:28.0586016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0586100Z self_outputs = self.self( 2025-08-26T20:32:28.0586400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0586480Z return func(*args, **kwargs) 2025-08-26T20:32:28.0586757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0586830Z return func(*args, **kwargs) 2025-08-26T20:32:28.0587102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0587177Z return func(*args, **kwargs) 2025-08-26T20:32:28.0587495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0587648Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0587672Z 2025-08-26T20:32:28.0587788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0588013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0588087Z return mod(**inputs) 2025-08-26T20:32:28.0588331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0588413Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0588729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0588828Z outputs = self.layoutlm( 2025-08-26T20:32:28.0589107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0589186Z return func(*args, **kwargs) 2025-08-26T20:32:28.0589463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0589544Z return func(*args, **kwargs) 2025-08-26T20:32:28.0589791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0589871Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0590166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0590248Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0590531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0590623Z return func(*args, **kwargs) 2025-08-26T20:32:28.0590903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0590985Z return func(*args, **kwargs) 2025-08-26T20:32:28.0591273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0591353Z return func(*args, **kwargs) 2025-08-26T20:32:28.0591436Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0591686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0591770Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0592075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0592160Z layer_outputs = layer_module( 2025-08-26T20:32:28.0592401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0592491Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0592759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0592831Z return func(*args, **kwargs) 2025-08-26T20:32:28.0593100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0593187Z return func(*args, **kwargs) 2025-08-26T20:32:28.0593464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0593536Z return func(*args, **kwargs) 2025-08-26T20:32:28.0593834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0593934Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0594187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0594267Z return func(*args, **kwargs) 2025-08-26T20:32:28.0594541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0594614Z return func(*args, **kwargs) 2025-08-26T20:32:28.0594877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0594948Z return func(*args, **kwargs) 2025-08-26T20:32:28.0595242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0595317Z self_outputs = self.self( 2025-08-26T20:32:28.0595581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0595676Z return func(*args, **kwargs) 2025-08-26T20:32:28.0595937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0596018Z return func(*args, **kwargs) 2025-08-26T20:32:28.0596898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0597014Z return func(*args, **kwargs) 2025-08-26T20:32:28.0597313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0597515Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0597522Z 2025-08-26T20:32:28.0597621Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0597710Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0597976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0598227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0598303Z return mod(**inputs) 2025-08-26T20:32:28.0598559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0598646Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0598961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0599042Z outputs = self.layoutlm( 2025-08-26T20:32:28.0599378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0599473Z return func(*args, **kwargs) 2025-08-26T20:32:28.0599740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0599828Z return func(*args, **kwargs) 2025-08-26T20:32:28.0600071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0600163Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0600472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0600583Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0600979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0601058Z return func(*args, **kwargs) 2025-08-26T20:32:28.0601334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0601407Z return func(*args, **kwargs) 2025-08-26T20:32:28.0601661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0601743Z return func(*args, **kwargs) 2025-08-26T20:32:28.0601827Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0602069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0602180Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0602471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0602557Z layer_outputs = layer_module( 2025-08-26T20:32:28.0602796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0602892Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0603146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0603229Z return func(*args, **kwargs) 2025-08-26T20:32:28.0603516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0603589Z return func(*args, **kwargs) 2025-08-26T20:32:28.0603853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0603927Z return func(*args, **kwargs) 2025-08-26T20:32:28.0604220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0604312Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0604563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0604641Z return func(*args, **kwargs) 2025-08-26T20:32:28.0604894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0604999Z return func(*args, **kwargs) 2025-08-26T20:32:28.0605254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0605324Z return func(*args, **kwargs) 2025-08-26T20:32:28.0605624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0605767Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0606070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0606161Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0606165Z 2025-08-26T20:32:28.0606287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0606505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0606578Z return mod(**inputs) 2025-08-26T20:32:28.0606823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0606906Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0607200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0607277Z outputs = self.layoutlm( 2025-08-26T20:32:28.0607563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0607645Z return func(*args, **kwargs) 2025-08-26T20:32:28.0607898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0607979Z return func(*args, **kwargs) 2025-08-26T20:32:28.0608211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0608296Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0608590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0608668Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0608950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0609023Z return func(*args, **kwargs) 2025-08-26T20:32:28.0609284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0609356Z return func(*args, **kwargs) 2025-08-26T20:32:28.0609613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0609692Z return func(*args, **kwargs) 2025-08-26T20:32:28.0609775Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0610024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0610128Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0610416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0610502Z layer_outputs = layer_module( 2025-08-26T20:32:28.0610739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0610830Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0611083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0611156Z return func(*args, **kwargs) 2025-08-26T20:32:28.0611418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0611511Z return func(*args, **kwargs) 2025-08-26T20:32:28.0611777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0611850Z return func(*args, **kwargs) 2025-08-26T20:32:28.0612140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0612239Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0612523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0612612Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0612932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0613072Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0613363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0613454Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0613458Z 2025-08-26T20:32:28.0613578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0613796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0613875Z return mod(**inputs) 2025-08-26T20:32:28.0614128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0614210Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0614503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0614578Z outputs = self.layoutlm( 2025-08-26T20:32:28.0614841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0614918Z return func(*args, **kwargs) 2025-08-26T20:32:28.0615181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0615255Z return func(*args, **kwargs) 2025-08-26T20:32:28.0615506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0615596Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0615885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0615972Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0616227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0616300Z return func(*args, **kwargs) 2025-08-26T20:32:28.0616562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0616653Z return func(*args, **kwargs) 2025-08-26T20:32:28.0616922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0616993Z return func(*args, **kwargs) 2025-08-26T20:32:28.0617079Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0617322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0617403Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0617702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0617777Z layer_outputs = layer_module( 2025-08-26T20:32:28.0618021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0618139Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0618397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0618476Z return func(*args, **kwargs) 2025-08-26T20:32:28.0618730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0618808Z return func(*args, **kwargs) 2025-08-26T20:32:28.0619064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0619134Z return func(*args, **kwargs) 2025-08-26T20:32:28.0619428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0619518Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0619806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0619890Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0620212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0620353Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0620640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0620787Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0621019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0621104Z return self.act(input) 2025-08-26T20:32:28.0621108Z 2025-08-26T20:32:28.0621222Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0621435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0621519Z return mod(**inputs) 2025-08-26T20:32:28.0621752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0621842Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0622148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0622226Z outputs = self.layoutlm( 2025-08-26T20:32:28.0622493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0622565Z return func(*args, **kwargs) 2025-08-26T20:32:28.0622829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0622901Z return func(*args, **kwargs) 2025-08-26T20:32:28.0623139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0623240Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0623513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0623597Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0623839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0623914Z return func(*args, **kwargs) 2025-08-26T20:32:28.0624157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0624227Z return func(*args, **kwargs) 2025-08-26T20:32:28.0624474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0624542Z return func(*args, **kwargs) 2025-08-26T20:32:28.0624649Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0624871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0624946Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0625223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0625295Z layer_outputs = layer_module( 2025-08-26T20:32:28.0625525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0625605Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0625848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0625923Z return func(*args, **kwargs) 2025-08-26T20:32:28.0626163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0626242Z return func(*args, **kwargs) 2025-08-26T20:32:28.0626484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0626552Z return func(*args, **kwargs) 2025-08-26T20:32:28.0626831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0626916Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0627201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0627281Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0627591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0627729Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0628001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0628091Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0628096Z 2025-08-26T20:32:28.0628221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0628432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0628500Z return mod(**inputs) 2025-08-26T20:32:28.0628720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0628805Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0629075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0629153Z outputs = self.layoutlm( 2025-08-26T20:32:28.0629404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0629499Z return func(*args, **kwargs) 2025-08-26T20:32:28.0629748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0629819Z return func(*args, **kwargs) 2025-08-26T20:32:28.0630054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0630129Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0630422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0630503Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0630773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0630882Z return func(*args, **kwargs) 2025-08-26T20:32:28.0631139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0631218Z return func(*args, **kwargs) 2025-08-26T20:32:28.0631473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0631546Z return func(*args, **kwargs) 2025-08-26T20:32:28.0631638Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0631880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0631966Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0632262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0632336Z layer_outputs = layer_module( 2025-08-26T20:32:28.0632565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0632649Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0632912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0632982Z return func(*args, **kwargs) 2025-08-26T20:32:28.0633232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0633301Z return func(*args, **kwargs) 2025-08-26T20:32:28.0633620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0633702Z return func(*args, **kwargs) 2025-08-26T20:32:28.0633992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0634088Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0634345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0634420Z return func(*args, **kwargs) 2025-08-26T20:32:28.0634693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0634786Z return func(*args, **kwargs) 2025-08-26T20:32:28.0635054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0635128Z return func(*args, **kwargs) 2025-08-26T20:32:28.0635411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0635497Z self_outputs = self.self( 2025-08-26T20:32:28.0635751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0635832Z return func(*args, **kwargs) 2025-08-26T20:32:28.0636106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0636185Z return func(*args, **kwargs) 2025-08-26T20:32:28.0636440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0636512Z return func(*args, **kwargs) 2025-08-26T20:32:28.0636805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0636967Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0636972Z 2025-08-26T20:32:28.0637090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0637305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0637378Z return mod(**inputs) 2025-08-26T20:32:28.0637646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0637730Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0638033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0638113Z outputs = self.layoutlm( 2025-08-26T20:32:28.0638376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0638462Z return func(*args, **kwargs) 2025-08-26T20:32:28.0638725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0638804Z return func(*args, **kwargs) 2025-08-26T20:32:28.0639045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0639136Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0639620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0639714Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0639997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0640073Z return func(*args, **kwargs) 2025-08-26T20:32:28.0640372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0640449Z return func(*args, **kwargs) 2025-08-26T20:32:28.0640714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0640799Z return func(*args, **kwargs) 2025-08-26T20:32:28.0640885Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0641131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0641217Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0641511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0641612Z layer_outputs = layer_module( 2025-08-26T20:32:28.0641841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0641941Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0642178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0642253Z return func(*args, **kwargs) 2025-08-26T20:32:28.0642489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0642556Z return func(*args, **kwargs) 2025-08-26T20:32:28.0642804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0642893Z return func(*args, **kwargs) 2025-08-26T20:32:28.0643179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0643269Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0643527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0643609Z return func(*args, **kwargs) 2025-08-26T20:32:28.0643866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0643946Z return func(*args, **kwargs) 2025-08-26T20:32:28.0644205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0644294Z return func(*args, **kwargs) 2025-08-26T20:32:28.0644587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0644660Z self_outputs = self.self( 2025-08-26T20:32:28.0644904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0644971Z return func(*args, **kwargs) 2025-08-26T20:32:28.0645208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0645283Z return func(*args, **kwargs) 2025-08-26T20:32:28.0645519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0645598Z return func(*args, **kwargs) 2025-08-26T20:32:28.0645900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0646064Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0646068Z 2025-08-26T20:32:28.0646182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0646398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0646479Z return mod(**inputs) 2025-08-26T20:32:28.0646719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0646828Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0647126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0647201Z outputs = self.layoutlm( 2025-08-26T20:32:28.0647474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0647548Z return func(*args, **kwargs) 2025-08-26T20:32:28.0647810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0647882Z return func(*args, **kwargs) 2025-08-26T20:32:28.0648140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0648226Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0648524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0648615Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0648906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0648985Z return func(*args, **kwargs) 2025-08-26T20:32:28.0649250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0649343Z return func(*args, **kwargs) 2025-08-26T20:32:28.0649603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0649675Z return func(*args, **kwargs) 2025-08-26T20:32:28.0649768Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0650002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0650081Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0650388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0650467Z layer_outputs = layer_module( 2025-08-26T20:32:28.0650722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0650808Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0651113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0651194Z return func(*args, **kwargs) 2025-08-26T20:32:28.0651474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0651556Z return func(*args, **kwargs) 2025-08-26T20:32:28.0651833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0651918Z return func(*args, **kwargs) 2025-08-26T20:32:28.0652233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0652335Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0652602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0652676Z return func(*args, **kwargs) 2025-08-26T20:32:28.0652940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0653012Z return func(*args, **kwargs) 2025-08-26T20:32:28.0653283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0653362Z return func(*args, **kwargs) 2025-08-26T20:32:28.0653688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0653774Z self_outputs = self.self( 2025-08-26T20:32:28.0654038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0654108Z return func(*args, **kwargs) 2025-08-26T20:32:28.0654380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0654454Z return func(*args, **kwargs) 2025-08-26T20:32:28.0654717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0654789Z return func(*args, **kwargs) 2025-08-26T20:32:28.0655106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0655268Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0655274Z 2025-08-26T20:32:28.0655364Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0655455Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0655569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0655799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0655871Z return mod(**inputs) 2025-08-26T20:32:28.0656108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0656226Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0656520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0656601Z outputs = self.layoutlm( 2025-08-26T20:32:28.0656854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0656928Z return func(*args, **kwargs) 2025-08-26T20:32:28.0657188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0657261Z return func(*args, **kwargs) 2025-08-26T20:32:28.0657496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0657595Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0657888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0657974Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0658234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0658316Z return func(*args, **kwargs) 2025-08-26T20:32:28.0658574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0658653Z return func(*args, **kwargs) 2025-08-26T20:32:28.0658912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0658982Z return func(*args, **kwargs) 2025-08-26T20:32:28.0659071Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0659309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0659398Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0659690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0659768Z layer_outputs = layer_module( 2025-08-26T20:32:28.0660017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0660101Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0660388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0660463Z return func(*args, **kwargs) 2025-08-26T20:32:28.0660718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0660799Z return func(*args, **kwargs) 2025-08-26T20:32:28.0661057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0661139Z return func(*args, **kwargs) 2025-08-26T20:32:28.0661451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0661553Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0661824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0661899Z return func(*args, **kwargs) 2025-08-26T20:32:28.0662162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0662233Z return func(*args, **kwargs) 2025-08-26T20:32:28.0662495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0662570Z return func(*args, **kwargs) 2025-08-26T20:32:28.0662873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0663021Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0663306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0663405Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0663409Z 2025-08-26T20:32:28.0663523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0663746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0663817Z return mod(**inputs) 2025-08-26T20:32:28.0664052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0664161Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0664456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0664539Z outputs = self.layoutlm( 2025-08-26T20:32:28.0664809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0664883Z return func(*args, **kwargs) 2025-08-26T20:32:28.0665157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0665231Z return func(*args, **kwargs) 2025-08-26T20:32:28.0665485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0665567Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0665886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0665979Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0666238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0666318Z return func(*args, **kwargs) 2025-08-26T20:32:28.0666582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0666653Z return func(*args, **kwargs) 2025-08-26T20:32:28.0666936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0667010Z return func(*args, **kwargs) 2025-08-26T20:32:28.0667102Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0667335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0667417Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0667725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0667807Z layer_outputs = layer_module( 2025-08-26T20:32:28.0668059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0668166Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0668441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0668516Z return func(*args, **kwargs) 2025-08-26T20:32:28.0668777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0668860Z return func(*args, **kwargs) 2025-08-26T20:32:28.0669126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0669209Z return func(*args, **kwargs) 2025-08-26T20:32:28.0669522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0669617Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0669912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0669995Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0670336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0670473Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0670772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0670862Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0670883Z 2025-08-26T20:32:28.0671001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0671230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0671302Z return mod(**inputs) 2025-08-26T20:32:28.0671551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0671632Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0671928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0672013Z outputs = self.layoutlm( 2025-08-26T20:32:28.0672278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0672360Z return func(*args, **kwargs) 2025-08-26T20:32:28.0672623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0672702Z return func(*args, **kwargs) 2025-08-26T20:32:28.0672954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0673036Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0673338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0673421Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0673711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0673787Z return func(*args, **kwargs) 2025-08-26T20:32:28.0674046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0674128Z return func(*args, **kwargs) 2025-08-26T20:32:28.0674389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0674472Z return func(*args, **kwargs) 2025-08-26T20:32:28.0674557Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0674796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0674902Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0675202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0675291Z layer_outputs = layer_module( 2025-08-26T20:32:28.0675537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0675624Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0675894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0675971Z return func(*args, **kwargs) 2025-08-26T20:32:28.0676264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0676338Z return func(*args, **kwargs) 2025-08-26T20:32:28.0676605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0676687Z return func(*args, **kwargs) 2025-08-26T20:32:28.0676989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0677091Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0677381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0677473Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0677807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0677964Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0678266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0678394Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0678636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0678717Z return self.act(input) 2025-08-26T20:32:28.0678721Z 2025-08-26T20:32:28.0678843Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0679063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0679135Z return mod(**inputs) 2025-08-26T20:32:28.0679552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0679648Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0679953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0680033Z outputs = self.layoutlm( 2025-08-26T20:32:28.0680302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0680388Z return func(*args, **kwargs) 2025-08-26T20:32:28.0680675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0680762Z return func(*args, **kwargs) 2025-08-26T20:32:28.0681002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0681086Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0681387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0681474Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0681747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0681821Z return func(*args, **kwargs) 2025-08-26T20:32:28.0682110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0682194Z return func(*args, **kwargs) 2025-08-26T20:32:28.0682458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0682539Z return func(*args, **kwargs) 2025-08-26T20:32:28.0682627Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0682863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0682954Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0683274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0683362Z layer_outputs = layer_module( 2025-08-26T20:32:28.0683607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0683701Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0683967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0684040Z return func(*args, **kwargs) 2025-08-26T20:32:28.0684311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0684392Z return func(*args, **kwargs) 2025-08-26T20:32:28.0684655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0684745Z return func(*args, **kwargs) 2025-08-26T20:32:28.0685033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0685132Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0685411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0685500Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0685821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0685974Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0686260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0686351Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0686356Z 2025-08-26T20:32:28.0686475Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0686689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0686765Z return mod(**inputs) 2025-08-26T20:32:28.0686998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0687077Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0687385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0687463Z outputs = self.layoutlm( 2025-08-26T20:32:28.0687726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0687798Z return func(*args, **kwargs) 2025-08-26T20:32:28.0688056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0688135Z return func(*args, **kwargs) 2025-08-26T20:32:28.0688364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0688467Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0688755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0688841Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0689104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0689177Z return func(*args, **kwargs) 2025-08-26T20:32:28.0689448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0689524Z return func(*args, **kwargs) 2025-08-26T20:32:28.0689793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0689884Z return func(*args, **kwargs) 2025-08-26T20:32:28.0689971Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0690219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0690300Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0690604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0690694Z layer_outputs = layer_module( 2025-08-26T20:32:28.0690931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0691022Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0691277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0691377Z return func(*args, **kwargs) 2025-08-26T20:32:28.0691636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0691712Z return func(*args, **kwargs) 2025-08-26T20:32:28.0691975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0692048Z return func(*args, **kwargs) 2025-08-26T20:32:28.0692354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0692447Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0692720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0692793Z return func(*args, **kwargs) 2025-08-26T20:32:28.0693059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0693142Z return func(*args, **kwargs) 2025-08-26T20:32:28.0693407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0693491Z return func(*args, **kwargs) 2025-08-26T20:32:28.0693787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0693886Z self_outputs = self.self( 2025-08-26T20:32:28.0694165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0694235Z return func(*args, **kwargs) 2025-08-26T20:32:28.0694495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0694568Z return func(*args, **kwargs) 2025-08-26T20:32:28.0694823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0694902Z return func(*args, **kwargs) 2025-08-26T20:32:28.0695205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0695377Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0695381Z 2025-08-26T20:32:28.0695498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0695725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0695797Z return mod(**inputs) 2025-08-26T20:32:28.0696035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0696126Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0696773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0697003Z outputs = self.layoutlm( 2025-08-26T20:32:28.0697312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0697390Z return func(*args, **kwargs) 2025-08-26T20:32:28.0697658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0697734Z return func(*args, **kwargs) 2025-08-26T20:32:28.0697980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0698062Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0698361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0698481Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0698747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0698829Z return func(*args, **kwargs) 2025-08-26T20:32:28.0699091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0699173Z return func(*args, **kwargs) 2025-08-26T20:32:28.0699432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0699508Z return func(*args, **kwargs) 2025-08-26T20:32:28.0699601Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0699839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0699926Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0700222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0700303Z layer_outputs = layer_module( 2025-08-26T20:32:28.0700554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0700641Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0700912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0700984Z return func(*args, **kwargs) 2025-08-26T20:32:28.0701269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0701354Z return func(*args, **kwargs) 2025-08-26T20:32:28.0701618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0701701Z return func(*args, **kwargs) 2025-08-26T20:32:28.0702001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0702102Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0702364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0702465Z return func(*args, **kwargs) 2025-08-26T20:32:28.0702738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0702813Z return func(*args, **kwargs) 2025-08-26T20:32:28.0703080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0703152Z return func(*args, **kwargs) 2025-08-26T20:32:28.0703443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0703531Z self_outputs = self.self( 2025-08-26T20:32:28.0703842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0703923Z return func(*args, **kwargs) 2025-08-26T20:32:28.0704183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0704256Z return func(*args, **kwargs) 2025-08-26T20:32:28.0704524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0704598Z return func(*args, **kwargs) 2025-08-26T20:32:28.0704898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0705053Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0705060Z 2025-08-26T20:32:28.0705199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0705419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0705491Z return mod(**inputs) 2025-08-26T20:32:28.0705741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0705824Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0706127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0706204Z outputs = self.layoutlm( 2025-08-26T20:32:28.0706466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0706547Z return func(*args, **kwargs) 2025-08-26T20:32:28.0706807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0706890Z return func(*args, **kwargs) 2025-08-26T20:32:28.0707132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0707213Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0707520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0707600Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0707897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0707973Z return func(*args, **kwargs) 2025-08-26T20:32:28.0708243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0708317Z return func(*args, **kwargs) 2025-08-26T20:32:28.0708577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0708662Z return func(*args, **kwargs) 2025-08-26T20:32:28.0708749Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0708992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0709089Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0709388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0709474Z layer_outputs = layer_module( 2025-08-26T20:32:28.0709720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0709814Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0710075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0710151Z return func(*args, **kwargs) 2025-08-26T20:32:28.0710421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0710516Z return func(*args, **kwargs) 2025-08-26T20:32:28.0710787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0710864Z return func(*args, **kwargs) 2025-08-26T20:32:28.0711159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0711260Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0711520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0711602Z return func(*args, **kwargs) 2025-08-26T20:32:28.0711862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0711964Z return func(*args, **kwargs) 2025-08-26T20:32:28.0712231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0712306Z return func(*args, **kwargs) 2025-08-26T20:32:28.0712608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0712688Z self_outputs = self.self( 2025-08-26T20:32:28.0712958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0713031Z return func(*args, **kwargs) 2025-08-26T20:32:28.0713292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0713376Z return func(*args, **kwargs) 2025-08-26T20:32:28.0713634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0713718Z return func(*args, **kwargs) 2025-08-26T20:32:28.0714011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0714176Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0714186Z 2025-08-26T20:32:28.0714275Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0714363Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0714502Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0714726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0714807Z return mod(**inputs) 2025-08-26T20:32:28.0715047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0715129Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0715437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0715517Z outputs = self.layoutlm( 2025-08-26T20:32:28.0715789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0715881Z return func(*args, **kwargs) 2025-08-26T20:32:28.0716148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0716232Z return func(*args, **kwargs) 2025-08-26T20:32:28.0716471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0716561Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0716856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0716942Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0717237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0717313Z return func(*args, **kwargs) 2025-08-26T20:32:28.0717586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0717660Z return func(*args, **kwargs) 2025-08-26T20:32:28.0717932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0718007Z return func(*args, **kwargs) 2025-08-26T20:32:28.0718091Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0718335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0718415Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0718719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0718821Z layer_outputs = layer_module( 2025-08-26T20:32:28.0719065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0719160Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0719551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0719639Z return func(*args, **kwargs) 2025-08-26T20:32:28.0719905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0719978Z return func(*args, **kwargs) 2025-08-26T20:32:28.0720251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0720325Z return func(*args, **kwargs) 2025-08-26T20:32:28.0720633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0720728Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0720989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0721070Z return func(*args, **kwargs) 2025-08-26T20:32:28.0721331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0721442Z return func(*args, **kwargs) 2025-08-26T20:32:28.0721709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0721789Z return func(*args, **kwargs) 2025-08-26T20:32:28.0722083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0722228Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0722535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0722626Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0722630Z 2025-08-26T20:32:28.0722774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0722996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0723075Z return mod(**inputs) 2025-08-26T20:32:28.0723325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0723410Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0723724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0723806Z outputs = self.layoutlm( 2025-08-26T20:32:28.0724080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0724159Z return func(*args, **kwargs) 2025-08-26T20:32:28.0724418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0724499Z return func(*args, **kwargs) 2025-08-26T20:32:28.0724732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0724822Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0725110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0725189Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0725455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0725548Z return func(*args, **kwargs) 2025-08-26T20:32:28.0725813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0725887Z return func(*args, **kwargs) 2025-08-26T20:32:28.0726146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0726226Z return func(*args, **kwargs) 2025-08-26T20:32:28.0726308Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0726552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0726630Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0726916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0727000Z layer_outputs = layer_module( 2025-08-26T20:32:28.0727238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0727332Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0727584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0727665Z return func(*args, **kwargs) 2025-08-26T20:32:28.0727916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0728004Z return func(*args, **kwargs) 2025-08-26T20:32:28.0728270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0728345Z return func(*args, **kwargs) 2025-08-26T20:32:28.0728646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0728741Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0729035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0729125Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0729459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0729601Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0729887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0729975Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0729986Z 2025-08-26T20:32:28.0730097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0730314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0730395Z return mod(**inputs) 2025-08-26T20:32:28.0730646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0730734Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0731021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0731097Z outputs = self.layoutlm( 2025-08-26T20:32:28.0731363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0731436Z return func(*args, **kwargs) 2025-08-26T20:32:28.0731699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0731772Z return func(*args, **kwargs) 2025-08-26T20:32:28.0732003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0732124Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0732417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0732507Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0732774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0732849Z return func(*args, **kwargs) 2025-08-26T20:32:28.0733120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0733195Z return func(*args, **kwargs) 2025-08-26T20:32:28.0733462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0733536Z return func(*args, **kwargs) 2025-08-26T20:32:28.0733630Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0733870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0733951Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0734253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0734334Z layer_outputs = layer_module( 2025-08-26T20:32:28.0734581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0734687Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0734952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0735034Z return func(*args, **kwargs) 2025-08-26T20:32:28.0735298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0735382Z return func(*args, **kwargs) 2025-08-26T20:32:28.0735645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0735718Z return func(*args, **kwargs) 2025-08-26T20:32:28.0736055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0736148Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0736435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0736518Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0736842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0736975Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0737267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0738386Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0738631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0738719Z return self.act(input) 2025-08-26T20:32:28.0738723Z 2025-08-26T20:32:28.0738838Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0739071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0739151Z return mod(**inputs) 2025-08-26T20:32:28.0739394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0739484Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0739781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0739895Z outputs = self.layoutlm( 2025-08-26T20:32:28.0740176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0740252Z return func(*args, **kwargs) 2025-08-26T20:32:28.0740558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0740633Z return func(*args, **kwargs) 2025-08-26T20:32:28.0740887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0740971Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0741277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0741369Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0741645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0741733Z return func(*args, **kwargs) 2025-08-26T20:32:28.0742005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0742081Z return func(*args, **kwargs) 2025-08-26T20:32:28.0742368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0742442Z return func(*args, **kwargs) 2025-08-26T20:32:28.0742553Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0742853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0742959Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0743442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0743533Z layer_outputs = layer_module( 2025-08-26T20:32:28.0743796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0743885Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0744196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0744275Z return func(*args, **kwargs) 2025-08-26T20:32:28.0744551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0744635Z return func(*args, **kwargs) 2025-08-26T20:32:28.0744908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0744990Z return func(*args, **kwargs) 2025-08-26T20:32:28.0745300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0745416Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0745717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0745800Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0746149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0746303Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0746610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0746709Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0746713Z 2025-08-26T20:32:28.0746826Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0747057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0747154Z return mod(**inputs) 2025-08-26T20:32:28.0747406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0747490Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0747798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0747885Z outputs = self.layoutlm( 2025-08-26T20:32:28.0748162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0748246Z return func(*args, **kwargs) 2025-08-26T20:32:28.0748518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0748592Z return func(*args, **kwargs) 2025-08-26T20:32:28.0748846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0748930Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0749241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0749324Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0749602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0749685Z return func(*args, **kwargs) 2025-08-26T20:32:28.0749978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0750062Z return func(*args, **kwargs) 2025-08-26T20:32:28.0750335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0750415Z return func(*args, **kwargs) 2025-08-26T20:32:28.0750499Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0750741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0750832Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0751161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0751249Z layer_outputs = layer_module( 2025-08-26T20:32:28.0751495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0751582Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0751857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0751931Z return func(*args, **kwargs) 2025-08-26T20:32:28.0752205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0752306Z return func(*args, **kwargs) 2025-08-26T20:32:28.0752582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0752662Z return func(*args, **kwargs) 2025-08-26T20:32:28.0752973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0753073Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0753347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0753427Z return func(*args, **kwargs) 2025-08-26T20:32:28.0753703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0753776Z return func(*args, **kwargs) 2025-08-26T20:32:28.0754069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0754145Z return func(*args, **kwargs) 2025-08-26T20:32:28.0754453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0754533Z self_outputs = self.self( 2025-08-26T20:32:28.0754805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0754885Z return func(*args, **kwargs) 2025-08-26T20:32:28.0755152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0755233Z return func(*args, **kwargs) 2025-08-26T20:32:28.0755501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0755577Z return func(*args, **kwargs) 2025-08-26T20:32:28.0755890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-26T20:32:28.0756057Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0756061Z 2025-08-26T20:32:28.0756186Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0756418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0756499Z return mod(**inputs) 2025-08-26T20:32:28.0756758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0756843Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0757150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0757229Z outputs = self.layoutlm( 2025-08-26T20:32:28.0757515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0757592Z return func(*args, **kwargs) 2025-08-26T20:32:28.0757864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0757963Z return func(*args, **kwargs) 2025-08-26T20:32:28.0758208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0758300Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0758597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0758678Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0758955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0759031Z return func(*args, **kwargs) 2025-08-26T20:32:28.0759418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0759499Z return func(*args, **kwargs) 2025-08-26T20:32:28.0759780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0759855Z return func(*args, **kwargs) 2025-08-26T20:32:28.0759941Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0760193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0760274Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0760576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0760655Z layer_outputs = layer_module( 2025-08-26T20:32:28.0760898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0761021Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0761302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0761389Z return func(*args, **kwargs) 2025-08-26T20:32:28.0761673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0761754Z return func(*args, **kwargs) 2025-08-26T20:32:28.0762045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0762123Z return func(*args, **kwargs) 2025-08-26T20:32:28.0762438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0762541Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0762814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0762902Z return func(*args, **kwargs) 2025-08-26T20:32:28.0763182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0763270Z return func(*args, **kwargs) 2025-08-26T20:32:28.0763556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0763660Z return func(*args, **kwargs) 2025-08-26T20:32:28.0763963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0764041Z self_outputs = self.self( 2025-08-26T20:32:28.0764314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0764390Z return func(*args, **kwargs) 2025-08-26T20:32:28.0764674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0764749Z return func(*args, **kwargs) 2025-08-26T20:32:28.0765040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0765123Z return func(*args, **kwargs) 2025-08-26T20:32:28.0765421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-26T20:32:28.0765581Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0765586Z 2025-08-26T20:32:28.0765700Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0765927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0766000Z return mod(**inputs) 2025-08-26T20:32:28.0766243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0766354Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0766647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0766734Z outputs = self.layoutlm( 2025-08-26T20:32:28.0767014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0767090Z return func(*args, **kwargs) 2025-08-26T20:32:28.0767362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0767433Z return func(*args, **kwargs) 2025-08-26T20:32:28.0767672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0767774Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0768070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0768156Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0768428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0768509Z return func(*args, **kwargs) 2025-08-26T20:32:28.0768778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0768852Z return func(*args, **kwargs) 2025-08-26T20:32:28.0769128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0769200Z return func(*args, **kwargs) 2025-08-26T20:32:28.0769292Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0769535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0769626Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0769943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0770021Z layer_outputs = layer_module( 2025-08-26T20:32:28.0770266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0770349Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0770636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0770711Z return func(*args, **kwargs) 2025-08-26T20:32:28.0770963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0771041Z return func(*args, **kwargs) 2025-08-26T20:32:28.0771297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0771378Z return func(*args, **kwargs) 2025-08-26T20:32:28.0771663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0771771Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0772034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0772107Z return func(*args, **kwargs) 2025-08-26T20:32:28.0772363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0772435Z return func(*args, **kwargs) 2025-08-26T20:32:28.0772695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0772768Z return func(*args, **kwargs) 2025-08-26T20:32:28.0773074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-26T20:32:28.0773158Z self_outputs = self.self( 2025-08-26T20:32:28.0773414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0773492Z return func(*args, **kwargs) 2025-08-26T20:32:28.0773747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0773818Z return func(*args, **kwargs) 2025-08-26T20:32:28.0774082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0774153Z return func(*args, **kwargs) 2025-08-26T20:32:28.0774446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-26T20:32:28.0774646Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-26T20:32:28.0774651Z 2025-08-26T20:32:28.0774738Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0774830Z cudagraph partition due to non gpu ops 2025-08-26T20:32:28.0774943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0775167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0775239Z return mod(**inputs) 2025-08-26T20:32:28.0775477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0775566Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0775849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0775934Z outputs = self.layoutlm( 2025-08-26T20:32:28.0776192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0776273Z return func(*args, **kwargs) 2025-08-26T20:32:28.0776526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0776599Z return func(*args, **kwargs) 2025-08-26T20:32:28.0776838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0776935Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0777230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0777312Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0777572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0777660Z return func(*args, **kwargs) 2025-08-26T20:32:28.0777923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0778008Z return func(*args, **kwargs) 2025-08-26T20:32:28.0778288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0778361Z return func(*args, **kwargs) 2025-08-26T20:32:28.0778452Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0778687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0778772Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0779061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0779143Z layer_outputs = layer_module( 2025-08-26T20:32:28.0779383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0779489Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0779752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0779827Z return func(*args, **kwargs) 2025-08-26T20:32:28.0780091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0780162Z return func(*args, **kwargs) 2025-08-26T20:32:28.0780415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0780495Z return func(*args, **kwargs) 2025-08-26T20:32:28.0780782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-26T20:32:28.0780876Z self_attention_outputs = self.attention( 2025-08-26T20:32:28.0781151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0781226Z return func(*args, **kwargs) 2025-08-26T20:32:28.0781489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0781562Z return func(*args, **kwargs) 2025-08-26T20:32:28.0781824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0781897Z return func(*args, **kwargs) 2025-08-26T20:32:28.0782184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-26T20:32:28.0782333Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:32:28.0782623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-26T20:32:28.0782725Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0782729Z 2025-08-26T20:32:28.0782841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0783063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0783135Z return mod(**inputs) 2025-08-26T20:32:28.0783368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0783479Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0783751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0783829Z outputs = self.layoutlm( 2025-08-26T20:32:28.0784077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0784152Z return func(*args, **kwargs) 2025-08-26T20:32:28.0784415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0784487Z return func(*args, **kwargs) 2025-08-26T20:32:28.0784751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0784833Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0785125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0785205Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0785463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0785542Z return func(*args, **kwargs) 2025-08-26T20:32:28.0785798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0785899Z return func(*args, **kwargs) 2025-08-26T20:32:28.0786162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0786236Z return func(*args, **kwargs) 2025-08-26T20:32:28.0786327Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0786570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0786652Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0786931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0787006Z layer_outputs = layer_module( 2025-08-26T20:32:28.0787254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0787339Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0787619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0787702Z return func(*args, **kwargs) 2025-08-26T20:32:28.0787940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0788016Z return func(*args, **kwargs) 2025-08-26T20:32:28.0788254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0788333Z return func(*args, **kwargs) 2025-08-26T20:32:28.0788613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0788711Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0788988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0789071Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0789400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0789530Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0789821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-26T20:32:28.0789910Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0789914Z 2025-08-26T20:32:28.0790042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0790263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0790335Z return mod(**inputs) 2025-08-26T20:32:28.0790577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0790660Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0790957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0791034Z outputs = self.layoutlm( 2025-08-26T20:32:28.0791308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0791394Z return func(*args, **kwargs) 2025-08-26T20:32:28.0791653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0791735Z return func(*args, **kwargs) 2025-08-26T20:32:28.0791967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0792046Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0792340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0792443Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0792707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0792780Z return func(*args, **kwargs) 2025-08-26T20:32:28.0793036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0793115Z return func(*args, **kwargs) 2025-08-26T20:32:28.0793375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0793454Z return func(*args, **kwargs) 2025-08-26T20:32:28.0793536Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0793768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0793854Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0794162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0794247Z layer_outputs = layer_module( 2025-08-26T20:32:28.0794483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0794577Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0794833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0794905Z return func(*args, **kwargs) 2025-08-26T20:32:28.0795164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0795237Z return func(*args, **kwargs) 2025-08-26T20:32:28.0795504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0795580Z return func(*args, **kwargs) 2025-08-26T20:32:28.0795875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0795978Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0796520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0796620Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0797012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-26T20:32:28.0797157Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:32:28.0797453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-26T20:32:28.0797578Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:32:28.0797822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:32:28.0797904Z return self.act(input) 2025-08-26T20:32:28.0797908Z 2025-08-26T20:32:28.0798029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0798281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0798356Z return mod(**inputs) 2025-08-26T20:32:28.0798609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0798693Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0798998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0799076Z outputs = self.layoutlm( 2025-08-26T20:32:28.0799391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0799513Z return func(*args, **kwargs) 2025-08-26T20:32:28.0799780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0799863Z return func(*args, **kwargs) 2025-08-26T20:32:28.0800106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0800195Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0800501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-26T20:32:28.0800579Z encoder_outputs = self.encoder( 2025-08-26T20:32:28.0800842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0800916Z return func(*args, **kwargs) 2025-08-26T20:32:28.0801176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0801283Z return func(*args, **kwargs) 2025-08-26T20:32:28.0801538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0801619Z return func(*args, **kwargs) 2025-08-26T20:32:28.0801703Z [Previous line repeated 1 more time] 2025-08-26T20:32:28.0801945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0802025Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0802310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-26T20:32:28.0802393Z layer_outputs = layer_module( 2025-08-26T20:32:28.0802629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:32:28.0802722Z return super().__call__(*args, **kwargs) 2025-08-26T20:32:28.0802978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0803050Z return func(*args, **kwargs) 2025-08-26T20:32:28.0803340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0803411Z return func(*args, **kwargs) 2025-08-26T20:32:28.0803707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0803780Z return func(*args, **kwargs) 2025-08-26T20:32:28.0804103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-26T20:32:28.0804196Z layer_output = apply_chunking_to_forward( 2025-08-26T20:32:28.0804472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:32:28.0804564Z return forward_fn(*input_tensors) 2025-08-26T20:32:28.0804898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-26T20:32:28.0805072Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:32:28.0805384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-26T20:32:28.0805474Z hidden_states = self.dense(hidden_states) 2025-08-26T20:32:28.0805486Z 2025-08-26T20:32:28.0805597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0805813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0805892Z return mod(**inputs) 2025-08-26T20:32:28.0806138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0806248Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0806558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0806633Z outputs = self.layoutlm( 2025-08-26T20:32:28.0806914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0806988Z return func(*args, **kwargs) 2025-08-26T20:32:28.0807273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0807345Z return func(*args, **kwargs) 2025-08-26T20:32:28.0807589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0807678Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0807963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-08-26T20:32:28.0808085Z pooled_output = self.pooler(sequence_output) 2025-08-26T20:32:28.0808352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 430, in forward 2025-08-26T20:32:28.0808449Z pooled_output = self.dense(first_token_tensor) 2025-08-26T20:32:28.0808460Z 2025-08-26T20:32:28.0808572Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0808786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0808865Z return mod(**inputs) 2025-08-26T20:32:28.0809146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0809227Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0809494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-26T20:32:28.0809568Z outputs = self.layoutlm( 2025-08-26T20:32:28.0809816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0809885Z return func(*args, **kwargs) 2025-08-26T20:32:28.0810137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:32:28.0810205Z return func(*args, **kwargs) 2025-08-26T20:32:28.0810452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0810542Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0810838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-08-26T20:32:28.0810950Z pooled_output = self.pooler(sequence_output) 2025-08-26T20:32:28.0811250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 431, in forward 2025-08-26T20:32:28.0811360Z pooled_output = self.activation(pooled_output) 2025-08-26T20:32:28.0811371Z 2025-08-26T20:32:28.0811484Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0811746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0811827Z return mod(**inputs) 2025-08-26T20:32:28.0812064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0812151Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0812439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 891, in forward 2025-08-26T20:32:28.0812527Z logits = self.classifier(pooled_output) 2025-08-26T20:32:28.0812531Z 2025-08-26T20:32:28.0812644Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0812867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0812940Z return mod(**inputs) 2025-08-26T20:32:28.0813161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0813238Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0813520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-26T20:32:28.0813662Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-26T20:32:28.0813666Z 2025-08-26T20:32:28.0813777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0813977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0814050Z return mod(**inputs) 2025-08-26T20:32:28.0814290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0814368Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0814644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-26T20:32:28.0814778Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-26T20:32:28.0814781Z 2025-08-26T20:32:28.0814892Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:32:28.0815093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:32:28.0815161Z return mod(**inputs) 2025-08-26T20:32:28.0815389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:32:28.0815463Z output = func(self, *args, **kwargs) 2025-08-26T20:32:28.0815738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-26T20:32:28.0815868Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-26T20:32:28.0815871Z 2025-08-26T20:32:40.4284307Z Compilation time (from dynamo_timed): 20.003159186 2025-08-26T20:32:40.4292694Z pass 2025-08-26T20:32:40.4298301Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:40.4304116Z TIMING: _recursive_pre_grad_passes:0.0135 _recursive_joint_graph_passes:0.48638 _recursive_post_grad_passes:0.07914 async_compile.wait:0.70425 code_gen:7.77865 inductor_compile:9.08128 backend_compile:13.66964 gc:0.00255 entire_frame_compile:20.00316 total_wall_time:20.00316 2025-08-26T20:32:40.4305203Z STATS: call_* op count: 860 | FakeTensorMode.__torch_dispatch__:16775 | FakeTensor.__torch_dispatch__:4359 | ProxyTorchDispatchMode.__torch_dispatch__:5774 2025-08-26T20:32:40.4305752Z Dynamo produced 2 graphs covering 860 ops with 0 graph breaks (0 unique) 2025-08-26T20:32:46.1958370Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:32:46.1959881Z from pkg_resources import resource_filename 2025-08-26T20:32:46.8200099Z 2025-08-26T20:32:53.8717726Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:32:53.8718039Z loading model: 0it [00:07, ?it/s] 2025-08-26T20:32:53.8754377Z cpu eval M2M100ForConditionalGeneration 2025-08-26T20:32:54.7837564Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:55.1625586Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:32:55.5632686Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:33:13.2501276Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2501957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2502471Z return mod(**inputs) 2025-08-26T20:33:13.2502961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2503411Z outputs = self.model( 2025-08-26T20:33:13.2503828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2504270Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2504672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-26T20:33:13.2505186Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-26T20:33:13.2506025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-26T20:33:13.2506469Z return func(*args, **kwargs) 2025-08-26T20:33:13.2506921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-26T20:33:13.2507514Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-26T20:33:13.2508171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-08-26T20:33:13.2508736Z mask = input_ids.ne(padding_idx).int() 2025-08-26T20:33:13.2508890Z 2025-08-26T20:33:13.2509016Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2509426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2509779Z return mod(**inputs) 2025-08-26T20:33:13.2510191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2510624Z outputs = self.model( 2025-08-26T20:33:13.2511052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.2511480Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.2511905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-26T20:33:13.2512504Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-26T20:33:13.2513030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-26T20:33:13.2513429Z return func(*args, **kwargs) 2025-08-26T20:33:13.2513850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-26T20:33:13.2514444Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-26T20:33:13.2515167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-08-26T20:33:13.2515678Z mask = input_ids.ne(padding_idx).int() 2025-08-26T20:33:13.2515838Z 2025-08-26T20:33:13.2515936Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2516192Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2516434Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2516675Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2516904Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2517143Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2517385Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2517635Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2517872Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2518137Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2518366Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2518590Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2518856Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2519399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2519781Z return mod(**inputs) 2025-08-26T20:33:13.2520199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2520625Z outputs = self.model( 2025-08-26T20:33:13.2521027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2521483Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2521925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-26T20:33:13.2522392Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-26T20:33:13.2522829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-26T20:33:13.2523221Z return func(*args, **kwargs) 2025-08-26T20:33:13.2523627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-26T20:33:13.2524189Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-26T20:33:13.2524822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-26T20:33:13.2525436Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:33:13.2525695Z 2025-08-26T20:33:13.2525812Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2526206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2526558Z return mod(**inputs) 2025-08-26T20:33:13.2526956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2527368Z outputs = self.model( 2025-08-26T20:33:13.2527767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2528185Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2528597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-26T20:33:13.2529066Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-26T20:33:13.2529508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-26T20:33:13.2529892Z return func(*args, **kwargs) 2025-08-26T20:33:13.2530297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-26T20:33:13.2530875Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-26T20:33:13.2531495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-26T20:33:13.2532088Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:33:13.2532341Z 2025-08-26T20:33:13.2532454Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2532841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2533191Z return mod(**inputs) 2025-08-26T20:33:13.2533616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2534000Z outputs = self.model( 2025-08-26T20:33:13.2534377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2534776Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2535194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2535608Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2535989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2536390Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2536852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2537335Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2537772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2538265Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2538498Z 2025-08-26T20:33:13.2538612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2539003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2539352Z return mod(**inputs) 2025-08-26T20:33:13.2539738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2540153Z outputs = self.model( 2025-08-26T20:33:13.2540551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2540952Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2541339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2541726Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2542112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2542507Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2542992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2543437Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2543861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2544284Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2544445Z 2025-08-26T20:33:13.2544558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2544945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2545294Z return mod(**inputs) 2025-08-26T20:33:13.2545699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2546113Z outputs = self.model( 2025-08-26T20:33:13.2546516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2546932Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2547331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2547743Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2548122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2548573Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2549004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2549445Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2549890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2550335Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2550488Z 2025-08-26T20:33:13.2550583Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2550816Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2551044Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2551264Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2551519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2551955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2552299Z return mod(**inputs) 2025-08-26T20:33:13.2552693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2553138Z outputs = self.model( 2025-08-26T20:33:13.2553531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2553964Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2554391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2554832Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2555227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2555628Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2556063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2556507Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2556957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2557410Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2557952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2558492Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2558706Z 2025-08-26T20:33:13.2558821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2559307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2559695Z return mod(**inputs) 2025-08-26T20:33:13.2560095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2560538Z outputs = self.model( 2025-08-26T20:33:13.2560972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2561422Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2561827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2562241Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2562621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2563020Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2563455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2563925Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2564372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2564833Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2565325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2565824Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2565999Z 2025-08-26T20:33:13.2566113Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2566505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2566854Z return mod(**inputs) 2025-08-26T20:33:13.2567250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2567700Z outputs = self.model( 2025-08-26T20:33:13.2568206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2568657Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2569072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2569488Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2569866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2570275Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2570724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2571154Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2571586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2572006Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2572162Z 2025-08-26T20:33:13.2572280Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2572674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2573026Z return mod(**inputs) 2025-08-26T20:33:13.2573442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2573847Z outputs = self.model( 2025-08-26T20:33:13.2574241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2574675Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2575088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2575496Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2575880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2576310Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2576743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2577227Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2577422Z 2025-08-26T20:33:13.2577541Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2577945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2578292Z return mod(**inputs) 2025-08-26T20:33:13.2578680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2579118Z outputs = self.model( 2025-08-26T20:33:13.2579513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2579936Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2580353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2580775Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2581155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2581554Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2581989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2582467Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2582679Z 2025-08-26T20:33:13.2582801Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2583185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2583544Z return mod(**inputs) 2025-08-26T20:33:13.2583941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2584356Z outputs = self.model( 2025-08-26T20:33:13.2584753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2585162Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2585575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2585989Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2586368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2586755Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2587219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2587645Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2587798Z 2025-08-26T20:33:13.2587921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2588348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2588691Z return mod(**inputs) 2025-08-26T20:33:13.2589081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2589494Z outputs = self.model( 2025-08-26T20:33:13.2589887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2590305Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2590710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2591125Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2591530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2591934Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2592343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2592778Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2593211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2593722Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2593966Z 2025-08-26T20:33:13.2594084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2594467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2594813Z return mod(**inputs) 2025-08-26T20:33:13.2595206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2595612Z outputs = self.model( 2025-08-26T20:33:13.2596002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2596598Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2597008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2597526Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2597964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2598369Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2598796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2599379Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2599860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2600301Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2600452Z 2025-08-26T20:33:13.2600569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2600977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2601342Z return mod(**inputs) 2025-08-26T20:33:13.2601734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2602159Z outputs = self.model( 2025-08-26T20:33:13.2602552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2602976Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2603394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2603815Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2604231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2604634Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2605064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2605508Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2605959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2606391Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2606559Z 2025-08-26T20:33:13.2606649Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2606917Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2607154Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2607385Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2607648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2608056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2608420Z return mod(**inputs) 2025-08-26T20:33:13.2608825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2609244Z outputs = self.model( 2025-08-26T20:33:13.2609648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2610121Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2610549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2610975Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2611355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2611766Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2612204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2612635Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2613061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2613526Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2614007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2614527Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2614724Z 2025-08-26T20:33:13.2614844Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2615227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2615579Z return mod(**inputs) 2025-08-26T20:33:13.2615975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2616384Z outputs = self.model( 2025-08-26T20:33:13.2616775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2617186Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2617596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2618007Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2618384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2618763Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2619218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2619655Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2620086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2620525Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2621000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2621508Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2621704Z 2025-08-26T20:33:13.2621817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2622224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2622584Z return mod(**inputs) 2025-08-26T20:33:13.2622987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2623405Z outputs = self.model( 2025-08-26T20:33:13.2623797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2624212Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2624628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2625058Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2625429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2625820Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2626233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2626655Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2627079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2627504Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2627650Z 2025-08-26T20:33:13.2627768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2628175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2628520Z return mod(**inputs) 2025-08-26T20:33:13.2628910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2629318Z outputs = self.model( 2025-08-26T20:33:13.2629713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2630127Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2630527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2630940Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2631313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2631701Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2632110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2632577Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2632770Z 2025-08-26T20:33:13.2632882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2633273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2633602Z return mod(**inputs) 2025-08-26T20:33:13.2633984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2634382Z outputs = self.model( 2025-08-26T20:33:13.2634769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2635181Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2635603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2636019Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2636397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2636806Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2637232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2637702Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2637895Z 2025-08-26T20:33:13.2638008Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2638405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2638765Z return mod(**inputs) 2025-08-26T20:33:13.2639165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2639701Z outputs = self.model( 2025-08-26T20:33:13.2640109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2640546Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2640959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2641372Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2641747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2642119Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2642516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2642919Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2643084Z 2025-08-26T20:33:13.2643899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2644299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2644668Z return mod(**inputs) 2025-08-26T20:33:13.2645077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2645488Z outputs = self.model( 2025-08-26T20:33:13.2645881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2646281Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2646712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2647128Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2647512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2647907Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2648336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-26T20:33:13.2648770Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.2648923Z 2025-08-26T20:33:13.2649041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2649477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2649834Z return mod(**inputs) 2025-08-26T20:33:13.2650235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2650656Z outputs = self.model( 2025-08-26T20:33:13.2651054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2651465Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2651873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2652288Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2652685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2653076Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2653493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2653933Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2654372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2654868Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2655110Z 2025-08-26T20:33:13.2655229Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2655613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2655959Z return mod(**inputs) 2025-08-26T20:33:13.2656349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2656750Z outputs = self.model( 2025-08-26T20:33:13.2657111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2657502Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2657886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2658274Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2658646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2659014Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2659429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2659866Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2660303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2660701Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2660839Z 2025-08-26T20:33:13.2660954Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2661313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2661631Z return mod(**inputs) 2025-08-26T20:33:13.2661987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2662366Z outputs = self.model( 2025-08-26T20:33:13.2662729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2663117Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2663502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2663890Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2664253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2664638Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2665061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2665494Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2665930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2666357Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2666515Z 2025-08-26T20:33:13.2666603Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2666863Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2667093Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2667308Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2667565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2667953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2668282Z return mod(**inputs) 2025-08-26T20:33:13.2668646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2669035Z outputs = self.model( 2025-08-26T20:33:13.2669407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2669817Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2670207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2670609Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2670984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2671375Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2671790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2672219Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2672635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2673089Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2673569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2674087Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2674286Z 2025-08-26T20:33:13.2674404Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2674786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2675132Z return mod(**inputs) 2025-08-26T20:33:13.2675524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2675949Z outputs = self.model( 2025-08-26T20:33:13.2676329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2676748Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2677152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2677584Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2677970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2678363Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2678811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2679326Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2679767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2680218Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2680696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2681191Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2681371Z 2025-08-26T20:33:13.2681484Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2681894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2682239Z return mod(**inputs) 2025-08-26T20:33:13.2682634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2683047Z outputs = self.model( 2025-08-26T20:33:13.2683440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2683870Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2684273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2684722Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2685103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2685501Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2685922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2686362Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2686779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2687185Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2687327Z 2025-08-26T20:33:13.2687444Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2687829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2688159Z return mod(**inputs) 2025-08-26T20:33:13.2688529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2688918Z outputs = self.model( 2025-08-26T20:33:13.2689302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2689710Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2690122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2690538Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2690919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2691284Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2691686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2692144Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2692338Z 2025-08-26T20:33:13.2692451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2692837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2693174Z return mod(**inputs) 2025-08-26T20:33:13.2693579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2693992Z outputs = self.model( 2025-08-26T20:33:13.2694367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2694757Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2695136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2695532Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2695885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2696468Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2696886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2697357Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2697552Z 2025-08-26T20:33:13.2697665Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2698054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2698384Z return mod(**inputs) 2025-08-26T20:33:13.2698743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2699170Z outputs = self.model( 2025-08-26T20:33:13.2699545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2699958Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2700337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2700722Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2701089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2701484Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2701904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2702400Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2702571Z 2025-08-26T20:33:13.2702681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2703047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2703374Z return mod(**inputs) 2025-08-26T20:33:13.2703754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2704157Z outputs = self.model( 2025-08-26T20:33:13.2704546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2704956Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2705363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2705770Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2706134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2706524Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2706944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2707373Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2707796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2708316Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2708546Z 2025-08-26T20:33:13.2708656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2709051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2709402Z return mod(**inputs) 2025-08-26T20:33:13.2709793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2710218Z outputs = self.model( 2025-08-26T20:33:13.2710615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2711038Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2711466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2711875Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2712254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2712675Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2713112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2713542Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2713991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2714412Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2714563Z 2025-08-26T20:33:13.2714674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2715064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2715405Z return mod(**inputs) 2025-08-26T20:33:13.2715793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2716203Z outputs = self.model( 2025-08-26T20:33:13.2716595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2717017Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2717432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2717846Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2718221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2718612Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2719017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2719525Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2719966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2720412Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2720567Z 2025-08-26T20:33:13.2720664Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2720897Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2721129Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2721352Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2721610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2722001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2722357Z return mod(**inputs) 2025-08-26T20:33:13.2722758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2723221Z outputs = self.model( 2025-08-26T20:33:13.2723615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2724042Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2724436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2724817Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2725178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2725559Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2725959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2726367Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2726774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2727192Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2727642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2728135Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2728333Z 2025-08-26T20:33:13.2728458Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2728830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2729148Z return mod(**inputs) 2025-08-26T20:33:13.2729502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2729885Z outputs = self.model( 2025-08-26T20:33:13.2730253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2730651Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2731025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2731394Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2731741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2732117Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2732503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2732892Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2733287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2733690Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2734128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2734579Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2734736Z 2025-08-26T20:33:13.2734840Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2735198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2735522Z return mod(**inputs) 2025-08-26T20:33:13.2735880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2736257Z outputs = self.model( 2025-08-26T20:33:13.2736611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2736993Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2737394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2737789Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2738138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2738506Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2738911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2739310Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2739722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2740107Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2740256Z 2025-08-26T20:33:13.2740362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2740741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2741061Z return mod(**inputs) 2025-08-26T20:33:13.2741419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2741798Z outputs = self.model( 2025-08-26T20:33:13.2742168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2742577Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2742958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2743349Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2743703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2744074Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2744469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2744906Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2745080Z 2025-08-26T20:33:13.2745185Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2745572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2745899Z return mod(**inputs) 2025-08-26T20:33:13.2746266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2746651Z outputs = self.model( 2025-08-26T20:33:13.2747020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2747413Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2747800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2748193Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2748541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2748905Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2749300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2749738Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2749912Z 2025-08-26T20:33:13.2750027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2750385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2750720Z return mod(**inputs) 2025-08-26T20:33:13.2751102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2751498Z outputs = self.model( 2025-08-26T20:33:13.2751867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2752250Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2752636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2753031Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2753386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2753763Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2754163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2754567Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2754708Z 2025-08-26T20:33:13.2754834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2755197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2755524Z return mod(**inputs) 2025-08-26T20:33:13.2755895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2756304Z outputs = self.model( 2025-08-26T20:33:13.2756671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2757065Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2757447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2757847Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2758233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2758665Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2759086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-26T20:33:13.2759601Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.2759784Z 2025-08-26T20:33:13.2759899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2760296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2760643Z return mod(**inputs) 2025-08-26T20:33:13.2761014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2761438Z outputs = self.model( 2025-08-26T20:33:13.2761866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2762290Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2762711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2763129Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2763504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2763901Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2764319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2764752Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2765194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2765713Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2765950Z 2025-08-26T20:33:13.2766071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2766469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2766830Z return mod(**inputs) 2025-08-26T20:33:13.2767241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2767655Z outputs = self.model( 2025-08-26T20:33:13.2768048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2768476Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2768907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2769334Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2769723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2770116Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2770533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2770976Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2771405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2771855Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2772000Z 2025-08-26T20:33:13.2772120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2772503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2772876Z return mod(**inputs) 2025-08-26T20:33:13.2773279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2773698Z outputs = self.model( 2025-08-26T20:33:13.2774097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2774504Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2774915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2775341Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2775721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2776111Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2776539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2776975Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2777403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2777826Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2777972Z 2025-08-26T20:33:13.2778055Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2778276Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2778493Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2778706Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2778934Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2779300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2779632Z return mod(**inputs) 2025-08-26T20:33:13.2779997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2780402Z outputs = self.model( 2025-08-26T20:33:13.2780772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2781167Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2781556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2781952Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2782305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2782670Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2783088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2783527Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2783958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2784391Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2784850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2785341Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2785531Z 2025-08-26T20:33:13.2785662Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2786029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2786350Z return mod(**inputs) 2025-08-26T20:33:13.2786724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2787114Z outputs = self.model( 2025-08-26T20:33:13.2787490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2787882Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2788280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2788694Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2789047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2789451Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2789838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2790248Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2790656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2791078Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2791555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2792034Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2792206Z 2025-08-26T20:33:13.2792316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2792707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2793059Z return mod(**inputs) 2025-08-26T20:33:13.2793428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2793813Z outputs = self.model( 2025-08-26T20:33:13.2794186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2794579Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2794993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2795406Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2795787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2796304Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2796746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2797187Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2797663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2798094Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2798249Z 2025-08-26T20:33:13.2798360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2798750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2799098Z return mod(**inputs) 2025-08-26T20:33:13.2799532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2799960Z outputs = self.model( 2025-08-26T20:33:13.2800353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2800800Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2801199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2801616Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2801996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2802386Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2802806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2803262Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2803457Z 2025-08-26T20:33:13.2803568Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2803987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2804339Z return mod(**inputs) 2025-08-26T20:33:13.2804739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2805146Z outputs = self.model( 2025-08-26T20:33:13.2805539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2805949Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2806361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2806762Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2807144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2807511Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2807906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2808343Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2808515Z 2025-08-26T20:33:13.2808622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2808991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2809317Z return mod(**inputs) 2025-08-26T20:33:13.2809711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2810101Z outputs = self.model( 2025-08-26T20:33:13.2810465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2810859Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2811246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2811639Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2811985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2812368Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2812759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2813160Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2813303Z 2025-08-26T20:33:13.2813418Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2813786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2814121Z return mod(**inputs) 2025-08-26T20:33:13.2814496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2814910Z outputs = self.model( 2025-08-26T20:33:13.2815286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2815678Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2816073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2816470Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2816830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2817195Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2817596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2818010Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2818435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2818905Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2819113Z 2025-08-26T20:33:13.2819218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2819587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2819918Z return mod(**inputs) 2025-08-26T20:33:13.2820312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2820729Z outputs = self.model( 2025-08-26T20:33:13.2821093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2821492Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2821881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2822295Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2822663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2823036Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2823434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2823863Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2824271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2824664Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2824810Z 2025-08-26T20:33:13.2824915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2825284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2825616Z return mod(**inputs) 2025-08-26T20:33:13.2826009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2826422Z outputs = self.model( 2025-08-26T20:33:13.2826841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2827250Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2827635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2828019Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2828372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2828738Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2829132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2829572Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2829975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2830383Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2830536Z 2025-08-26T20:33:13.2830619Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2830841Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2831054Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2831258Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2831498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2831867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2832211Z return mod(**inputs) 2025-08-26T20:33:13.2832572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2832960Z outputs = self.model( 2025-08-26T20:33:13.2833328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2833718Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2834096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2834482Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2834835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2835220Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2835635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2836064Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2836489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2836921Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2837400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2837930Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2838129Z 2025-08-26T20:33:13.2838242Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2838634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2838991Z return mod(**inputs) 2025-08-26T20:33:13.2839475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2839912Z outputs = self.model( 2025-08-26T20:33:13.2840307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2840745Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2841177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2841593Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2841966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2842360Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2842782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2843215Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2843642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2844093Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2844577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2845079Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2845257Z 2025-08-26T20:33:13.2845379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2845773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2846124Z return mod(**inputs) 2025-08-26T20:33:13.2846526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2846943Z outputs = self.model( 2025-08-26T20:33:13.2847357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2847767Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2848176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2848590Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2848966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2849357Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2849789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2850231Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2850658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2851081Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2851229Z 2025-08-26T20:33:13.2851348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2851722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2852053Z return mod(**inputs) 2025-08-26T20:33:13.2852420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2852809Z outputs = self.model( 2025-08-26T20:33:13.2853189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2853587Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2853971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2854360Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2854717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2855082Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2855495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2855933Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2856109Z 2025-08-26T20:33:13.2856221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2856588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2856913Z return mod(**inputs) 2025-08-26T20:33:13.2857278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2857693Z outputs = self.model( 2025-08-26T20:33:13.2858066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2858468Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2858852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2859241Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2859596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2859962Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2860350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2860781Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2860959Z 2025-08-26T20:33:13.2861066Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2861452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2861783Z return mod(**inputs) 2025-08-26T20:33:13.2862140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2862525Z outputs = self.model( 2025-08-26T20:33:13.2862897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2863289Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2863666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2864057Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2864411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2864775Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2865171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2865556Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2865699Z 2025-08-26T20:33:13.2865801Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2866159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2866485Z return mod(**inputs) 2025-08-26T20:33:13.2866874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2867274Z outputs = self.model( 2025-08-26T20:33:13.2867662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2868085Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2868469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2868859Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2869203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2869571Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2869953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-26T20:33:13.2870347Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.2870485Z 2025-08-26T20:33:13.2870591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2870972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2871329Z return mod(**inputs) 2025-08-26T20:33:13.2871695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2872113Z outputs = self.model( 2025-08-26T20:33:13.2872472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2872863Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2873252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2873644Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2873995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2874368Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2874764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2875179Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2875608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2876077Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2876290Z 2025-08-26T20:33:13.2876395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2876764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2877106Z return mod(**inputs) 2025-08-26T20:33:13.2877507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2877923Z outputs = self.model( 2025-08-26T20:33:13.2878322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2878744Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2879169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2879675Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2880060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2880461Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2880888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2881323Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2881717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2882128Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2882281Z 2025-08-26T20:33:13.2882393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2882787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2883142Z return mod(**inputs) 2025-08-26T20:33:13.2883523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2883932Z outputs = self.model( 2025-08-26T20:33:13.2884348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2884768Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2885167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2885591Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2885968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2886364Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2886783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2887225Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2887659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2888087Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2888239Z 2025-08-26T20:33:13.2888334Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2888567Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2888786Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2889007Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2889260Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2889649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2889990Z return mod(**inputs) 2025-08-26T20:33:13.2890356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2890745Z outputs = self.model( 2025-08-26T20:33:13.2891117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2891494Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2891867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2892252Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2892602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2892959Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2893335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2893737Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2894133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2894542Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2894986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2895454Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2895666Z 2025-08-26T20:33:13.2895773Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2896135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2896580Z return mod(**inputs) 2025-08-26T20:33:13.2896948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2897330Z outputs = self.model( 2025-08-26T20:33:13.2897697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2898083Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2898510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2898899Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2899251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2899620Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2900014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2900428Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2900817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2901251Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2901731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2902222Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2902388Z 2025-08-26T20:33:13.2902498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2902861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2903201Z return mod(**inputs) 2025-08-26T20:33:13.2903559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2903939Z outputs = self.model( 2025-08-26T20:33:13.2904316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2904711Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2905110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2905495Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2905718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2905809Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2906057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2906149Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2906401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2906488Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2906493Z 2025-08-26T20:33:13.2906606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2906809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2906887Z return mod(**inputs) 2025-08-26T20:33:13.2907142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2907214Z outputs = self.model( 2025-08-26T20:33:13.2908328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2908411Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2908668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2908740Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2908960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2909049Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2909311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2909440Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2909444Z 2025-08-26T20:33:13.2909548Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2909754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2909821Z return mod(**inputs) 2025-08-26T20:33:13.2910071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2910147Z outputs = self.model( 2025-08-26T20:33:13.2910394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2910494Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2910743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2910817Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2911045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2911127Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2911385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2911506Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2911509Z 2025-08-26T20:33:13.2911621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2911841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2911908Z return mod(**inputs) 2025-08-26T20:33:13.2912173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2912241Z outputs = self.model( 2025-08-26T20:33:13.2912503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2912577Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2912835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2912915Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2913134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2913221Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2913473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2913557Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2913569Z 2025-08-26T20:33:13.2913673Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2913875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2913949Z return mod(**inputs) 2025-08-26T20:33:13.2914221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2914300Z outputs = self.model( 2025-08-26T20:33:13.2914556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2914630Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2914905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2914985Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2915229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2915330Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2915596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2915704Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2915972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2916143Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2916147Z 2025-08-26T20:33:13.2916256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2916476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2916567Z return mod(**inputs) 2025-08-26T20:33:13.2916837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2916921Z outputs = self.model( 2025-08-26T20:33:13.2917192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2917275Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2917543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2917619Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2917860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2917944Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2918239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2918340Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2918606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2918700Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2918704Z 2025-08-26T20:33:13.2918813Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2919033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2919102Z return mod(**inputs) 2025-08-26T20:33:13.2919462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2919544Z outputs = self.model( 2025-08-26T20:33:13.2919836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2919924Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2920191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2920279Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2920522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2920629Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2920938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2921033Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2921290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2921381Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2921386Z 2025-08-26T20:33:13.2921476Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2921558Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2921636Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2921741Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2921847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2922054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2922122Z return mod(**inputs) 2025-08-26T20:33:13.2922380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2922458Z outputs = self.model( 2025-08-26T20:33:13.2922716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2922798Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2923071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2923144Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2923376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2923457Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2923719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2923809Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2924065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2924171Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2924487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2924632Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2924636Z 2025-08-26T20:33:13.2924739Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2924952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2925020Z return mod(**inputs) 2025-08-26T20:33:13.2925279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2925355Z outputs = self.model( 2025-08-26T20:33:13.2925610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2925691Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2925941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2926017Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2926249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2926328Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2926591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2926683Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2926956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2927061Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2927358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2927485Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2927491Z 2025-08-26T20:33:13.2927598Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2927810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2927894Z return mod(**inputs) 2025-08-26T20:33:13.2928154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2928232Z outputs = self.model( 2025-08-26T20:33:13.2928488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2928569Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2928825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2928905Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2929144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2929225Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2929487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2929580Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2929840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2929923Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2929927Z 2025-08-26T20:33:13.2930031Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2930241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2930308Z return mod(**inputs) 2025-08-26T20:33:13.2930597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2930667Z outputs = self.model( 2025-08-26T20:33:13.2930922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2930995Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2931250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2931327Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2931536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2931620Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2931856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2931973Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2931978Z 2025-08-26T20:33:13.2932083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2932271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2932343Z return mod(**inputs) 2025-08-26T20:33:13.2932602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2932671Z outputs = self.model( 2025-08-26T20:33:13.2932962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2933036Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2933295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2933369Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2933598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2933678Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2933946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2934074Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2934078Z 2025-08-26T20:33:13.2934180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2934388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2934455Z return mod(**inputs) 2025-08-26T20:33:13.2934709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2934786Z outputs = self.model( 2025-08-26T20:33:13.2935041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2935137Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2935389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2935466Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2935701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2935786Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2936062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2936149Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2936153Z 2025-08-26T20:33:13.2936268Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2936502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2936574Z return mod(**inputs) 2025-08-26T20:33:13.2936853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2936933Z outputs = self.model( 2025-08-26T20:33:13.2937197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2937270Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2937536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2937609Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2937832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2937922Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2938184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-26T20:33:13.2938273Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.2938277Z 2025-08-26T20:33:13.2938377Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2938575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2938648Z return mod(**inputs) 2025-08-26T20:33:13.2938910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2938986Z outputs = self.model( 2025-08-26T20:33:13.2939232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2939302Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2939563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2939639Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2939868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2939949Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2940226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2940322Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2940576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2940739Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2940743Z 2025-08-26T20:33:13.2940847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2941058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2941141Z return mod(**inputs) 2025-08-26T20:33:13.2941397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2941475Z outputs = self.model( 2025-08-26T20:33:13.2941731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2941811Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2942064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2942144Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2942366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2942446Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2942738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2942833Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2943097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2943182Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2943186Z 2025-08-26T20:33:13.2943296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2943518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2943589Z return mod(**inputs) 2025-08-26T20:33:13.2943866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2943940Z outputs = self.model( 2025-08-26T20:33:13.2944223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2944301Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2944563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2944643Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2944862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2944965Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2945216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2945304Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2945567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2945660Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2945665Z 2025-08-26T20:33:13.2945761Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2945848Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2945931Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2946022Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2946148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2946369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2946440Z return mod(**inputs) 2025-08-26T20:33:13.2946712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2946792Z outputs = self.model( 2025-08-26T20:33:13.2947062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2947150Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2947440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2947524Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2947763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2947846Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2948125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2948221Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2948506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2948609Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2948943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2949097Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2949101Z 2025-08-26T20:33:13.2949209Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2949430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2949500Z return mod(**inputs) 2025-08-26T20:33:13.2949782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2949853Z outputs = self.model( 2025-08-26T20:33:13.2950124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2950208Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2950475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2950563Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2950802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2950886Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2951161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2951255Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2951551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2951656Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2951973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2952095Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2952101Z 2025-08-26T20:33:13.2952210Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2952432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2952504Z return mod(**inputs) 2025-08-26T20:33:13.2952796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2952869Z outputs = self.model( 2025-08-26T20:33:13.2953141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2953227Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2953497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2953578Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2953801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2953916Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2954185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2954280Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2954555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2954641Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2954645Z 2025-08-26T20:33:13.2954771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2954973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2955039Z return mod(**inputs) 2025-08-26T20:33:13.2955322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2955394Z outputs = self.model( 2025-08-26T20:33:13.2955655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2955731Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2955989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2956076Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2956311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2956403Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2956671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2956807Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2956812Z 2025-08-26T20:33:13.2956923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2957133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2957212Z return mod(**inputs) 2025-08-26T20:33:13.2957483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2957563Z outputs = self.model( 2025-08-26T20:33:13.2957858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2957938Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2958214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2958291Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2958536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2958623Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2958910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2959046Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2959050Z 2025-08-26T20:33:13.2959159Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2959464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2959541Z return mod(**inputs) 2025-08-26T20:33:13.2959819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2959894Z outputs = self.model( 2025-08-26T20:33:13.2960165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2960274Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2960547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2960630Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2960854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2960935Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2961197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.2961279Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.2961282Z 2025-08-26T20:33:13.2961393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2961593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2961687Z return mod(**inputs) 2025-08-26T20:33:13.2961942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2962012Z outputs = self.model( 2025-08-26T20:33:13.2962274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2962348Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2962611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2962681Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2962894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2962981Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2963232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2963330Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2963583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.2963752Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.2963756Z 2025-08-26T20:33:13.2963865Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2964092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2964169Z return mod(**inputs) 2025-08-26T20:33:13.2964438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2964515Z outputs = self.model( 2025-08-26T20:33:13.2964776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2964852Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2965115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2965205Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2965435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2965517Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2965770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2965869Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2966123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.2966215Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.2966245Z 2025-08-26T20:33:13.2966350Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2966557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2966626Z return mod(**inputs) 2025-08-26T20:33:13.2966881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2966958Z outputs = self.model( 2025-08-26T20:33:13.2967212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2967293Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2967543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2967634Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2967863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2967952Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2968206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2968294Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2968548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.2968634Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.2968637Z 2025-08-26T20:33:13.2968717Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2968804Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2968882Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2968964Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.2969073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2969290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2969367Z return mod(**inputs) 2025-08-26T20:33:13.2969642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2969725Z outputs = self.model( 2025-08-26T20:33:13.2970024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2970104Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2970379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2970458Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2970700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2970788Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2971059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2971162Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2971451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2971567Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2971881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.2972032Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.2972037Z 2025-08-26T20:33:13.2972148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2972362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2972465Z return mod(**inputs) 2025-08-26T20:33:13.2972745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2972825Z outputs = self.model( 2025-08-26T20:33:13.2973105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2973183Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2973463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2973539Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2973784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2973870Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2974166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2974263Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2974530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.2974643Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.2974956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.2975082Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.2975087Z 2025-08-26T20:33:13.2975197Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2975417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2975489Z return mod(**inputs) 2025-08-26T20:33:13.2975763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2975847Z outputs = self.model( 2025-08-26T20:33:13.2976118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2976205Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2976477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2976572Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2976824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2976911Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2977188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.2977286Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.2977555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.2977652Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.2977656Z 2025-08-26T20:33:13.2977781Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2978000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2996963Z return mod(**inputs) 2025-08-26T20:33:13.2997500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.2997598Z outputs = self.model( 2025-08-26T20:33:13.2997891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.2997996Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.2998275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.2998515Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.2998773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.2998869Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.2999158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.2999363Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.2999371Z 2025-08-26T20:33:13.2999510Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.2999747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.2999826Z return mod(**inputs) 2025-08-26T20:33:13.3000164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3000245Z outputs = self.model( 2025-08-26T20:33:13.3000529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3000616Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3000893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3000976Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3001220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3001319Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3001593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.3001733Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3001740Z 2025-08-26T20:33:13.3001859Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3002080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3002162Z return mod(**inputs) 2025-08-26T20:33:13.3002438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3002523Z outputs = self.model( 2025-08-26T20:33:13.3002833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3002927Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3003199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3003280Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3003530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3003619Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3003944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.3004036Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3004041Z 2025-08-26T20:33:13.3004155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3004383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3004455Z return mod(**inputs) 2025-08-26T20:33:13.3004736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3004809Z outputs = self.model( 2025-08-26T20:33:13.3005084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3005191Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3005465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3005550Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3005791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3005885Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3006158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-26T20:33:13.3006246Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3006250Z 2025-08-26T20:33:13.3006370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3006587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3006687Z return mod(**inputs) 2025-08-26T20:33:13.3006970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3007044Z outputs = self.model( 2025-08-26T20:33:13.3007327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3007407Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3007689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3007767Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3008014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3008097Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3008369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3008482Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3008754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3008929Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3008933Z 2025-08-26T20:33:13.3009048Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3009290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3009363Z return mod(**inputs) 2025-08-26T20:33:13.3009634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3009715Z outputs = self.model( 2025-08-26T20:33:13.3009987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3010074Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3010342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3010435Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3010679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3010767Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3011040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3011142Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3011407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3011502Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3011523Z 2025-08-26T20:33:13.3011636Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3011855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3011926Z return mod(**inputs) 2025-08-26T20:33:13.3012204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3012277Z outputs = self.model( 2025-08-26T20:33:13.3012547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3012631Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3012901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3012985Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3013239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3013320Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3013582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3013675Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3013936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3014024Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3014028Z 2025-08-26T20:33:13.3014118Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3014200Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3014278Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3014363Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3014470Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3014680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3014750Z return mod(**inputs) 2025-08-26T20:33:13.3015008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3015083Z outputs = self.model( 2025-08-26T20:33:13.3015338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3015434Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3015688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3015762Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3015991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3016076Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3016335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3016427Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3016696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3016809Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3017111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3017261Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3017265Z 2025-08-26T20:33:13.3017370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3017577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3017664Z return mod(**inputs) 2025-08-26T20:33:13.3017924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3018003Z outputs = self.model( 2025-08-26T20:33:13.3018262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3018344Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3018597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3018672Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3018903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3018984Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3019263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3019358Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3019620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3019721Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3020019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3020145Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3020148Z 2025-08-26T20:33:13.3020253Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3020463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3020530Z return mod(**inputs) 2025-08-26T20:33:13.3020787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3020866Z outputs = self.model( 2025-08-26T20:33:13.3021122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3021207Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3021459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3021566Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3021794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3021875Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3022139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3022233Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3022495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3022579Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3022583Z 2025-08-26T20:33:13.3022703Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3022914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3022982Z return mod(**inputs) 2025-08-26T20:33:13.3023249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3023317Z outputs = self.model( 2025-08-26T20:33:13.3023574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3023656Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3023909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3024009Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3024238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3024327Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3024585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.3024709Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3024713Z 2025-08-26T20:33:13.3024827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3025030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3025101Z return mod(**inputs) 2025-08-26T20:33:13.3025377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3025446Z outputs = self.model( 2025-08-26T20:33:13.3025714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3025789Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3026055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3026128Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3026361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3026441Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3026696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.3026825Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3026830Z 2025-08-26T20:33:13.3026934Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3027141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3027206Z return mod(**inputs) 2025-08-26T20:33:13.3027475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3027550Z outputs = self.model( 2025-08-26T20:33:13.3027817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3027897Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3028145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3028216Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3028446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3028528Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3028804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.3028889Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3028893Z 2025-08-26T20:33:13.3029003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3029205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3029273Z return mod(**inputs) 2025-08-26T20:33:13.3029534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3029600Z outputs = self.model( 2025-08-26T20:33:13.3029861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3029955Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3030208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3030290Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3030510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3030597Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3030851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3030950Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3031201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3031381Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3031387Z 2025-08-26T20:33:13.3031508Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3031726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3031805Z return mod(**inputs) 2025-08-26T20:33:13.3032079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3032148Z outputs = self.model( 2025-08-26T20:33:13.3032415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3032491Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3032755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3032829Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3033063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3033148Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3033418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3033522Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3033802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3033897Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3033900Z 2025-08-26T20:33:13.3034010Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3034225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3034304Z return mod(**inputs) 2025-08-26T20:33:13.3034585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3034668Z outputs = self.model( 2025-08-26T20:33:13.3034943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3035041Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3035329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3035410Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3035663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3035750Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3036037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3036137Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3036437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3036539Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3036543Z 2025-08-26T20:33:13.3036631Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3036723Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3036806Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3036887Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3037009Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3037225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3037303Z return mod(**inputs) 2025-08-26T20:33:13.3037575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3037668Z outputs = self.model( 2025-08-26T20:33:13.3037950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3038031Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3038309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3038387Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3038633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3038718Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3039005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3039113Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3039494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3039622Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3039945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3040101Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3040105Z 2025-08-26T20:33:13.3040227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3040470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3040554Z return mod(**inputs) 2025-08-26T20:33:13.3040844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3040922Z outputs = self.model( 2025-08-26T20:33:13.3041184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3041264Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3041536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3041626Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3041858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3041938Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3042205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3042304Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3042551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3042655Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3042954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3043074Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3043078Z 2025-08-26T20:33:13.3043181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3043374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3043447Z return mod(**inputs) 2025-08-26T20:33:13.3043695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3043768Z outputs = self.model( 2025-08-26T20:33:13.3044020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3044119Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3044387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3044459Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3044697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3044777Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3045034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-26T20:33:13.3045131Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:33:13.3045385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3045477Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3045481Z 2025-08-26T20:33:13.3045586Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3045801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3045867Z return mod(**inputs) 2025-08-26T20:33:13.3046125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3046203Z outputs = self.model( 2025-08-26T20:33:13.3046458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3046556Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3046815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3046886Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3047121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3047205Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3047468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.3047592Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3047596Z 2025-08-26T20:33:13.3047732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3047932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3047997Z return mod(**inputs) 2025-08-26T20:33:13.3048258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3048327Z outputs = self.model( 2025-08-26T20:33:13.3048590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3048666Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3048937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3049016Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3049244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3049333Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3049589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-26T20:33:13.3049711Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3049722Z 2025-08-26T20:33:13.3049825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3050027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3050101Z return mod(**inputs) 2025-08-26T20:33:13.3050374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3050452Z outputs = self.model( 2025-08-26T20:33:13.3050706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3050781Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3051044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3051117Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3051344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3051424Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3051677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-26T20:33:13.3051771Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3051776Z 2025-08-26T20:33:13.3051880Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3052085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3052152Z return mod(**inputs) 2025-08-26T20:33:13.3052415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3052483Z outputs = self.model( 2025-08-26T20:33:13.3052752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-26T20:33:13.3052837Z encoder_outputs = self.encoder( 2025-08-26T20:33:13.3053089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-26T20:33:13.3053168Z layer_outputs = encoder_layer( 2025-08-26T20:33:13.3053392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3053473Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3053733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-26T20:33:13.3053830Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3053834Z 2025-08-26T20:33:13.3053946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3054147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3054221Z return mod(**inputs) 2025-08-26T20:33:13.3054478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3054547Z outputs = self.model( 2025-08-26T20:33:13.3054809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3054899Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3055171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-26T20:33:13.3055348Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-26T20:33:13.3055592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-26T20:33:13.3055676Z return func(*args, **kwargs) 2025-08-26T20:33:13.3055937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-26T20:33:13.3056163Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-26T20:33:13.3056491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-26T20:33:13.3056717Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:33:13.3056722Z 2025-08-26T20:33:13.3056826Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3057030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3057107Z return mod(**inputs) 2025-08-26T20:33:13.3057369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3057446Z outputs = self.model( 2025-08-26T20:33:13.3057702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3057777Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3058038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-26T20:33:13.3058209Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-26T20:33:13.3058449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-26T20:33:13.3058523Z return func(*args, **kwargs) 2025-08-26T20:33:13.3058787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-26T20:33:13.3059015Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-26T20:33:13.3059335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-26T20:33:13.3059532Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:33:13.3059538Z 2025-08-26T20:33:13.3059645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3059856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3059923Z return mod(**inputs) 2025-08-26T20:33:13.3060199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3060270Z outputs = self.model( 2025-08-26T20:33:13.3060530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3060611Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3060868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3060949Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3061172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3061275Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3061540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3061645Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3061911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3062066Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3062069Z 2025-08-26T20:33:13.3062180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3062387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3062455Z return mod(**inputs) 2025-08-26T20:33:13.3062723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3062813Z outputs = self.model( 2025-08-26T20:33:13.3063095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3063177Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3063464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3063551Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3063789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3063882Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3064163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3064284Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3064549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3064631Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3064635Z 2025-08-26T20:33:13.3064749Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3064949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3065022Z return mod(**inputs) 2025-08-26T20:33:13.3065292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3065363Z outputs = self.model( 2025-08-26T20:33:13.3065626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3065699Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3065963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3066038Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3066261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3066365Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3066622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3066730Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3066985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3067080Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3067084Z 2025-08-26T20:33:13.3067165Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3067250Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3067363Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3067440Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3067551Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3067759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3067827Z return mod(**inputs) 2025-08-26T20:33:13.3068091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3068162Z outputs = self.model( 2025-08-26T20:33:13.3068422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3068496Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3068750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3068851Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3069077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3069165Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3069432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3069537Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3069790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3069886Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3070183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3070318Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3070325Z 2025-08-26T20:33:13.3070433Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3070628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3070692Z return mod(**inputs) 2025-08-26T20:33:13.3070950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3071017Z outputs = self.model( 2025-08-26T20:33:13.3071291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3071366Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3071621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3071693Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3071913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3072000Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3072248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3072408Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3072664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3072765Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3073068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3073180Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3073184Z 2025-08-26T20:33:13.3073296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3073501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3073592Z return mod(**inputs) 2025-08-26T20:33:13.3073849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3073919Z outputs = self.model( 2025-08-26T20:33:13.3074196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3074276Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3074558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3074635Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3074873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3074985Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3075259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3075374Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3075653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3075753Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3075756Z 2025-08-26T20:33:13.3075872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3076093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3076175Z return mod(**inputs) 2025-08-26T20:33:13.3076456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3076539Z outputs = self.model( 2025-08-26T20:33:13.3076815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3076896Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3077182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3077261Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3077511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3077615Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3077892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3078023Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3078298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3078477Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3078482Z 2025-08-26T20:33:13.3078595Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3078837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3078911Z return mod(**inputs) 2025-08-26T20:33:13.3079190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3079554Z outputs = self.model( 2025-08-26T20:33:13.3079851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3079938Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3080206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3080313Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3080561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3080645Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3080925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3081043Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3081332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3081415Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3081419Z 2025-08-26T20:33:13.3081524Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3081735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3081819Z return mod(**inputs) 2025-08-26T20:33:13.3082087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3082159Z outputs = self.model( 2025-08-26T20:33:13.3082413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3082491Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3082743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3082820Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3083042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3083127Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3083380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3083492Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3083750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3083841Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3083845Z 2025-08-26T20:33:13.3083932Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3084011Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3084106Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3084194Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3084300Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3084507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3084573Z return mod(**inputs) 2025-08-26T20:33:13.3084836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3084916Z outputs = self.model( 2025-08-26T20:33:13.3085173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3085270Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3085522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3085595Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3085828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3085909Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3086171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3086282Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3086564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3086669Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3086984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3087138Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3087142Z 2025-08-26T20:33:13.3087253Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3087472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3087545Z return mod(**inputs) 2025-08-26T20:33:13.3087819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3087928Z outputs = self.model( 2025-08-26T20:33:13.3088184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3088265Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3088524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3088606Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3088831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3088912Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3089170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3089279Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3089539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3089639Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3089931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3090049Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3090052Z 2025-08-26T20:33:13.3090157Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3090380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3090448Z return mod(**inputs) 2025-08-26T20:33:13.3090709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3090779Z outputs = self.model( 2025-08-26T20:33:13.3091029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3091117Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3091372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3091451Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3091699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3091781Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3092044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3092153Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3092416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3092502Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3092523Z 2025-08-26T20:33:13.3092636Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3092837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3092904Z return mod(**inputs) 2025-08-26T20:33:13.3093174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3093242Z outputs = self.model( 2025-08-26T20:33:13.3093511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3093585Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3093840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3093920Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3094159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3094247Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3094501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3094624Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3094634Z 2025-08-26T20:33:13.3094738Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3094939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3095015Z return mod(**inputs) 2025-08-26T20:33:13.3095272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3095348Z outputs = self.model( 2025-08-26T20:33:13.3095604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3095682Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3095943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3096016Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3096532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3096620Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3096929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3097062Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3097066Z 2025-08-26T20:33:13.3097170Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3097384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3097451Z return mod(**inputs) 2025-08-26T20:33:13.3097713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3097780Z outputs = self.model( 2025-08-26T20:33:13.3098059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3098144Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3098398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3098478Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3098698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3098778Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3099038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3099152Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3099156Z 2025-08-26T20:33:13.3099266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3099468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3099541Z return mod(**inputs) 2025-08-26T20:33:13.3099802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3099870Z outputs = self.model( 2025-08-26T20:33:13.3100133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3100206Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3100467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3100564Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3100789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3100879Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3101157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3101273Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3101572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3101734Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3101744Z 2025-08-26T20:33:13.3101854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3102068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3102147Z return mod(**inputs) 2025-08-26T20:33:13.3102429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3102508Z outputs = self.model( 2025-08-26T20:33:13.3102787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3102865Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3103170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3103249Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3103493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3103578Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3103866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3103984Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3104282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3104378Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3104382Z 2025-08-26T20:33:13.3104491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3104711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3104781Z return mod(**inputs) 2025-08-26T20:33:13.3105049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3105130Z outputs = self.model( 2025-08-26T20:33:13.3105400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3105502Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3105778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3105855Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3106106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3106192Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3106538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3106646Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3106936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3107047Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3107052Z 2025-08-26T20:33:13.3107140Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3107236Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3107319Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3107407Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3107518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3107733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3107813Z return mod(**inputs) 2025-08-26T20:33:13.3108086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3108165Z outputs = self.model( 2025-08-26T20:33:13.3108450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3108530Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3108827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3108904Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3109149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3109232Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3109533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3109649Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3109928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3110037Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3110349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3110501Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3110505Z 2025-08-26T20:33:13.3110615Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3110843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3110923Z return mod(**inputs) 2025-08-26T20:33:13.3111192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3111271Z outputs = self.model( 2025-08-26T20:33:13.3111540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3111619Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3111894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3111990Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3112234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3112318Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3112593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3112700Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3112969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3113077Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3113385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3113546Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3113551Z 2025-08-26T20:33:13.3113660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3113877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3113955Z return mod(**inputs) 2025-08-26T20:33:13.3114235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3114315Z outputs = self.model( 2025-08-26T20:33:13.3114596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3114683Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3114960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3115038Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3115290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3115373Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3115658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3115763Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3116066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3116165Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3116169Z 2025-08-26T20:33:13.3116279Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3116500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3116570Z return mod(**inputs) 2025-08-26T20:33:13.3116849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3116923Z outputs = self.model( 2025-08-26T20:33:13.3117188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3117290Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3117559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3117646Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3117883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3117967Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3118245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-26T20:33:13.3118334Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3118355Z 2025-08-26T20:33:13.3118473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3118689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3118760Z return mod(**inputs) 2025-08-26T20:33:13.3119037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3119108Z outputs = self.model( 2025-08-26T20:33:13.3119476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3119562Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3119846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3119932Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3120193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3120290Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3120567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3120695Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3120976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3121129Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3121133Z 2025-08-26T20:33:13.3121247Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3121449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3121527Z return mod(**inputs) 2025-08-26T20:33:13.3121783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3121853Z outputs = self.model( 2025-08-26T20:33:13.3122119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3122195Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3122456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3122547Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3122779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3122859Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3123110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3123232Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3123484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3123574Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3123578Z 2025-08-26T20:33:13.3123698Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3123901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3123976Z return mod(**inputs) 2025-08-26T20:33:13.3124255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3124329Z outputs = self.model( 2025-08-26T20:33:13.3124584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3124667Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3124938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3125009Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3125247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3125330Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3125612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3125724Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3125992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3126091Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3126113Z 2025-08-26T20:33:13.3126199Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3126293Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3126376Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3126458Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3126574Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3126790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3126867Z return mod(**inputs) 2025-08-26T20:33:13.3127139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3127221Z outputs = self.model( 2025-08-26T20:33:13.3127481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3127556Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3127815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3127890Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3128122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3128202Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3128459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3128592Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3128846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3128950Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3129244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3129379Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3129390Z 2025-08-26T20:33:13.3129493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3129691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3129782Z return mod(**inputs) 2025-08-26T20:33:13.3130044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3130120Z outputs = self.model( 2025-08-26T20:33:13.3130379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3130454Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3130719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3130794Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3131043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3131125Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3131379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3131498Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3131755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3131857Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3132152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3132266Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3132287Z 2025-08-26T20:33:13.3132393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3132595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3132671Z return mod(**inputs) 2025-08-26T20:33:13.3132928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3133006Z outputs = self.model( 2025-08-26T20:33:13.3133262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3133337Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3133600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3133673Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3133903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3133986Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3134244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3134351Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3134603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3134694Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3134716Z 2025-08-26T20:33:13.3134822Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3135028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3135094Z return mod(**inputs) 2025-08-26T20:33:13.3135345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3135423Z outputs = self.model( 2025-08-26T20:33:13.3135675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3135756Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3136025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3136108Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3136335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3136415Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3136680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3136801Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3136806Z 2025-08-26T20:33:13.3136918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3137135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3137200Z return mod(**inputs) 2025-08-26T20:33:13.3137463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3137533Z outputs = self.model( 2025-08-26T20:33:13.3137802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3137880Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3138148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3138233Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3138471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3138593Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3138863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3138998Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3139002Z 2025-08-26T20:33:13.3139113Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3139326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3139404Z return mod(**inputs) 2025-08-26T20:33:13.3139673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3139752Z outputs = self.model( 2025-08-26T20:33:13.3140025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3140106Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3140380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3140456Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3140702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3140787Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3141077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3141168Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3141172Z 2025-08-26T20:33:13.3141283Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3141502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3141574Z return mod(**inputs) 2025-08-26T20:33:13.3141852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3141924Z outputs = self.model( 2025-08-26T20:33:13.3142206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3142290Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3142562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3142644Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3142881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3142965Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3143265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3143390Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3143685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3143847Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3143851Z 2025-08-26T20:33:13.3143968Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3144202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3144271Z return mod(**inputs) 2025-08-26T20:33:13.3144561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3144632Z outputs = self.model( 2025-08-26T20:33:13.3144925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3145023Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3145303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3145387Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3145627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3145720Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3146016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3146129Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3146424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3146512Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3146517Z 2025-08-26T20:33:13.3146637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3146867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3146944Z return mod(**inputs) 2025-08-26T20:33:13.3147246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3147319Z outputs = self.model( 2025-08-26T20:33:13.3147646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3147728Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3148020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3148098Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3148342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3148431Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3148708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3148838Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3149110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3149213Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3149216Z 2025-08-26T20:33:13.3149303Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3149389Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3149480Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3149562Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3149679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3149894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3149983Z return mod(**inputs) 2025-08-26T20:33:13.3150261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3150336Z outputs = self.model( 2025-08-26T20:33:13.3150618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3150691Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3150950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3151030Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3151252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3151358Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3151615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3151724Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3151985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3152083Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3152389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3152526Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3152530Z 2025-08-26T20:33:13.3152642Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3152843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3152912Z return mod(**inputs) 2025-08-26T20:33:13.3153179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3153247Z outputs = self.model( 2025-08-26T20:33:13.3153513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3153587Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3153864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3153941Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3154165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3154253Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3154505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3154617Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3154872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3154985Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3155288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3155401Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3155405Z 2025-08-26T20:33:13.3155516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3155716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3155787Z return mod(**inputs) 2025-08-26T20:33:13.3156040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3156126Z outputs = self.model( 2025-08-26T20:33:13.3156388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3156463Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3156725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3156798Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3157023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3157110Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3157361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3157464Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3157743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3157833Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3157837Z 2025-08-26T20:33:13.3157944Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3158162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3158243Z return mod(**inputs) 2025-08-26T20:33:13.3158518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3158601Z outputs = self.model( 2025-08-26T20:33:13.3158871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3158952Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3159303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3159396Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3159645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3159733Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3160005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3160158Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3160430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3160598Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3160602Z 2025-08-26T20:33:13.3160715Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3160942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3161012Z return mod(**inputs) 2025-08-26T20:33:13.3161266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3161361Z outputs = self.model( 2025-08-26T20:33:13.3161617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3161700Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3161953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3162026Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3162259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3162341Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3162627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3162736Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3162998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3163078Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3163082Z 2025-08-26T20:33:13.3163188Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3163397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3163462Z return mod(**inputs) 2025-08-26T20:33:13.3163722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3163811Z outputs = self.model( 2025-08-26T20:33:13.3164063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3164167Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3164423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3164504Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3164727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3164814Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3165067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3165177Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3165435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3165524Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3165527Z 2025-08-26T20:33:13.3165615Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3165695Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3165774Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3165860Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3165966Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3166185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3166252Z return mod(**inputs) 2025-08-26T20:33:13.3166507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3166584Z outputs = self.model( 2025-08-26T20:33:13.3166838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3166920Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3167171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3167243Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3167489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3167570Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3167834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3167941Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3168202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3168299Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3168612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3168755Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3168759Z 2025-08-26T20:33:13.3168864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3169071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3169137Z return mod(**inputs) 2025-08-26T20:33:13.3169399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3169475Z outputs = self.model( 2025-08-26T20:33:13.3169732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3169813Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3170377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3170459Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3170683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3170762Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3171023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3171132Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3171392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3171490Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3171786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3171902Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3171906Z 2025-08-26T20:33:13.3172007Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3172212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3172279Z return mod(**inputs) 2025-08-26T20:33:13.3172533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3172639Z outputs = self.model( 2025-08-26T20:33:13.3172888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3172967Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3173213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3173293Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3173510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3173588Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3173860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3173966Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3174218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3174299Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3174302Z 2025-08-26T20:33:13.3174411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3174608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3174675Z return mod(**inputs) 2025-08-26T20:33:13.3175016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3175085Z outputs = self.model( 2025-08-26T20:33:13.3175348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3175421Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3175676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3175758Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3175984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3176070Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3176322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-26T20:33:13.3176423Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3176434Z 2025-08-26T20:33:13.3176539Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3176752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3176826Z return mod(**inputs) 2025-08-26T20:33:13.3177083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3177165Z outputs = self.model( 2025-08-26T20:33:13.3177425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3177499Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3177768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3177846Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3178085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3178166Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3178427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3178557Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3178561Z 2025-08-26T20:33:13.3178685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3178895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3178961Z return mod(**inputs) 2025-08-26T20:33:13.3179222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3179292Z outputs = self.model( 2025-08-26T20:33:13.3179552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3179633Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3179908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3179995Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3180222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3180302Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3180563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3180683Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3180686Z 2025-08-26T20:33:13.3180799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3181019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3181091Z return mod(**inputs) 2025-08-26T20:33:13.3181349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3181419Z outputs = self.model( 2025-08-26T20:33:13.3181678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3181752Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3182017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3182087Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3182304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3182406Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3182663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3182753Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3182756Z 2025-08-26T20:33:13.3182862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3183060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3183135Z return mod(**inputs) 2025-08-26T20:33:13.3183388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3183465Z outputs = self.model( 2025-08-26T20:33:13.3183723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3183804Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3184059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3184130Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3184360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3184439Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3184713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3184816Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3185068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3185229Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3185235Z 2025-08-26T20:33:13.3185340Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3185550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3185616Z return mod(**inputs) 2025-08-26T20:33:13.3185905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3185976Z outputs = self.model( 2025-08-26T20:33:13.3186229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3186312Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3186562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3186640Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3186861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3186962Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3187217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3187317Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3187582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3187664Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3187668Z 2025-08-26T20:33:13.3187779Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3187981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3188046Z return mod(**inputs) 2025-08-26T20:33:13.3188321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3188411Z outputs = self.model( 2025-08-26T20:33:13.3188686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3188764Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3189034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3189119Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3189355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3189443Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3189713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3189818Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3190093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3190187Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3190191Z 2025-08-26T20:33:13.3190283Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3190369Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3190459Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3190540Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3190648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3190884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3190956Z return mod(**inputs) 2025-08-26T20:33:13.3191242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3191312Z outputs = self.model( 2025-08-26T20:33:13.3191568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3191652Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3191905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3192000Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3192226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3192308Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3192583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3192689Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3192965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3193072Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3193411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3193556Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3193560Z 2025-08-26T20:33:13.3193672Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3193889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3193959Z return mod(**inputs) 2025-08-26T20:33:13.3194239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3194311Z outputs = self.model( 2025-08-26T20:33:13.3194577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3194680Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3194952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3195035Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3195277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3195360Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3195640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3195747Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3196025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3196127Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3196590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3196718Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3196722Z 2025-08-26T20:33:13.3196834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3197059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3197131Z return mod(**inputs) 2025-08-26T20:33:13.3197456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3197531Z outputs = self.model( 2025-08-26T20:33:13.3197797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3197883Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3198156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3198245Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3198485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3198578Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3198879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3198989Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3199325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3199426Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3202783Z 2025-08-26T20:33:13.3202913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3203149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3203227Z return mod(**inputs) 2025-08-26T20:33:13.3203512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3203588Z outputs = self.model( 2025-08-26T20:33:13.3203874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3203953Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3204224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3204309Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3204551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3204676Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3204955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3205115Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3205384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3205556Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3205560Z 2025-08-26T20:33:13.3205671Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3205896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3205967Z return mod(**inputs) 2025-08-26T20:33:13.3206244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3206322Z outputs = self.model( 2025-08-26T20:33:13.3206592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3206684Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3206954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3207039Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3207280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3207365Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3207662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3207782Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3208057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3208139Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3208143Z 2025-08-26T20:33:13.3208251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3208445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3208510Z return mod(**inputs) 2025-08-26T20:33:13.3208801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3208869Z outputs = self.model( 2025-08-26T20:33:13.3209121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3209192Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3209438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3209587Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3209808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3209893Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3210145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3210258Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3210508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3210593Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3210596Z 2025-08-26T20:33:13.3210685Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3210763Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3210847Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3210922Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3211050Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3211248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3211312Z return mod(**inputs) 2025-08-26T20:33:13.3211566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3211634Z outputs = self.model( 2025-08-26T20:33:13.3211879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3211955Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3212195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3212272Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3212488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3212565Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3212815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3212915Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3213166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3213259Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3213570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3213702Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3213707Z 2025-08-26T20:33:13.3213807Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3214003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3214068Z return mod(**inputs) 2025-08-26T20:33:13.3214315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3214381Z outputs = self.model( 2025-08-26T20:33:13.3214639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3214718Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3214957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3215033Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3215249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3215364Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3215611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3215717Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3215972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3216064Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3216355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3216462Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3216465Z 2025-08-26T20:33:13.3216574Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3216771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3216835Z return mod(**inputs) 2025-08-26T20:33:13.3217116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3217185Z outputs = self.model( 2025-08-26T20:33:13.3217448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3217524Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3217777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3217860Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3218084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3218172Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3218427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3218536Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3218808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3218890Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3218894Z 2025-08-26T20:33:13.3219006Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3219200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3219289Z return mod(**inputs) 2025-08-26T20:33:13.3219537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3219605Z outputs = self.model( 2025-08-26T20:33:13.3219862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3219938Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3220196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3220268Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3220508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3220600Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3220854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3220984Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3220988Z 2025-08-26T20:33:13.3221095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3221325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3221392Z return mod(**inputs) 2025-08-26T20:33:13.3221650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3221726Z outputs = self.model( 2025-08-26T20:33:13.3221988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3222065Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3222307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3222376Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3222597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3222673Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3222926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3223058Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3223062Z 2025-08-26T20:33:13.3223169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3223358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3223423Z return mod(**inputs) 2025-08-26T20:33:13.3223669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3223735Z outputs = self.model( 2025-08-26T20:33:13.3223983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3224051Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3224293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3224370Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3224585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3224670Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3224916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3224997Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3225000Z 2025-08-26T20:33:13.3225108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3225316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3225388Z return mod(**inputs) 2025-08-26T20:33:13.3225637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3225712Z outputs = self.model( 2025-08-26T20:33:13.3225961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3226032Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3226291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3226375Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3226594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3226671Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3226919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-26T20:33:13.3227005Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3227025Z 2025-08-26T20:33:13.3227128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3227331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3227396Z return mod(**inputs) 2025-08-26T20:33:13.3227652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3227727Z outputs = self.model( 2025-08-26T20:33:13.3227979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3228058Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3228310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3228386Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3228604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3228683Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3228964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3229064Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3229332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3229488Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3229492Z 2025-08-26T20:33:13.3229597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3229805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3229871Z return mod(**inputs) 2025-08-26T20:33:13.3230137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3230210Z outputs = self.model( 2025-08-26T20:33:13.3230478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3230550Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3230811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3230891Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3231116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3231220Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3231477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3231581Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3231841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3231924Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3231928Z 2025-08-26T20:33:13.3232039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3232244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3232334Z return mod(**inputs) 2025-08-26T20:33:13.3232599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3232667Z outputs = self.model( 2025-08-26T20:33:13.3232924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3232996Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3233270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3233342Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3233559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3233645Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3233893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3233995Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3234244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3234330Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3234340Z 2025-08-26T20:33:13.3234423Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3234508Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3234594Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3234697Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3234803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3235012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3235079Z return mod(**inputs) 2025-08-26T20:33:13.3235342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3235410Z outputs = self.model( 2025-08-26T20:33:13.3235672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3235746Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3236007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3236088Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3236305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3236390Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3236642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3236743Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3237009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3237125Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3237430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3237568Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3237573Z 2025-08-26T20:33:13.3237685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3237893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3237963Z return mod(**inputs) 2025-08-26T20:33:13.3238243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3238331Z outputs = self.model( 2025-08-26T20:33:13.3238612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3238690Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3238959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3239044Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3239389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3239486Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3239763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3239880Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3240163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3240268Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3240598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3240710Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3240714Z 2025-08-26T20:33:13.3240825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3241028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3241118Z return mod(**inputs) 2025-08-26T20:33:13.3241383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3241453Z outputs = self.model( 2025-08-26T20:33:13.3241721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3241795Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3242058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3242139Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3242360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3242457Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3242707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3242813Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3243062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3243150Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3243155Z 2025-08-26T20:33:13.3243255Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3243452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3243540Z return mod(**inputs) 2025-08-26T20:33:13.3243798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3243879Z outputs = self.model( 2025-08-26T20:33:13.3244150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3244235Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3244501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3244576Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3244836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3244922Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3245202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3245318Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3245588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3245785Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3245790Z 2025-08-26T20:33:13.3245897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3246106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3246172Z return mod(**inputs) 2025-08-26T20:33:13.3246446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3246518Z outputs = self.model( 2025-08-26T20:33:13.3246788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3246873Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3247144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3247228Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3247466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3247569Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3247845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3247962Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3248235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3248321Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3248325Z 2025-08-26T20:33:13.3248439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3248649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3248721Z return mod(**inputs) 2025-08-26T20:33:13.3248997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3249069Z outputs = self.model( 2025-08-26T20:33:13.3249347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3249423Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3249691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3249774Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3250028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3250120Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3250390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3250504Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3250779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3250869Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3250873Z 2025-08-26T20:33:13.3250966Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3251070Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3251161Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3251241Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3251352Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3251572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3251643Z return mod(**inputs) 2025-08-26T20:33:13.3251921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3252010Z outputs = self.model( 2025-08-26T20:33:13.3252282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3252366Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3252642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3252725Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3252962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3253048Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3253325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3253441Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3253721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3253843Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3254158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3254302Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3254307Z 2025-08-26T20:33:13.3254415Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3254638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3254709Z return mod(**inputs) 2025-08-26T20:33:13.3254986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3255059Z outputs = self.model( 2025-08-26T20:33:13.3255341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3255438Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3255696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3255776Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3256002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3256089Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3256360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3256469Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3256728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3256838Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3257129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3257235Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3257239Z 2025-08-26T20:33:13.3257356Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3257558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3257622Z return mod(**inputs) 2025-08-26T20:33:13.3257880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3257948Z outputs = self.model( 2025-08-26T20:33:13.3258202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3258305Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3258556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3258635Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3258855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3258941Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3259188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3259293Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3259548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3259632Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3259636Z 2025-08-26T20:33:13.3259742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3259959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3260030Z return mod(**inputs) 2025-08-26T20:33:13.3260296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3260370Z outputs = self.model( 2025-08-26T20:33:13.3260635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3260712Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3260976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3261053Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3261283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3261375Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3261634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3261769Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3261772Z 2025-08-26T20:33:13.3261882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3262087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3262164Z return mod(**inputs) 2025-08-26T20:33:13.3262452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3262527Z outputs = self.model( 2025-08-26T20:33:13.3262777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3262856Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3263108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3263177Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3263403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3263498Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3263753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3263871Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3263875Z 2025-08-26T20:33:13.3263975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3264177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3264260Z return mod(**inputs) 2025-08-26T20:33:13.3264514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3264583Z outputs = self.model( 2025-08-26T20:33:13.3264836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3264911Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3265167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3265246Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3265472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3265558Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3265813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3265915Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3265918Z 2025-08-26T20:33:13.3266029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3266230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3266308Z return mod(**inputs) 2025-08-26T20:33:13.3266578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3266651Z outputs = self.model( 2025-08-26T20:33:13.3266931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3267003Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3267260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3267335Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3267564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3267644Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3267896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3268006Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3268267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3268435Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3268440Z 2025-08-26T20:33:13.3268542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3268737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3268811Z return mod(**inputs) 2025-08-26T20:33:13.3269063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3269137Z outputs = self.model( 2025-08-26T20:33:13.3269386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3269480Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3269731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3269803Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3270031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3270109Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3270380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3270481Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3270726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3270812Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3270816Z 2025-08-26T20:33:13.3270918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3271119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3271194Z return mod(**inputs) 2025-08-26T20:33:13.3271446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3271510Z outputs = self.model( 2025-08-26T20:33:13.3271754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3271834Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3272114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3272192Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3272413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3272489Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3272744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3272842Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3273097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3273184Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3273188Z 2025-08-26T20:33:13.3273273Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3273356Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3273431Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3273513Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3273613Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3273810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3273882Z return mod(**inputs) 2025-08-26T20:33:13.3274144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3274220Z outputs = self.model( 2025-08-26T20:33:13.3274466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3274545Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3274794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3274868Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3275098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3275177Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3275461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3275560Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3275808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3275911Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3276195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3276355Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3276359Z 2025-08-26T20:33:13.3276461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3276668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3276734Z return mod(**inputs) 2025-08-26T20:33:13.3276990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3277066Z outputs = self.model( 2025-08-26T20:33:13.3277323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3277404Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3277656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3277729Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3277980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3278060Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3278318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3278419Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3278678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3278777Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3279068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3279186Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3279190Z 2025-08-26T20:33:13.3279379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3279592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3279659Z return mod(**inputs) 2025-08-26T20:33:13.3279913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3279995Z outputs = self.model( 2025-08-26T20:33:13.3280248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3280358Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3280607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3280679Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3280906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3280985Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3281241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3281339Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3281617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3281700Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3281703Z 2025-08-26T20:33:13.3281806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3282010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3282076Z return mod(**inputs) 2025-08-26T20:33:13.3282351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3282421Z outputs = self.model( 2025-08-26T20:33:13.3282672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3282750Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3283002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3283082Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3283307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3283394Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3283642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-26T20:33:13.3283725Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3283729Z 2025-08-26T20:33:13.3283840Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3284052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3284126Z return mod(**inputs) 2025-08-26T20:33:13.3284374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3284442Z outputs = self.model( 2025-08-26T20:33:13.3284696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3284768Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3285018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3285090Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3285310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3285398Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3285651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3285768Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3286027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3286180Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3286184Z 2025-08-26T20:33:13.3286299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3286495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3286568Z return mod(**inputs) 2025-08-26T20:33:13.3286819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3286894Z outputs = self.model( 2025-08-26T20:33:13.3287144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3287214Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3287485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3287558Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3287784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3287860Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3288110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3288235Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3288482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3288574Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3288578Z 2025-08-26T20:33:13.3288684Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3288890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3288960Z return mod(**inputs) 2025-08-26T20:33:13.3289221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3289300Z outputs = self.model( 2025-08-26T20:33:13.3289566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3289656Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3289928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3290032Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3290269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3290352Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3290630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3290754Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3291015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3291102Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3291105Z 2025-08-26T20:33:13.3291188Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3291275Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3291353Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3291435Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3291549Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3291741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3291813Z return mod(**inputs) 2025-08-26T20:33:13.3292062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3292135Z outputs = self.model( 2025-08-26T20:33:13.3292402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3292483Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3292748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3292825Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3293069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3293151Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3293427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3293565Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3293835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3293949Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3294259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3294429Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3294433Z 2025-08-26T20:33:13.3294545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3294764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3294833Z return mod(**inputs) 2025-08-26T20:33:13.3295108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3295189Z outputs = self.model( 2025-08-26T20:33:13.3295460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3295545Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3295816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3295894Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3296134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3296381Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3296662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3296778Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3297053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3297166Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3297480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3297608Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3297614Z 2025-08-26T20:33:13.3297725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3297946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3298019Z return mod(**inputs) 2025-08-26T20:33:13.3298291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3298372Z outputs = self.model( 2025-08-26T20:33:13.3298645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3298733Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3299052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3299132Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3299378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3299466Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3299744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3299858Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3300132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3300248Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3300253Z 2025-08-26T20:33:13.3300365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3300588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3300659Z return mod(**inputs) 2025-08-26T20:33:13.3300941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3301054Z outputs = self.model( 2025-08-26T20:33:13.3301324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3301412Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3301683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3301769Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3302014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3302107Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3302360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3302481Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3302486Z 2025-08-26T20:33:13.3302596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3302795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3302896Z return mod(**inputs) 2025-08-26T20:33:13.3303151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3303219Z outputs = self.model( 2025-08-26T20:33:13.3303479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3303553Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3303813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3303884Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3304109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3304197Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3304451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3304577Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3304581Z 2025-08-26T20:33:13.3304687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3304895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3304962Z return mod(**inputs) 2025-08-26T20:33:13.3305236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3305314Z outputs = self.model( 2025-08-26T20:33:13.3305571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3305654Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3305911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3305985Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3306218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3306300Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3306581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3306666Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3306672Z 2025-08-26T20:33:13.3306781Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3306982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3307086Z return mod(**inputs) 2025-08-26T20:33:13.3307355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3307425Z outputs = self.model( 2025-08-26T20:33:13.3307691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3307764Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3308022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3308100Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3308329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3308414Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3308671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3308773Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3309051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3309201Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3309205Z 2025-08-26T20:33:13.3309318Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3309522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3309594Z return mod(**inputs) 2025-08-26T20:33:13.3309849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3309918Z outputs = self.model( 2025-08-26T20:33:13.3310192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3310271Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3310549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3310624Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3310859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3310953Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3311222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3311354Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3311627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3311719Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3311725Z 2025-08-26T20:33:13.3311835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3312052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3312127Z return mod(**inputs) 2025-08-26T20:33:13.3312381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3312458Z outputs = self.model( 2025-08-26T20:33:13.3312727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3312802Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3313069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3313142Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3313374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3313471Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3313731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3313830Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3314083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3314178Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3314182Z 2025-08-26T20:33:13.3314264Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3314354Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3314434Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3314511Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3314624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3314840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3314936Z return mod(**inputs) 2025-08-26T20:33:13.3315210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3315283Z outputs = self.model( 2025-08-26T20:33:13.3315561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3315638Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3315922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3315997Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3316220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3316308Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3316560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3316668Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3316918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3317021Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3317317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3317472Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3317476Z 2025-08-26T20:33:13.3317588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3317799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3317877Z return mod(**inputs) 2025-08-26T20:33:13.3318152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3318226Z outputs = self.model( 2025-08-26T20:33:13.3318505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3318583Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3318876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3318957Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3319209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3319361Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3319644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3319784Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3320063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3320186Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3320496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3320613Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3320624Z 2025-08-26T20:33:13.3320734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3320947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3321028Z return mod(**inputs) 2025-08-26T20:33:13.3321299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3321382Z outputs = self.model( 2025-08-26T20:33:13.3321674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3321751Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3322031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3322109Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3322351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3322438Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3322707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3322822Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3323093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3323191Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3323194Z 2025-08-26T20:33:13.3323305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3323523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3323593Z return mod(**inputs) 2025-08-26T20:33:13.3323863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3323943Z outputs = self.model( 2025-08-26T20:33:13.3324231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3324317Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3324590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3324669Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3324914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3324997Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3325288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3325406Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3325673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3325837Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3325841Z 2025-08-26T20:33:13.3325949Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3326188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3326259Z return mod(**inputs) 2025-08-26T20:33:13.3326536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3326607Z outputs = self.model( 2025-08-26T20:33:13.3326878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3326962Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3327228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3327305Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3327521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3327601Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3327850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3327983Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3328233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3328313Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3328318Z 2025-08-26T20:33:13.3328428Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3328622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3328688Z return mod(**inputs) 2025-08-26T20:33:13.3328945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3329011Z outputs = self.model( 2025-08-26T20:33:13.3329270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3329342Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3329587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3329672Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3329907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3329999Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3330292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3330406Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3330660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3330747Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3330752Z 2025-08-26T20:33:13.3330842Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3330923Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3331009Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3331087Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3331190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3331415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3331483Z return mod(**inputs) 2025-08-26T20:33:13.3331756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3331823Z outputs = self.model( 2025-08-26T20:33:13.3332071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3332165Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3332415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3332492Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3332708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3332786Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3333041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3333148Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3333402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3333495Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3333788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3333943Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3333946Z 2025-08-26T20:33:13.3334047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3334251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3334319Z return mod(**inputs) 2025-08-26T20:33:13.3334573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3334643Z outputs = self.model( 2025-08-26T20:33:13.3334887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3334966Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3335214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3335294Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3335507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3335592Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3335838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3335944Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3336214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3336310Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3336604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3336712Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3336717Z 2025-08-26T20:33:13.3336825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3337019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3337083Z return mod(**inputs) 2025-08-26T20:33:13.3337348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3337418Z outputs = self.model( 2025-08-26T20:33:13.3337670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3337740Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3337987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3338080Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3338297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3338386Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3338630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3338735Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3338987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3339069Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3339072Z 2025-08-26T20:33:13.3339180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3339375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3339445Z return mod(**inputs) 2025-08-26T20:33:13.3339693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3339777Z outputs = self.model( 2025-08-26T20:33:13.3340034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3340107Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3340373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3340445Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3340683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3340766Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3341012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-26T20:33:13.3341101Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3341107Z 2025-08-26T20:33:13.3341208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3341412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3341478Z return mod(**inputs) 2025-08-26T20:33:13.3341737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3341814Z outputs = self.model( 2025-08-26T20:33:13.3342095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3342175Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3342424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3342496Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3342722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3342801Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3343054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3343187Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3343192Z 2025-08-26T20:33:13.3343295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3343502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3343570Z return mod(**inputs) 2025-08-26T20:33:13.3343824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3343907Z outputs = self.model( 2025-08-26T20:33:13.3344161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3344236Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3344481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3344560Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3344779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3344864Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3345111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3345227Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3345230Z 2025-08-26T20:33:13.3345340Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3345533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3345622Z return mod(**inputs) 2025-08-26T20:33:13.3345870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3345941Z outputs = self.model( 2025-08-26T20:33:13.3346188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3346259Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3346513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3346583Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3346805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3346885Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3347139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3347230Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3347234Z 2025-08-26T20:33:13.3347337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3347545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3347611Z return mod(**inputs) 2025-08-26T20:33:13.3347864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3347956Z outputs = self.model( 2025-08-26T20:33:13.3348215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3348297Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3348554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3348639Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3348877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3348961Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3349258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3349368Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3349641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3349798Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3349802Z 2025-08-26T20:33:13.3349937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3350155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3350230Z return mod(**inputs) 2025-08-26T20:33:13.3350513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3350589Z outputs = self.model( 2025-08-26T20:33:13.3350869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3350949Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3351223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3351309Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3351552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3351646Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3351922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3352055Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3352318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3352404Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3352408Z 2025-08-26T20:33:13.3352519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3352724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3352799Z return mod(**inputs) 2025-08-26T20:33:13.3353057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3353129Z outputs = self.model( 2025-08-26T20:33:13.3353396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3353473Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3353735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3353809Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3354037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3354127Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3354409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3354515Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3354771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3354861Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3354872Z 2025-08-26T20:33:13.3354960Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3355046Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3355137Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3355216Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3355339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3355573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3355639Z return mod(**inputs) 2025-08-26T20:33:13.3355904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3355972Z outputs = self.model( 2025-08-26T20:33:13.3356235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3356331Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3356595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3356676Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3356915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3357010Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3357288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3357397Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3357680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3357783Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3358107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3358266Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3358270Z 2025-08-26T20:33:13.3358387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3358602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3358673Z return mod(**inputs) 2025-08-26T20:33:13.3358952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3359028Z outputs = self.model( 2025-08-26T20:33:13.3359396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3359486Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3359763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3359853Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3360099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3360197Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3360480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3360606Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3360897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3361001Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3361325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3361446Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3361451Z 2025-08-26T20:33:13.3361576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3361789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3361859Z return mod(**inputs) 2025-08-26T20:33:13.3362156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3362229Z outputs = self.model( 2025-08-26T20:33:13.3362511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3362589Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3362857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3362958Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3363207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3363294Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3363546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3363655Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3363913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3363997Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3364001Z 2025-08-26T20:33:13.3364110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3364312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3364387Z return mod(**inputs) 2025-08-26T20:33:13.3364645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3364734Z outputs = self.model( 2025-08-26T20:33:13.3364992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3365064Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3365326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3365399Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3365629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3365707Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3365959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3366078Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3366330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3366487Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3366491Z 2025-08-26T20:33:13.3366595Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3366795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3366870Z return mod(**inputs) 2025-08-26T20:33:13.3367136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3367213Z outputs = self.model( 2025-08-26T20:33:13.3367466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3367547Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3367802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3367871Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3368116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3368195Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3368457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3368570Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3368841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3368970Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3368973Z 2025-08-26T20:33:13.3369084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3369302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3369372Z return mod(**inputs) 2025-08-26T20:33:13.3369647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3369721Z outputs = self.model( 2025-08-26T20:33:13.3369988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3370073Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3370342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3370425Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3370661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3370763Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3371039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3371152Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3371427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3371519Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3371523Z 2025-08-26T20:33:13.3371619Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3371705Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3371788Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3371877Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3371988Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3372199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3372276Z return mod(**inputs) 2025-08-26T20:33:13.3372549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3372629Z outputs = self.model( 2025-08-26T20:33:13.3372898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3372982Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3373270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3373348Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3373591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3373677Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3373950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3374068Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3374335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3374468Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3374783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3374932Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3374936Z 2025-08-26T20:33:13.3375044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3375264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3375355Z return mod(**inputs) 2025-08-26T20:33:13.3375634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3375716Z outputs = self.model( 2025-08-26T20:33:13.3375990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3376076Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3376351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3376429Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3376677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3376760Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3377043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3377176Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3377448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3377549Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3377858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3377981Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3377985Z 2025-08-26T20:33:13.3378097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3378314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3378385Z return mod(**inputs) 2025-08-26T20:33:13.3378657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3378738Z outputs = self.model( 2025-08-26T20:33:13.3379006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3379092Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3379361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3379454Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3379693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3379774Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3380032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3380141Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3380400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3380485Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3380488Z 2025-08-26T20:33:13.3380592Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3380815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3380883Z return mod(**inputs) 2025-08-26T20:33:13.3381145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3381213Z outputs = self.model( 2025-08-26T20:33:13.3381463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3381562Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3381814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3381895Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3382128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3382217Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3382485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3382614Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3382617Z 2025-08-26T20:33:13.3382735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3382943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3383020Z return mod(**inputs) 2025-08-26T20:33:13.3383290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3383387Z outputs = self.model( 2025-08-26T20:33:13.3383649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3383722Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3383981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3384052Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3384285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3384362Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3384612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3384741Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3384744Z 2025-08-26T20:33:13.3384849Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3385054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3385119Z return mod(**inputs) 2025-08-26T20:33:13.3385372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3385448Z outputs = self.model( 2025-08-26T20:33:13.3385705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3385801Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3386046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3386116Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3386341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3386419Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3386670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3386751Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3386754Z 2025-08-26T20:33:13.3386876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3387073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3387138Z return mod(**inputs) 2025-08-26T20:33:13.3387396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3387461Z outputs = self.model( 2025-08-26T20:33:13.3387738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3387812Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3388066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3388145Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3388370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3388458Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3388712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-26T20:33:13.3388802Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3388806Z 2025-08-26T20:33:13.3388920Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3389115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3389186Z return mod(**inputs) 2025-08-26T20:33:13.3389453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3389530Z outputs = self.model( 2025-08-26T20:33:13.3389783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3389857Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3390119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3390192Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3390422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3390503Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3390755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3390866Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3391120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3391277Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3391282Z 2025-08-26T20:33:13.3391385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3391593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3391675Z return mod(**inputs) 2025-08-26T20:33:13.3391929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3392007Z outputs = self.model( 2025-08-26T20:33:13.3392265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3392348Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3392613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3392682Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3392922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3393001Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3393256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3393353Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3393606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3393703Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3393708Z 2025-08-26T20:33:13.3393809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3394018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3394085Z return mod(**inputs) 2025-08-26T20:33:13.3394350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3394428Z outputs = self.model( 2025-08-26T20:33:13.3394676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3394756Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3395004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3395084Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3395306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3395407Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3395658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3395758Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3396020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3396108Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3396112Z 2025-08-26T20:33:13.3396323Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3396414Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3396494Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3396581Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3396689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3396897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3396965Z return mod(**inputs) 2025-08-26T20:33:13.3397225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3397302Z outputs = self.model( 2025-08-26T20:33:13.3397557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3397640Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3397944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3398019Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3398250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3398330Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3398590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3398691Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3398978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3399075Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3399437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3399597Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3399602Z 2025-08-26T20:33:13.3399715Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3399970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3400042Z return mod(**inputs) 2025-08-26T20:33:13.3400327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3400407Z outputs = self.model( 2025-08-26T20:33:13.3400685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3400773Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3401048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3401137Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3401385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3401465Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3401728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3401852Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3402115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3402212Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3402512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3402629Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3402634Z 2025-08-26T20:33:13.3402740Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3402947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3403015Z return mod(**inputs) 2025-08-26T20:33:13.3403284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3403353Z outputs = self.model( 2025-08-26T20:33:13.3403609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3403692Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3403950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3404032Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3404275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3404356Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3404613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3404716Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3404976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3405058Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3405062Z 2025-08-26T20:33:13.3405175Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3405389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3405458Z return mod(**inputs) 2025-08-26T20:33:13.3405723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3405791Z outputs = self.model( 2025-08-26T20:33:13.3406052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3406145Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3406403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3406483Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3406708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3406793Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3407050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3407160Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3407425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3407575Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3407580Z 2025-08-26T20:33:13.3407690Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3407907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3407980Z return mod(**inputs) 2025-08-26T20:33:13.3408235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3408302Z outputs = self.model( 2025-08-26T20:33:13.3408569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3408640Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3408902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3408974Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3409199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3409291Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3409563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3409685Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3409957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3410044Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3410048Z 2025-08-26T20:33:13.3410153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3410373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3410450Z return mod(**inputs) 2025-08-26T20:33:13.3410703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3410781Z outputs = self.model( 2025-08-26T20:33:13.3411037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3411111Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3411371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3411459Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3411689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3411769Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3412027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3412134Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3412691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3412788Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3412792Z 2025-08-26T20:33:13.3412872Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3412958Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3413037Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3413115Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3413226Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3413427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3413502Z return mod(**inputs) 2025-08-26T20:33:13.3413756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3413826Z outputs = self.model( 2025-08-26T20:33:13.3414091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3414182Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3414450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3414522Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3414750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3414839Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3415096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3415214Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3415472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3415578Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3415875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3416008Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3416011Z 2025-08-26T20:33:13.3416122Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3416326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3416398Z return mod(**inputs) 2025-08-26T20:33:13.3416674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3416745Z outputs = self.model( 2025-08-26T20:33:13.3417005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3417082Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3417347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3417418Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3417645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3417739Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3417987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3418101Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3418347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3418452Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3418761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3418872Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3418883Z 2025-08-26T20:33:13.3418987Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3419190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3419267Z return mod(**inputs) 2025-08-26T20:33:13.3419523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3419598Z outputs = self.model( 2025-08-26T20:33:13.3419866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3419943Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3420224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3420328Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3420574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3420658Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3420929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3421051Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3421320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3421414Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3421418Z 2025-08-26T20:33:13.3421533Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3421760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3421828Z return mod(**inputs) 2025-08-26T20:33:13.3422084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3422159Z outputs = self.model( 2025-08-26T20:33:13.3422418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3422499Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3422752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3422847Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3423080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3423159Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3423421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3423541Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3423545Z 2025-08-26T20:33:13.3423656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3423855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3423937Z return mod(**inputs) 2025-08-26T20:33:13.3424201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3424272Z outputs = self.model( 2025-08-26T20:33:13.3424529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3424604Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3424873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3424956Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3425175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3425261Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3425514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3425635Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3425645Z 2025-08-26T20:33:13.3425752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3425952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3426024Z return mod(**inputs) 2025-08-26T20:33:13.3426277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3426354Z outputs = self.model( 2025-08-26T20:33:13.3426625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3426699Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3426959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3427030Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3427258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3427339Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3427589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3427680Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3427683Z 2025-08-26T20:33:13.3427788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3427996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3428062Z return mod(**inputs) 2025-08-26T20:33:13.3428320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3428390Z outputs = self.model( 2025-08-26T20:33:13.3428642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3428722Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3428992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3429073Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3429298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3429380Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3429651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3429748Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3430017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3430166Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3430169Z 2025-08-26T20:33:13.3430279Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3430479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3430545Z return mod(**inputs) 2025-08-26T20:33:13.3430823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3430891Z outputs = self.model( 2025-08-26T20:33:13.3431150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3431223Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3431479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3431559Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3431781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3431869Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3432119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3432219Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3432478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3432579Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3432583Z 2025-08-26T20:33:13.3432693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3432895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3432968Z return mod(**inputs) 2025-08-26T20:33:13.3433223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3433293Z outputs = self.model( 2025-08-26T20:33:13.3433558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3433631Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3433897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3433972Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3434196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3434281Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3434541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3434646Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3434922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3435019Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3435022Z 2025-08-26T20:33:13.3435104Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3435186Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3435273Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3435352Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3435461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3435662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3435729Z return mod(**inputs) 2025-08-26T20:33:13.3436003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3436073Z outputs = self.model( 2025-08-26T20:33:13.3436337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3436410Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3436667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3436763Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3436989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3437077Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3437335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3437437Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3437699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3437804Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3438119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3438261Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3438267Z 2025-08-26T20:33:13.3438385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3438615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3438688Z return mod(**inputs) 2025-08-26T20:33:13.3438969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3439044Z outputs = self.model( 2025-08-26T20:33:13.3439400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3439487Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3439759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3439843Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3440082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3440177Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3440448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3440562Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3440831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3440934Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3441274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3441392Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3441396Z 2025-08-26T20:33:13.3441514Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3441729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3441802Z return mod(**inputs) 2025-08-26T20:33:13.3442082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3442155Z outputs = self.model( 2025-08-26T20:33:13.3442454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3442530Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3442794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3442867Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3443094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3443206Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3443459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3443570Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3443839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3443928Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3443932Z 2025-08-26T20:33:13.3444049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3444262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3444337Z return mod(**inputs) 2025-08-26T20:33:13.3444603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3444684Z outputs = self.model( 2025-08-26T20:33:13.3444952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3445051Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3445327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3445403Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3445649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3445734Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3446007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-26T20:33:13.3446100Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3446104Z 2025-08-26T20:33:13.3446215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3446438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3446508Z return mod(**inputs) 2025-08-26T20:33:13.3446776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3446855Z outputs = self.model( 2025-08-26T20:33:13.3447128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3447213Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3447499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3447583Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3447821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3447907Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3448185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3448304Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3448581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3448767Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3448771Z 2025-08-26T20:33:13.3448884Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3449087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3449154Z return mod(**inputs) 2025-08-26T20:33:13.3449419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3449511Z outputs = self.model( 2025-08-26T20:33:13.3449788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3449869Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3450143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3450223Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3450447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3450533Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3450804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3450921Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3451200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3451287Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3451313Z 2025-08-26T20:33:13.3451431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3451640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3451716Z return mod(**inputs) 2025-08-26T20:33:13.3451984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3452057Z outputs = self.model( 2025-08-26T20:33:13.3452331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3452408Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3452681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3452758Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3452994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3453084Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3453349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3453472Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3453738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3453850Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3453862Z 2025-08-26T20:33:13.3453950Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3454035Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3454129Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3454212Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3454331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3454548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3454618Z return mod(**inputs) 2025-08-26T20:33:13.3454898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3454991Z outputs = self.model( 2025-08-26T20:33:13.3455271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3455350Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3455619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3455704Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3455967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3456059Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3456329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3456443Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3456724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3456827Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3457150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3457290Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3457294Z 2025-08-26T20:33:13.3457413Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3457626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3457714Z return mod(**inputs) 2025-08-26T20:33:13.3457992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3458063Z outputs = self.model( 2025-08-26T20:33:13.3458341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3458419Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3458689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3458772Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3459007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3459099Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3459369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3459490Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3459767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3459865Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3460166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3460289Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3460293Z 2025-08-26T20:33:13.3460407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3460608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3460674Z return mod(**inputs) 2025-08-26T20:33:13.3460935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3461005Z outputs = self.model( 2025-08-26T20:33:13.3461265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3461355Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3461617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3461688Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3461913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3462001Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3462271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3462387Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3462640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3462723Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3462727Z 2025-08-26T20:33:13.3462841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3463042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3463117Z return mod(**inputs) 2025-08-26T20:33:13.3463373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3463441Z outputs = self.model( 2025-08-26T20:33:13.3463702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3463776Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3464057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3464128Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3464364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3464446Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3464707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3464837Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3464841Z 2025-08-26T20:33:13.3464946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3465157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3465225Z return mod(**inputs) 2025-08-26T20:33:13.3465490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3465567Z outputs = self.model( 2025-08-26T20:33:13.3465828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3465909Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3466172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3466249Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3466493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3466573Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3466841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3466962Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3466966Z 2025-08-26T20:33:13.3467076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3467276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3467341Z return mod(**inputs) 2025-08-26T20:33:13.3467652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3467724Z outputs = self.model( 2025-08-26T20:33:13.3467986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3468061Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3468322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3468411Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3468640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3468727Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3468983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3469072Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3469076Z 2025-08-26T20:33:13.3469181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3469385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3469461Z return mod(**inputs) 2025-08-26T20:33:13.3469716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3469794Z outputs = self.model( 2025-08-26T20:33:13.3470050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3470144Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3470407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3470481Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3470713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3470794Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3471057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3471157Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3471415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3471576Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3471579Z 2025-08-26T20:33:13.3471683Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3471891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3471956Z return mod(**inputs) 2025-08-26T20:33:13.3472213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3472290Z outputs = self.model( 2025-08-26T20:33:13.3472557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3472639Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3472891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3472972Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3473194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3473274Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3473551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3473653Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3473923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3474003Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3474007Z 2025-08-26T20:33:13.3474110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3474319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3474401Z return mod(**inputs) 2025-08-26T20:33:13.3474665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3474733Z outputs = self.model( 2025-08-26T20:33:13.3474993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3475069Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3475325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3475405Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3475628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3475713Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3475970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3476089Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3476348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3476436Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3476440Z 2025-08-26T20:33:13.3476530Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3476612Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3476692Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3476776Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3476882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3477103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3477174Z return mod(**inputs) 2025-08-26T20:33:13.3477443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3477524Z outputs = self.model( 2025-08-26T20:33:13.3477792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3477879Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3478147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3478229Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3478482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3478568Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3478847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3478953Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3479303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3479418Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3479730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3479903Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3479907Z 2025-08-26T20:33:13.3480019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3480242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3480313Z return mod(**inputs) 2025-08-26T20:33:13.3480597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3480690Z outputs = self.model( 2025-08-26T20:33:13.3480965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3481052Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3481326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3481421Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3481664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3481745Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3482025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3482131Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3482408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3482528Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3482847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3482964Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3482968Z 2025-08-26T20:33:13.3483079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3483296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3483369Z return mod(**inputs) 2025-08-26T20:33:13.3483660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3483732Z outputs = self.model( 2025-08-26T20:33:13.3484000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3484085Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3484353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3484437Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3484676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3484758Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3485051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3485159Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3485435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3485523Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3485527Z 2025-08-26T20:33:13.3485646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3485859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3485928Z return mod(**inputs) 2025-08-26T20:33:13.3486221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3486295Z outputs = self.model( 2025-08-26T20:33:13.3486572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3486652Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3486919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3487002Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3487258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3487351Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3487621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3487744Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3488015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3488181Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3488186Z 2025-08-26T20:33:13.3488297Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3488497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3488571Z return mod(**inputs) 2025-08-26T20:33:13.3488825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3488911Z outputs = self.model( 2025-08-26T20:33:13.3489173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3489245Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3489509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3489581Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3489812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3489891Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3490146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3490263Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3490519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3490606Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3490610Z 2025-08-26T20:33:13.3490712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3490914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3490986Z return mod(**inputs) 2025-08-26T20:33:13.3491262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3491339Z outputs = self.model( 2025-08-26T20:33:13.3491599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3491680Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3491930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3492001Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3492229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3492305Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3492575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3492684Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3492942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3493038Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3493058Z 2025-08-26T20:33:13.3493141Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3493230Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3493308Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3493384Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3493495Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3493699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3493774Z return mod(**inputs) 2025-08-26T20:33:13.3494029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3494099Z outputs = self.model( 2025-08-26T20:33:13.3494361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3494435Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3494706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3494796Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3495022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3495097Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3495343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3495458Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3495702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3495801Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3496085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3496385Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3496402Z 2025-08-26T20:33:13.3496635Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3496834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3496908Z return mod(**inputs) 2025-08-26T20:33:13.3497156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3497231Z outputs = self.model( 2025-08-26T20:33:13.3497521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3497595Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3497849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3497922Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3498148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3498229Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3498481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3498597Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3498875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3498982Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3499277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3499394Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3499423Z 2025-08-26T20:33:13.3499528Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3499729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3499805Z return mod(**inputs) 2025-08-26T20:33:13.3500070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3500145Z outputs = self.model( 2025-08-26T20:33:13.3500391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3500461Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3500716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3500786Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3501010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3501087Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3501373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3501478Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3501725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3501814Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3501818Z 2025-08-26T20:33:13.3501919Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3502124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3502188Z return mod(**inputs) 2025-08-26T20:33:13.3502443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3502517Z outputs = self.model( 2025-08-26T20:33:13.3502771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3502848Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3503100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3503177Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3503398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3503473Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3503747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-26T20:33:13.3503828Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3503832Z 2025-08-26T20:33:13.3503939Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3504131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3504196Z return mod(**inputs) 2025-08-26T20:33:13.3504450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3504516Z outputs = self.model( 2025-08-26T20:33:13.3504780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3504854Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3505102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3505179Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3505397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3505505Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3505752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3505875Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3505878Z 2025-08-26T20:33:13.3505979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3506179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3506254Z return mod(**inputs) 2025-08-26T20:33:13.3506511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3506590Z outputs = self.model( 2025-08-26T20:33:13.3506842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3506917Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3507177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3507266Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3507501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3507582Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3507851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3507972Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3507975Z 2025-08-26T20:33:13.3508079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3508288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3508356Z return mod(**inputs) 2025-08-26T20:33:13.3508624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3508693Z outputs = self.model( 2025-08-26T20:33:13.3508950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3509029Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3509289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3509368Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3509614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3509704Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3509955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3510038Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3510043Z 2025-08-26T20:33:13.3510151Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3510352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3510424Z return mod(**inputs) 2025-08-26T20:33:13.3510695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3510766Z outputs = self.model( 2025-08-26T20:33:13.3511032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3511105Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3511366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3511455Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3511681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3511772Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3512026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3512133Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3512388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3512550Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3512553Z 2025-08-26T20:33:13.3512656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3512857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3512931Z return mod(**inputs) 2025-08-26T20:33:13.3513187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3513300Z outputs = self.model( 2025-08-26T20:33:13.3513559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3513633Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3513900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3513972Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3514209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3514289Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3514556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3514665Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3514940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3515034Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3515037Z 2025-08-26T20:33:13.3515146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3515369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3515437Z return mod(**inputs) 2025-08-26T20:33:13.3515729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3515813Z outputs = self.model( 2025-08-26T20:33:13.3516092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3516183Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3516473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3516557Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3516798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3516898Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3517177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3517283Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3517555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3517646Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3517667Z 2025-08-26T20:33:13.3517754Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3517848Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3517932Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3518020Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3518128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3518339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3518416Z return mod(**inputs) 2025-08-26T20:33:13.3518688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3518768Z outputs = self.model( 2025-08-26T20:33:13.3519037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3519115Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3519470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3519572Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3519823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3519910Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3520229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3520336Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3520616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3520729Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3521039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3521194Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3521199Z 2025-08-26T20:33:13.3521309Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3521558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3521637Z return mod(**inputs) 2025-08-26T20:33:13.3521911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3521989Z outputs = self.model( 2025-08-26T20:33:13.3522264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3522350Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3522610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3522686Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3522923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3523004Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3523268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3523389Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3523643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3523749Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3524041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3524160Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3524180Z 2025-08-26T20:33:13.3524285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3524494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3524572Z return mod(**inputs) 2025-08-26T20:33:13.3524824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3524899Z outputs = self.model( 2025-08-26T20:33:13.3525199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3525281Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3525531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3525601Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3525828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3525907Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3526180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-26T20:33:13.3526275Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:13.3526526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3526615Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3526619Z 2025-08-26T20:33:13.3526720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3526924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3526988Z return mod(**inputs) 2025-08-26T20:33:13.3527243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3527311Z outputs = self.model( 2025-08-26T20:33:13.3527557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3527636Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3527882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3527960Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3528176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3528270Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3528528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3528637Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3528892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-26T20:33:13.3529041Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:13.3529044Z 2025-08-26T20:33:13.3529150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3529362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3529429Z return mod(**inputs) 2025-08-26T20:33:13.3529691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3529760Z outputs = self.model( 2025-08-26T20:33:13.3530016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3530087Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3530350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3530430Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3530650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3530741Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3531010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3531133Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3531406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-26T20:33:13.3531487Z key_states = self.k_proj(current_states) 2025-08-26T20:33:13.3531490Z 2025-08-26T20:33:13.3531602Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3531806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3531904Z return mod(**inputs) 2025-08-26T20:33:13.3532174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3532245Z outputs = self.model( 2025-08-26T20:33:13.3532522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3532599Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3532872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3532944Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3533174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3533252Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3533513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3533626Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3533872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-26T20:33:13.3533963Z value_states = self.v_proj(current_states) 2025-08-26T20:33:13.3533967Z 2025-08-26T20:33:13.3534046Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3534123Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3534205Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3534298Z cudagraph partition due to non gpu ops 2025-08-26T20:33:13.3534409Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3534605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3534671Z return mod(**inputs) 2025-08-26T20:33:13.3534929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3534997Z outputs = self.model( 2025-08-26T20:33:13.3535259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3535330Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3535594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3535675Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3535895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3535979Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3536226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3536364Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3536611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3536706Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3536998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:13.3537128Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:13.3537131Z 2025-08-26T20:33:13.3537239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3537434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3537499Z return mod(**inputs) 2025-08-26T20:33:13.3537756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3537840Z outputs = self.model( 2025-08-26T20:33:13.3538092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3538165Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3538430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3538501Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3538721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3538809Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3539068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3539185Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3539447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-26T20:33:13.3539545Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:13.3539839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:13.3539947Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:13.3539951Z 2025-08-26T20:33:13.3540062Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3540278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3540355Z return mod(**inputs) 2025-08-26T20:33:13.3540611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3540684Z outputs = self.model( 2025-08-26T20:33:13.3540966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3541045Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3541318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3541393Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3541644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3541738Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3542008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-26T20:33:13.3542123Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:33:13.3542373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-26T20:33:13.3542482Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:13.3542487Z 2025-08-26T20:33:13.3542591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3542791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3542868Z return mod(**inputs) 2025-08-26T20:33:13.3543123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3543200Z outputs = self.model( 2025-08-26T20:33:13.3543453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3543527Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3543787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3543861Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3544089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3544185Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3544440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3544572Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3544576Z 2025-08-26T20:33:13.3544678Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3544889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3544954Z return mod(**inputs) 2025-08-26T20:33:13.3545216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3545286Z outputs = self.model( 2025-08-26T20:33:13.3545540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3545624Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3545875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3545954Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3546176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3546255Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3546527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-26T20:33:13.3546648Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:13.3546651Z 2025-08-26T20:33:13.3546764Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3546962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3547035Z return mod(**inputs) 2025-08-26T20:33:13.3547288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3547355Z outputs = self.model( 2025-08-26T20:33:13.3547630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3547704Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3547963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3548033Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3548255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3548359Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3548613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-26T20:33:13.3548705Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:13.3548708Z 2025-08-26T20:33:13.3548811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3549020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3549085Z return mod(**inputs) 2025-08-26T20:33:13.3549340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-26T20:33:13.3549417Z outputs = self.model( 2025-08-26T20:33:13.3549668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-26T20:33:13.3549750Z decoder_outputs = self.decoder( 2025-08-26T20:33:13.3550003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-26T20:33:13.3550092Z layer_outputs = decoder_layer( 2025-08-26T20:33:13.3550319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:13.3550400Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:13.3550657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-26T20:33:13.3550739Z hidden_states = residual + hidden_states 2025-08-26T20:33:13.3550743Z 2025-08-26T20:33:13.3550848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3551056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3551122Z return mod(**inputs) 2025-08-26T20:33:13.3551383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1422, in forward 2025-08-26T20:33:13.3551469Z lm_logits = self.lm_head(outputs[0]) 2025-08-26T20:33:13.3551473Z 2025-08-26T20:33:13.3551584Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:13.3551782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:13.3551849Z return mod(**inputs) 2025-08-26T20:33:13.3552112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1429, in forward 2025-08-26T20:33:13.3552300Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:33:13.3552305Z 2025-08-26T20:33:25.7099296Z Compilation time (from dynamo_timed): 28.602257557 2025-08-26T20:33:25.7195751Z pass 2025-08-26T20:33:25.7196680Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:33:25.7197616Z TIMING: _recursive_pre_grad_passes:0.01556 _recursive_joint_graph_passes:1.17601 _recursive_post_grad_passes:0.16034 async_compile.wait:0.80465 code_gen:11.88453 inductor_compile:15.07683 backend_compile:22.54757 gc:0.00082 entire_frame_compile:28.60226 total_wall_time:28.60226 2025-08-26T20:33:25.7198637Z STATS: call_* op count: 1014 | FakeTensorMode.__torch_dispatch__:33758 | FakeTensor.__torch_dispatch__:10654 | ProxyTorchDispatchMode.__torch_dispatch__:12417 2025-08-26T20:33:25.7199706Z Dynamo produced 1 graphs covering 1014 ops with 0 graph breaks (0 unique) 2025-08-26T20:33:31.7342880Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:33:31.7344344Z from pkg_resources import resource_filename 2025-08-26T20:33:32.3301632Z 2025-08-26T20:33:35.2834686Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:33:35.2835318Z loading model: 0it [00:02, ?it/s] 2025-08-26T20:33:35.2849650Z cpu eval MBartForCausalLM 2025-08-26T20:33:37.0280936Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:33:37.6284430Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:33:38.3024692Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:33:46.3161073Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3161458Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3161682Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3161895Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3162105Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3162325Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3162546Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3162751Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3163340Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3163568Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3163790Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3164018Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3164295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3164741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3165129Z return mod(**inputs) 2025-08-26T20:33:46.3165589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3166044Z outputs = self.model.decoder( 2025-08-26T20:33:46.3166542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3166966Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3167354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3167730Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3168134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3168565Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3168982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3169524Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3169753Z 2025-08-26T20:33:46.3169916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3170286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3170626Z return mod(**inputs) 2025-08-26T20:33:46.3171011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3171444Z outputs = self.model.decoder( 2025-08-26T20:33:46.3171857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3172307Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3172675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3173080Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3173525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3173979Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3174501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3174934Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3175084Z 2025-08-26T20:33:46.3175205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3175607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3175936Z return mod(**inputs) 2025-08-26T20:33:46.3176344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3176773Z outputs = self.model.decoder( 2025-08-26T20:33:46.3177193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3177613Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3178003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3178397Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3178839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3179282Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3179713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3180151Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3180318Z 2025-08-26T20:33:46.3180410Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3180639Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3180861Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3181077Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3181364Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3181756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3182103Z return mod(**inputs) 2025-08-26T20:33:46.3182500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3182919Z outputs = self.model.decoder( 2025-08-26T20:33:46.3183334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3183754Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3184139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3184547Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3184978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3185425Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3185862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3186316Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3186803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3187294Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3187506Z 2025-08-26T20:33:46.3187629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3188014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3188366Z return mod(**inputs) 2025-08-26T20:33:46.3188753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3189179Z outputs = self.model.decoder( 2025-08-26T20:33:46.3189620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3190044Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3190426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3190828Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3191262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3191704Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3192153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3192603Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3193089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3193592Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3193788Z 2025-08-26T20:33:46.3193900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3194291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3194645Z return mod(**inputs) 2025-08-26T20:33:46.3195052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3195497Z outputs = self.model.decoder( 2025-08-26T20:33:46.3195907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3196523Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3196911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3197314Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3197731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3198180Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3198632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3199075Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3199228Z 2025-08-26T20:33:46.3199354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3200189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3200565Z return mod(**inputs) 2025-08-26T20:33:46.3200976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3201429Z outputs = self.model.decoder( 2025-08-26T20:33:46.3201847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3202265Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3202655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3203056Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3203528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3204010Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3204215Z 2025-08-26T20:33:46.3204342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3204730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3205076Z return mod(**inputs) 2025-08-26T20:33:46.3205541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3205956Z outputs = self.model.decoder( 2025-08-26T20:33:46.3206359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3206781Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3207161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3207547Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3207959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3208425Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3208845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3209215Z return self.act(input) 2025-08-26T20:33:46.3209335Z 2025-08-26T20:33:46.3209446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3209869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3210221Z return mod(**inputs) 2025-08-26T20:33:46.3210615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3211040Z outputs = self.model.decoder( 2025-08-26T20:33:46.3211442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3211859Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3212232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3212643Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3213084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3213519Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3213673Z 2025-08-26T20:33:46.3213784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3214173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3214528Z return mod(**inputs) 2025-08-26T20:33:46.3214914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3215333Z outputs = self.model.decoder( 2025-08-26T20:33:46.3215762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3216185Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3216559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3216995Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3217421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3217866Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3218325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3218826Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3219048Z 2025-08-26T20:33:46.3219160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3219552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3219898Z return mod(**inputs) 2025-08-26T20:33:46.3220284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3220711Z outputs = self.model.decoder( 2025-08-26T20:33:46.3221174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3221588Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3221963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3222362Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3222772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3223214Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3223652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3224083Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3224232Z 2025-08-26T20:33:46.3224352Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3224744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3225090Z return mod(**inputs) 2025-08-26T20:33:46.3225479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3225906Z outputs = self.model.decoder( 2025-08-26T20:33:46.3226315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3226729Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3227112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3227499Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3227915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3228350Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3228826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3229256Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3229407Z 2025-08-26T20:33:46.3229504Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3229731Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3229959Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3230181Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3230449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3230844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3231187Z return mod(**inputs) 2025-08-26T20:33:46.3231578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3232002Z outputs = self.model.decoder( 2025-08-26T20:33:46.3232408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3232815Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3233206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3233595Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3234013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3234459Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3234890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3235356Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3235838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3236367Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3236571Z 2025-08-26T20:33:46.3236693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3237090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3237452Z return mod(**inputs) 2025-08-26T20:33:46.3237858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3238290Z outputs = self.model.decoder( 2025-08-26T20:33:46.3238706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3239138Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3239593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3240033Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3240469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3240910Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3241328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3241743Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3242212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3242705Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3242881Z 2025-08-26T20:33:46.3242993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3243385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3243759Z return mod(**inputs) 2025-08-26T20:33:46.3244128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3244521Z outputs = self.model.decoder( 2025-08-26T20:33:46.3244907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3245298Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3245668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3246042Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3246443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3246871Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3247297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3247706Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3247850Z 2025-08-26T20:33:46.3247968Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3248359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3248696Z return mod(**inputs) 2025-08-26T20:33:46.3249070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3249465Z outputs = self.model.decoder( 2025-08-26T20:33:46.3249845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3250269Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3250623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3250992Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3251390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3251841Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3252038Z 2025-08-26T20:33:46.3252150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3252552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3252916Z return mod(**inputs) 2025-08-26T20:33:46.3253301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3253691Z outputs = self.model.decoder( 2025-08-26T20:33:46.3254081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3254519Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3254900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3255288Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3255709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3256180Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3256582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3256935Z return self.act(input) 2025-08-26T20:33:46.3257048Z 2025-08-26T20:33:46.3257154Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3257548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3257910Z return mod(**inputs) 2025-08-26T20:33:46.3258309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3258783Z outputs = self.model.decoder( 2025-08-26T20:33:46.3259194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3259631Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3260036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3260448Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3260862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3261303Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3261458Z 2025-08-26T20:33:46.3261571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3261976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3262335Z return mod(**inputs) 2025-08-26T20:33:46.3262727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3263185Z outputs = self.model.decoder( 2025-08-26T20:33:46.3263606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3264033Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3264408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3265054Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3265477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:33:46.3265932Z hidden_states = residual + hidden_states 2025-08-26T20:33:46.3266086Z 2025-08-26T20:33:46.3266212Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3266602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3266958Z return mod(**inputs) 2025-08-26T20:33:46.3267356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3267794Z outputs = self.model.decoder( 2025-08-26T20:33:46.3268223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3268646Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3269037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3269434Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3269861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3270323Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3270790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3271295Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3271520Z 2025-08-26T20:33:46.3271631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3272024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3272366Z return mod(**inputs) 2025-08-26T20:33:46.3272760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3273180Z outputs = self.model.decoder( 2025-08-26T20:33:46.3273587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3274005Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3274376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3274773Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3275194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3275635Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3276094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3276511Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3276668Z 2025-08-26T20:33:46.3276781Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3277166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3277512Z return mod(**inputs) 2025-08-26T20:33:46.3277890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3278317Z outputs = self.model.decoder( 2025-08-26T20:33:46.3278739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3279159Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3279614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3280009Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3280429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3280965Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3281407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3281853Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3282007Z 2025-08-26T20:33:46.3282105Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3282330Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3282550Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3282765Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3282998Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3283372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3283705Z return mod(**inputs) 2025-08-26T20:33:46.3284078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3284477Z outputs = self.model.decoder( 2025-08-26T20:33:46.3284885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3285276Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3285632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3286001Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3286390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3286812Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3287226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3287640Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3288090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3288569Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3288762Z 2025-08-26T20:33:46.3288868Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3289233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3289563Z return mod(**inputs) 2025-08-26T20:33:46.3289929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3290384Z outputs = self.model.decoder( 2025-08-26T20:33:46.3290798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3291202Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3291561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3291922Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3292318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3292731Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3293164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3293582Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3294025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3294489Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3294659Z 2025-08-26T20:33:46.3294763Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3295146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3295479Z return mod(**inputs) 2025-08-26T20:33:46.3295837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3296411Z outputs = self.model.decoder( 2025-08-26T20:33:46.3296802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3297196Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3297544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3297911Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3298310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3298720Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3299121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3299568Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3299717Z 2025-08-26T20:33:46.3299823Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3300193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3300523Z return mod(**inputs) 2025-08-26T20:33:46.3300891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3301344Z outputs = self.model.decoder( 2025-08-26T20:33:46.3301721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3302112Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3302462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3302824Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3303225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3303671Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3303847Z 2025-08-26T20:33:46.3303963Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3304327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3304676Z return mod(**inputs) 2025-08-26T20:33:46.3305048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3305446Z outputs = self.model.decoder( 2025-08-26T20:33:46.3305836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3306235Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3306584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3306963Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3307378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3307807Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3308187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3308526Z return self.act(input) 2025-08-26T20:33:46.3308644Z 2025-08-26T20:33:46.3308750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3309118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3309479Z return mod(**inputs) 2025-08-26T20:33:46.3309840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3310239Z outputs = self.model.decoder( 2025-08-26T20:33:46.3310624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3311018Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3311369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3311732Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3312117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3312509Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3312650Z 2025-08-26T20:33:46.3312759Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3313115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3313460Z return mod(**inputs) 2025-08-26T20:33:46.3313829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3314216Z outputs = self.model.decoder( 2025-08-26T20:33:46.3314602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3314986Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3315339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3315710Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3316107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3316523Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3316937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3317411Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3317628Z 2025-08-26T20:33:46.3317734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3318101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3318423Z return mod(**inputs) 2025-08-26T20:33:46.3318847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3319271Z outputs = self.model.decoder( 2025-08-26T20:33:46.3319743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3320176Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3320548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3320951Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3321352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3321793Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3322211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3322621Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3322769Z 2025-08-26T20:33:46.3322876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3323238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3323589Z return mod(**inputs) 2025-08-26T20:33:46.3323951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3324355Z outputs = self.model.decoder( 2025-08-26T20:33:46.3324746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3325143Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3325500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3325858Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3326254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3326674Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3327086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3327498Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3327660Z 2025-08-26T20:33:46.3327743Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3327961Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3328176Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3328385Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3328617Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3328984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3329312Z return mod(**inputs) 2025-08-26T20:33:46.3329678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3330072Z outputs = self.model.decoder( 2025-08-26T20:33:46.3330446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3330830Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3331175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3331533Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3331905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3332310Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3332711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3333139Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3333583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3334056Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3334247Z 2025-08-26T20:33:46.3334353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3334716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3335044Z return mod(**inputs) 2025-08-26T20:33:46.3335436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3335827Z outputs = self.model.decoder( 2025-08-26T20:33:46.3336205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3336593Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3336938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3337289Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3337693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3338103Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3338510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3338916Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3339359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3339823Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3339994Z 2025-08-26T20:33:46.3340099Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3340470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3340792Z return mod(**inputs) 2025-08-26T20:33:46.3341155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3341599Z outputs = self.model.decoder( 2025-08-26T20:33:46.3342011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3342445Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3342814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3343200Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3343588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3343995Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3344394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3344781Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3344925Z 2025-08-26T20:33:46.3345028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3345379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3345698Z return mod(**inputs) 2025-08-26T20:33:46.3346047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3346432Z outputs = self.model.decoder( 2025-08-26T20:33:46.3346822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3347203Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3347547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3347899Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3348282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3348719Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3348893Z 2025-08-26T20:33:46.3349003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3349364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3349705Z return mod(**inputs) 2025-08-26T20:33:46.3350078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3350501Z outputs = self.model.decoder( 2025-08-26T20:33:46.3350956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3351346Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3351725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3352117Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3352535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3352975Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3353362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3353712Z return self.act(input) 2025-08-26T20:33:46.3353831Z 2025-08-26T20:33:46.3353937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3354308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3354636Z return mod(**inputs) 2025-08-26T20:33:46.3354996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3355410Z outputs = self.model.decoder( 2025-08-26T20:33:46.3355841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3356266Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3356632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3357025Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3357425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3357831Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3357971Z 2025-08-26T20:33:46.3358083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3358451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3358802Z return mod(**inputs) 2025-08-26T20:33:46.3359191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3359712Z outputs = self.model.decoder( 2025-08-26T20:33:46.3360128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3360568Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3360941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3361307Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3361720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:33:46.3362117Z hidden_states = residual + hidden_states 2025-08-26T20:33:46.3362273Z 2025-08-26T20:33:46.3362389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3362789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3363121Z return mod(**inputs) 2025-08-26T20:33:46.3363487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3363876Z outputs = self.model.decoder( 2025-08-26T20:33:46.3364273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3364670Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3365028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3365388Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3365785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3366228Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3366652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3367124Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3367332Z 2025-08-26T20:33:46.3367437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3367807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3368135Z return mod(**inputs) 2025-08-26T20:33:46.3368503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3368902Z outputs = self.model.decoder( 2025-08-26T20:33:46.3369283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3369679Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3370036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3370440Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3370830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3371253Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3371672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3372075Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3372213Z 2025-08-26T20:33:46.3372327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3372685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3373016Z return mod(**inputs) 2025-08-26T20:33:46.3373382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3373785Z outputs = self.model.decoder( 2025-08-26T20:33:46.3374176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3374562Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3374918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3375281Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3375691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3376109Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3376608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3377017Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3377162Z 2025-08-26T20:33:46.3377252Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3377473Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3377680Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3377899Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3378152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3378566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3378920Z return mod(**inputs) 2025-08-26T20:33:46.3379327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3379789Z outputs = self.model.decoder( 2025-08-26T20:33:46.3380196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3380644Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3381014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3381412Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3381829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3382345Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3382784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3383236Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3383738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3384271Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3384470Z 2025-08-26T20:33:46.3384588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3384997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3385361Z return mod(**inputs) 2025-08-26T20:33:46.3385759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3386192Z outputs = self.model.decoder( 2025-08-26T20:33:46.3386612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3387036Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3387422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3387814Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3388238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3388687Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3389139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3389580Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3390065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3390556Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3390728Z 2025-08-26T20:33:46.3390867Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3391251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3391601Z return mod(**inputs) 2025-08-26T20:33:46.3392018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3392450Z outputs = self.model.decoder( 2025-08-26T20:33:46.3392850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3393249Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3393655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3394053Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3394473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3394910Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3395364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3395826Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3395971Z 2025-08-26T20:33:46.3396093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3396624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3396969Z return mod(**inputs) 2025-08-26T20:33:46.3397370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3397793Z outputs = self.model.decoder( 2025-08-26T20:33:46.3398205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3398627Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3399007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3399393Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3399873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3400414Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3400612Z 2025-08-26T20:33:46.3400731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3401134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3417773Z return mod(**inputs) 2025-08-26T20:33:46.3418428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3418872Z outputs = self.model.decoder( 2025-08-26T20:33:46.3419313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3419740Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3420141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3420553Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3421002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3421489Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3421934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3422328Z return self.act(input) 2025-08-26T20:33:46.3422457Z 2025-08-26T20:33:46.3422589Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3423110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3423481Z return mod(**inputs) 2025-08-26T20:33:46.3423886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3424318Z outputs = self.model.decoder( 2025-08-26T20:33:46.3424735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3425161Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3425543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3425993Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3426397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3426801Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3426962Z 2025-08-26T20:33:46.3427081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3427475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3427881Z return mod(**inputs) 2025-08-26T20:33:46.3428276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3428709Z outputs = self.model.decoder( 2025-08-26T20:33:46.3429125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3429555Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3429922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3430296Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3430710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3431172Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3431631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3432140Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3432391Z 2025-08-26T20:33:46.3432507Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3432903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3433255Z return mod(**inputs) 2025-08-26T20:33:46.3433652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3434078Z outputs = self.model.decoder( 2025-08-26T20:33:46.3434484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3434903Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3435276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3435671Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3436085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3436528Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3436973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3437397Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3437541Z 2025-08-26T20:33:46.3437660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3438064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3438419Z return mod(**inputs) 2025-08-26T20:33:46.3438811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3439234Z outputs = self.model.decoder( 2025-08-26T20:33:46.3439756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3440179Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3440571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3440971Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3441378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3441778Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3442187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3442585Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3442730Z 2025-08-26T20:33:46.3442844Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3443061Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3443265Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3443471Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3443704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3444065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3444379Z return mod(**inputs) 2025-08-26T20:33:46.3444739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3445122Z outputs = self.model.decoder( 2025-08-26T20:33:46.3445500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3445882Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3446216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3446577Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3446987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3447402Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3447810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3448227Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3448680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3449181Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3449375Z 2025-08-26T20:33:46.3449493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3449865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3450196Z return mod(**inputs) 2025-08-26T20:33:46.3450566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3450960Z outputs = self.model.decoder( 2025-08-26T20:33:46.3451344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3451730Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3452086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3452468Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3452854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3453260Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3453659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3454062Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3454502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3454951Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3455131Z 2025-08-26T20:33:46.3455243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3455596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3455923Z return mod(**inputs) 2025-08-26T20:33:46.3456329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3456734Z outputs = self.model.decoder( 2025-08-26T20:33:46.3457137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3457537Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3457896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3458268Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3458665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3459082Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3459488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3459881Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3460018Z 2025-08-26T20:33:46.3460129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3460482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3460828Z return mod(**inputs) 2025-08-26T20:33:46.3461193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3461583Z outputs = self.model.decoder( 2025-08-26T20:33:46.3461968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3462358Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3462718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3463092Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3463490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3463950Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3464125Z 2025-08-26T20:33:46.3464234Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3464605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3464946Z return mod(**inputs) 2025-08-26T20:33:46.3465325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3465720Z outputs = self.model.decoder( 2025-08-26T20:33:46.3466113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3467303Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3467670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3468038Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3468427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3468871Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3469263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3469609Z return self.act(input) 2025-08-26T20:33:46.3469723Z 2025-08-26T20:33:46.3469857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3470223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3470552Z return mod(**inputs) 2025-08-26T20:33:46.3470922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3471322Z outputs = self.model.decoder( 2025-08-26T20:33:46.3471699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3472128Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3472482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3472850Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3473243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3473639Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3473787Z 2025-08-26T20:33:46.3473893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3474264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3474596Z return mod(**inputs) 2025-08-26T20:33:46.3474959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3475364Z outputs = self.model.decoder( 2025-08-26T20:33:46.3475757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3476180Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3476532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3476917Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3477340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:33:46.3477760Z hidden_states = residual + hidden_states 2025-08-26T20:33:46.3477906Z 2025-08-26T20:33:46.3478028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3478413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3478753Z return mod(**inputs) 2025-08-26T20:33:46.3479143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3479663Z outputs = self.model.decoder( 2025-08-26T20:33:46.3480078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3480490Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3480880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3481267Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3481686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3482108Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3482519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3482998Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3483223Z 2025-08-26T20:33:46.3483327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3483685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3484007Z return mod(**inputs) 2025-08-26T20:33:46.3484377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3484775Z outputs = self.model.decoder( 2025-08-26T20:33:46.3485170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3485571Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3485924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3486309Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3486703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3487119Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3487531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3487924Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3488071Z 2025-08-26T20:33:46.3488178Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3488542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3488888Z return mod(**inputs) 2025-08-26T20:33:46.3489277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3489688Z outputs = self.model.decoder( 2025-08-26T20:33:46.3490092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3490524Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3490901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3491282Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3491701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3492141Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3492581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3493025Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3493170Z 2025-08-26T20:33:46.3493253Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3493479Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3493706Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3493933Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3494182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3494575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3494926Z return mod(**inputs) 2025-08-26T20:33:46.3495319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3495739Z outputs = self.model.decoder( 2025-08-26T20:33:46.3496318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3496760Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3497146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3497539Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3497952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3498398Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3498836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3499341Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3499825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3500339Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3500548Z 2025-08-26T20:33:46.3500661Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3501057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3501437Z return mod(**inputs) 2025-08-26T20:33:46.3501829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3502242Z outputs = self.model.decoder( 2025-08-26T20:33:46.3502655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3503071Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3503443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3503823Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3504234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3504679Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3505092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3505533Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3505985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3506441Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3506610Z 2025-08-26T20:33:46.3506720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3507087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3507418Z return mod(**inputs) 2025-08-26T20:33:46.3507787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3508188Z outputs = self.model.decoder( 2025-08-26T20:33:46.3508578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3508977Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3509352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3509728Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3510146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3510586Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3511055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3511458Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3511604Z 2025-08-26T20:33:46.3511716Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3512101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3512443Z return mod(**inputs) 2025-08-26T20:33:46.3512824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3513232Z outputs = self.model.decoder( 2025-08-26T20:33:46.3513656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3514125Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3514508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3514916Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3515327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3515793Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3516000Z 2025-08-26T20:33:46.3516112Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3516497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3516840Z return mod(**inputs) 2025-08-26T20:33:46.3517228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3517645Z outputs = self.model.decoder( 2025-08-26T20:33:46.3518082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3518517Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3518898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3519298Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3519786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3520274Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3520726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3521102Z return self.act(input) 2025-08-26T20:33:46.3521219Z 2025-08-26T20:33:46.3521323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3521697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3522059Z return mod(**inputs) 2025-08-26T20:33:46.3522464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3522889Z outputs = self.model.decoder( 2025-08-26T20:33:46.3523315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3523752Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3524141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3524535Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3524961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3525400Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3525550Z 2025-08-26T20:33:46.3525674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3526090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3526446Z return mod(**inputs) 2025-08-26T20:33:46.3526848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3527282Z outputs = self.model.decoder( 2025-08-26T20:33:46.3527698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3528119Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3528504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3528903Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3529361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3529814Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3530263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3530778Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3531018Z 2025-08-26T20:33:46.3531169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3531558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3531912Z return mod(**inputs) 2025-08-26T20:33:46.3532292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3532712Z outputs = self.model.decoder( 2025-08-26T20:33:46.3533122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3533542Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3533925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3534319Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3534738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3535184Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3535648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3536059Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3536211Z 2025-08-26T20:33:46.3536324Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3536711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3537059Z return mod(**inputs) 2025-08-26T20:33:46.3537448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3537869Z outputs = self.model.decoder( 2025-08-26T20:33:46.3538274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3538666Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3539037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3539428Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3539817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3540229Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3540648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3541059Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3541203Z 2025-08-26T20:33:46.3541304Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3541529Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3541745Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3541957Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3542190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3542551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3542889Z return mod(**inputs) 2025-08-26T20:33:46.3543247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3543633Z outputs = self.model.decoder( 2025-08-26T20:33:46.3544027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3544422Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3544775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3545140Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3545536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3545966Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3546405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3546847Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3547327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3547835Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3548042Z 2025-08-26T20:33:46.3548155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3548545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3548897Z return mod(**inputs) 2025-08-26T20:33:46.3549290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3549706Z outputs = self.model.decoder( 2025-08-26T20:33:46.3550130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3550542Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3550915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3551307Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3551705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3552110Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3552518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3552930Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3553372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3553833Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3554002Z 2025-08-26T20:33:46.3554108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3554469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3554804Z return mod(**inputs) 2025-08-26T20:33:46.3555183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3555621Z outputs = self.model.decoder( 2025-08-26T20:33:46.3556030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3556447Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3556822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3557202Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3557621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3558061Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3558518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3558958Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3559112Z 2025-08-26T20:33:46.3559229Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3559708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3560080Z return mod(**inputs) 2025-08-26T20:33:46.3560488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3560942Z outputs = self.model.decoder( 2025-08-26T20:33:46.3561359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3561755Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3562109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3562479Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3562868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3563305Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3563487Z 2025-08-26T20:33:46.3563592Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3563959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3564283Z return mod(**inputs) 2025-08-26T20:33:46.3564675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3565068Z outputs = self.model.decoder( 2025-08-26T20:33:46.3565455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3565854Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3566199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3566566Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3566958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3567395Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3567788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3568131Z return self.act(input) 2025-08-26T20:33:46.3568253Z 2025-08-26T20:33:46.3568360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3568725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3569049Z return mod(**inputs) 2025-08-26T20:33:46.3569407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3569804Z outputs = self.model.decoder( 2025-08-26T20:33:46.3570203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3570595Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3570944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3571303Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3571700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3572099Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3572247Z 2025-08-26T20:33:46.3572368Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3572768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3573117Z return mod(**inputs) 2025-08-26T20:33:46.3573505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3573919Z outputs = self.model.decoder( 2025-08-26T20:33:46.3574324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3574749Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3575119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3575505Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3575927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:33:46.3576348Z hidden_states = residual + hidden_states 2025-08-26T20:33:46.3576493Z 2025-08-26T20:33:46.3576608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3576834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3576907Z return mod(**inputs) 2025-08-26T20:33:46.3577183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3577264Z outputs = self.model.decoder( 2025-08-26T20:33:46.3577548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3577655Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3577896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3577987Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3578262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3578370Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3578660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3578822Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3578826Z 2025-08-26T20:33:46.3578944Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3579160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3579238Z return mod(**inputs) 2025-08-26T20:33:46.3579513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3579592Z outputs = self.model.decoder( 2025-08-26T20:33:46.3579891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3579968Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3580239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3580325Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3580598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3580712Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3580989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3581086Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3581090Z 2025-08-26T20:33:46.3581201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3581442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3581516Z return mod(**inputs) 2025-08-26T20:33:46.3581788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3581878Z outputs = self.model.decoder( 2025-08-26T20:33:46.3582164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3582248Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3582503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3582589Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3582878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3582983Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3583257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3583349Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3583353Z 2025-08-26T20:33:46.3583450Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3583538Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3583620Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3583709Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3583821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3584033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3584129Z return mod(**inputs) 2025-08-26T20:33:46.3584406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3584493Z outputs = self.model.decoder( 2025-08-26T20:33:46.3584771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3584855Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3585094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3585179Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3585458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3585566Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3585845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3585952Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3586274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3586428Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3586433Z 2025-08-26T20:33:46.3586543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3586785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3586858Z return mod(**inputs) 2025-08-26T20:33:46.3587133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3587215Z outputs = self.model.decoder( 2025-08-26T20:33:46.3587486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3587574Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3587808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3587898Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3588185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3588294Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3588570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3588673Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3589012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3589133Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3589138Z 2025-08-26T20:33:46.3589257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3589472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3589545Z return mod(**inputs) 2025-08-26T20:33:46.3589824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3589902Z outputs = self.model.decoder( 2025-08-26T20:33:46.3590182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3590259Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3590493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3590588Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3590881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3590991Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3591259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3591347Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3591358Z 2025-08-26T20:33:46.3591468Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3591684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3591763Z return mod(**inputs) 2025-08-26T20:33:46.3592034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3592121Z outputs = self.model.decoder( 2025-08-26T20:33:46.3592404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3592476Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3592704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3592785Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3593043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3593183Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3593187Z 2025-08-26T20:33:46.3593291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3593500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3593569Z return mod(**inputs) 2025-08-26T20:33:46.3593851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3593930Z outputs = self.model.decoder( 2025-08-26T20:33:46.3594202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3594283Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3594541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3594638Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3594918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3595058Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3595316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3595406Z return self.act(input) 2025-08-26T20:33:46.3595412Z 2025-08-26T20:33:46.3595527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3595736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3595812Z return mod(**inputs) 2025-08-26T20:33:46.3596083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3596365Z outputs = self.model.decoder( 2025-08-26T20:33:46.3596656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3596734Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3596981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3597067Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3597349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3597483Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3597488Z 2025-08-26T20:33:46.3597598Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3597821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3597893Z return mod(**inputs) 2025-08-26T20:33:46.3598174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3598256Z outputs = self.model.decoder( 2025-08-26T20:33:46.3598531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3598618Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3598860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3598957Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3599233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3599348Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3599681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3599854Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3599859Z 2025-08-26T20:33:46.3600022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3600242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3600321Z return mod(**inputs) 2025-08-26T20:33:46.3600610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3600690Z outputs = self.model.decoder( 2025-08-26T20:33:46.3600967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3601041Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3601314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3601396Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3601657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3601757Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3602012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3602125Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3602128Z 2025-08-26T20:33:46.3602234Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3602447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3602514Z return mod(**inputs) 2025-08-26T20:33:46.3602773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3602855Z outputs = self.model.decoder( 2025-08-26T20:33:46.3603117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3603199Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3603424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3603505Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3603772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3603895Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3604162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3604255Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3604259Z 2025-08-26T20:33:46.3604357Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3604444Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3604527Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3604617Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3604727Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3604947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3605019Z return mod(**inputs) 2025-08-26T20:33:46.3605295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3605386Z outputs = self.model.decoder( 2025-08-26T20:33:46.3605658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3605743Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3605982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3606067Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3606404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3606506Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3606774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3606878Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3607200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3607342Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3607346Z 2025-08-26T20:33:46.3607455Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3607693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3607766Z return mod(**inputs) 2025-08-26T20:33:46.3608047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3608127Z outputs = self.model.decoder( 2025-08-26T20:33:46.3608398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3608504Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3608742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3608831Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3609100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3609210Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3609477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3609579Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3609900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3610022Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3610026Z 2025-08-26T20:33:46.3610145Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3610374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3610444Z return mod(**inputs) 2025-08-26T20:33:46.3610724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3610803Z outputs = self.model.decoder( 2025-08-26T20:33:46.3611080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3611159Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3611393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3611484Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3611751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3611865Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3612133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3612226Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3612230Z 2025-08-26T20:33:46.3612340Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3612547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3612625Z return mod(**inputs) 2025-08-26T20:33:46.3612910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3613039Z outputs = self.model.decoder( 2025-08-26T20:33:46.3613316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3613397Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3613639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3613722Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3613994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3614141Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3614145Z 2025-08-26T20:33:46.3614263Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3614477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3614547Z return mod(**inputs) 2025-08-26T20:33:46.3614823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3614919Z outputs = self.model.decoder( 2025-08-26T20:33:46.3615196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3615274Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3615507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3615599Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3615867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3615999Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3616228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3616307Z return self.act(input) 2025-08-26T20:33:46.3616313Z 2025-08-26T20:33:46.3616420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3616633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3616732Z return mod(**inputs) 2025-08-26T20:33:46.3617007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3617093Z outputs = self.model.decoder( 2025-08-26T20:33:46.3617366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3617444Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3617690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3617775Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3618055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3618148Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3618154Z 2025-08-26T20:33:46.3618268Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3618498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3618571Z return mod(**inputs) 2025-08-26T20:33:46.3618864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3618944Z outputs = self.model.decoder( 2025-08-26T20:33:46.3619241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3619320Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3619557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3619649Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3619922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:33:46.3620017Z hidden_states = residual + hidden_states 2025-08-26T20:33:46.3620021Z 2025-08-26T20:33:46.3620130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3620341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3620438Z return mod(**inputs) 2025-08-26T20:33:46.3620706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3620795Z outputs = self.model.decoder( 2025-08-26T20:33:46.3621066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3621142Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3621404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3621492Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3621776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3621881Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3622164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3622327Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3622331Z 2025-08-26T20:33:46.3622444Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3622670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3622739Z return mod(**inputs) 2025-08-26T20:33:46.3623029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3623129Z outputs = self.model.decoder( 2025-08-26T20:33:46.3623405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3623489Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3623733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3623824Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3624101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3624213Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3624488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3624576Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3624580Z 2025-08-26T20:33:46.3624697Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3624921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3624997Z return mod(**inputs) 2025-08-26T20:33:46.3625271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3625352Z outputs = self.model.decoder( 2025-08-26T20:33:46.3625637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3625733Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3625979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3626063Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3626337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3626442Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3626714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3626817Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3626821Z 2025-08-26T20:33:46.3626925Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3627021Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3627107Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3627190Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3627313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3627533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3627612Z return mod(**inputs) 2025-08-26T20:33:46.3627923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3628009Z outputs = self.model.decoder( 2025-08-26T20:33:46.3628299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3628377Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3628638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3628723Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3628994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3629104Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3629380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3629496Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3629841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3629995Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3630000Z 2025-08-26T20:33:46.3630113Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3630330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3630410Z return mod(**inputs) 2025-08-26T20:33:46.3630693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3630781Z outputs = self.model.decoder( 2025-08-26T20:33:46.3631062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3631142Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3631394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3631482Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3631766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3631873Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3632162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3632286Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3632607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3632734Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3632740Z 2025-08-26T20:33:46.3632851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3633076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3633147Z return mod(**inputs) 2025-08-26T20:33:46.3633424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3633512Z outputs = self.model.decoder( 2025-08-26T20:33:46.3633804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3633893Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3634136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3634229Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3634505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3634630Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3634921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3635012Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3635016Z 2025-08-26T20:33:46.3635136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3635355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3635427Z return mod(**inputs) 2025-08-26T20:33:46.3635714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3635795Z outputs = self.model.decoder( 2025-08-26T20:33:46.3636084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3636164Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3636433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3636519Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3636808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3636947Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3636951Z 2025-08-26T20:33:46.3637064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3637290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3637363Z return mod(**inputs) 2025-08-26T20:33:46.3637664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3637754Z outputs = self.model.decoder( 2025-08-26T20:33:46.3638057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3638144Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3638416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3638502Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3638813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3638941Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3639200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3639279Z return self.act(input) 2025-08-26T20:33:46.3639283Z 2025-08-26T20:33:46.3639403Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3639718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3639799Z return mod(**inputs) 2025-08-26T20:33:46.3640091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3640172Z outputs = self.model.decoder( 2025-08-26T20:33:46.3640479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3640562Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3640846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3640955Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3641234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3641352Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3641357Z 2025-08-26T20:33:46.3641467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3641691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3641761Z return mod(**inputs) 2025-08-26T20:33:46.3642048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3642138Z outputs = self.model.decoder( 2025-08-26T20:33:46.3642428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3642513Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3642752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3642836Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3643127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3643250Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3643534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:33:46.3643694Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:33:46.3643698Z 2025-08-26T20:33:46.3643816Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3644028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3644098Z return mod(**inputs) 2025-08-26T20:33:46.3644386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3644465Z outputs = self.model.decoder( 2025-08-26T20:33:46.3644761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3644838Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3645072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3645162Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3645451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3645562Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3645858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:33:46.3645947Z key_states = self.k_proj(current_states) 2025-08-26T20:33:46.3645957Z 2025-08-26T20:33:46.3646066Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3646278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3646355Z return mod(**inputs) 2025-08-26T20:33:46.3646622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3646708Z outputs = self.model.decoder( 2025-08-26T20:33:46.3647004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3647082Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3647329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3647415Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3647689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3647791Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3648080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:33:46.3648182Z value_states = self.v_proj(current_states) 2025-08-26T20:33:46.3648186Z 2025-08-26T20:33:46.3648273Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3648365Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3648448Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3648531Z cudagraph partition due to non gpu ops 2025-08-26T20:33:46.3648648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3648859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3648936Z return mod(**inputs) 2025-08-26T20:33:46.3649202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3649282Z outputs = self.model.decoder( 2025-08-26T20:33:46.3649555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3649646Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3649873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3649953Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3650212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3650311Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3650568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3650672Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3650965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:33:46.3651108Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:33:46.3651113Z 2025-08-26T20:33:46.3651218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3651426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3651491Z return mod(**inputs) 2025-08-26T20:33:46.3651747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3651830Z outputs = self.model.decoder( 2025-08-26T20:33:46.3652129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3652211Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3652435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3652516Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3652782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3652883Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3653145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:33:46.3653270Z attn_output, attn_weights = attention_interface( 2025-08-26T20:33:46.3653565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:33:46.3653686Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:33:46.3653690Z 2025-08-26T20:33:46.3653796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3654004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3654089Z return mod(**inputs) 2025-08-26T20:33:46.3654354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3654430Z outputs = self.model.decoder( 2025-08-26T20:33:46.3654686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3654772Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3654995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3655082Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3655340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:33:46.3655438Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:33:46.3655701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:33:46.3655807Z attn_output = self.out_proj(attn_output) 2025-08-26T20:33:46.3655810Z 2025-08-26T20:33:46.3655923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3656124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3656195Z return mod(**inputs) 2025-08-26T20:33:46.3656452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3656528Z outputs = self.model.decoder( 2025-08-26T20:33:46.3656794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3656867Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3657095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3657174Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3657432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3657559Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3657563Z 2025-08-26T20:33:46.3657666Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3657872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3657938Z return mod(**inputs) 2025-08-26T20:33:46.3658207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3658293Z outputs = self.model.decoder( 2025-08-26T20:33:46.3658551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3658635Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3658858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3658947Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3659203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:33:46.3659342Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:33:46.3659566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:33:46.3659638Z return self.act(input) 2025-08-26T20:33:46.3659643Z 2025-08-26T20:33:46.3659754Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3659954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3660043Z return mod(**inputs) 2025-08-26T20:33:46.3660309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3660385Z outputs = self.model.decoder( 2025-08-26T20:33:46.3660646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3660718Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3660948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3661027Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3661281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:33:46.3661372Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:33:46.3661375Z 2025-08-26T20:33:46.3661478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3661685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3661769Z return mod(**inputs) 2025-08-26T20:33:46.3662023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-26T20:33:46.3662108Z outputs = self.model.decoder( 2025-08-26T20:33:46.3662366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:33:46.3662445Z layer_outputs = decoder_layer( 2025-08-26T20:33:46.3662668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:33:46.3662750Z return super().__call__(*args, **kwargs) 2025-08-26T20:33:46.3663012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:33:46.3663094Z hidden_states = residual + hidden_states 2025-08-26T20:33:46.3663097Z 2025-08-26T20:33:46.3663207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3663408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3663478Z return mod(**inputs) 2025-08-26T20:33:46.3663733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1880, in forward 2025-08-26T20:33:46.3663814Z logits = self.lm_head(outputs[0]) 2025-08-26T20:33:46.3663818Z 2025-08-26T20:33:46.3663928Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:33:46.3664145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:33:46.3664219Z return mod(**inputs) 2025-08-26T20:33:46.3664490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1886, in forward 2025-08-26T20:33:46.3664649Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:33:46.3664660Z 2025-08-26T20:33:56.4433937Z Compilation time (from dynamo_timed): 16.122253828 2025-08-26T20:33:56.4634689Z pass 2025-08-26T20:33:56.4635384Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:33:56.4636625Z TIMING: _recursive_pre_grad_passes:0.00789 _recursive_joint_graph_passes:0.66484 _recursive_post_grad_passes:0.08279 async_compile.wait:0.71079 code_gen:8.99003 inductor_compile:10.36431 backend_compile:13.72944 gc:0.00116 entire_frame_compile:16.12225 total_wall_time:16.12225 2025-08-26T20:33:56.4637658Z STATS: call_* op count: 373 | FakeTensorMode.__torch_dispatch__:13260 | FakeTensor.__torch_dispatch__:4593 | ProxyTorchDispatchMode.__torch_dispatch__:4844 2025-08-26T20:33:56.4638199Z Dynamo produced 1 graphs covering 373 ops with 0 graph breaks (0 unique) 2025-08-26T20:34:01.9421804Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:34:01.9423164Z from pkg_resources import resource_filename 2025-08-26T20:34:02.6092236Z 2025-08-26T20:34:07.9901542Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:34:07.9901854Z loading model: 0it [00:05, ?it/s] 2025-08-26T20:34:07.9932759Z cpu eval MBartForConditionalGeneration 2025-08-26T20:34:11.3793129Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:34:12.6643294Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:34:13.9695594Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:34:32.3072243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3075515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3076415Z return mod(**inputs) 2025-08-26T20:34:32.3076907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1436, in forward 2025-08-26T20:34:32.3077496Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-08-26T20:34:32.3078076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 76, in shift_tokens_right 2025-08-26T20:34:32.3078644Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-08-26T20:34:32.3078989Z 2025-08-26T20:34:32.3079131Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3079397Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3079859Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3080122Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3080359Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3080595Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3080838Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3081105Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3081325Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3081551Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3081771Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3082009Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3082271Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3082771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3083128Z return mod(**inputs) 2025-08-26T20:34:32.3083542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3083985Z outputs = self.model( 2025-08-26T20:34:32.3084398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3084838Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3085266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3085692Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3086126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3086528Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3086959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3087415Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3087867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3088489Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3088713Z 2025-08-26T20:34:32.3088831Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3089220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3089580Z return mod(**inputs) 2025-08-26T20:34:32.3089987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3090404Z outputs = self.model( 2025-08-26T20:34:32.3090829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3091265Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3091710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3092157Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3092688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3093143Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3093567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3094014Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3094478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3094911Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3095071Z 2025-08-26T20:34:32.3095188Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3095597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3095968Z return mod(**inputs) 2025-08-26T20:34:32.3096542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3096994Z outputs = self.model( 2025-08-26T20:34:32.3097411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3097841Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3098257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3098666Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3099100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3099507Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3099938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3100397Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3100827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3101262Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3101435Z 2025-08-26T20:34:32.3101526Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3101761Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3102006Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3102239Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3102500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3102890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3103231Z return mod(**inputs) 2025-08-26T20:34:32.3103624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3104064Z outputs = self.model( 2025-08-26T20:34:32.3104471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3104915Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3105331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3105759Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3106387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3106787Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3107224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3107665Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3108110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3108589Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3109129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3109684Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3109887Z 2025-08-26T20:34:32.3110008Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3110400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3110761Z return mod(**inputs) 2025-08-26T20:34:32.3111175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3111605Z outputs = self.model( 2025-08-26T20:34:32.3112018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3112473Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3112906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3113336Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3113725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3114135Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3114577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3115058Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3115509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3115976Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3116480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3117003Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3117185Z 2025-08-26T20:34:32.3117311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3117716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3118091Z return mod(**inputs) 2025-08-26T20:34:32.3118504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3118943Z outputs = self.model( 2025-08-26T20:34:32.3119351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3119869Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3120335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3120777Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3121178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3121587Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3122023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3122477Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3122923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3123367Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3123520Z 2025-08-26T20:34:32.3123642Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3124042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3124424Z return mod(**inputs) 2025-08-26T20:34:32.3124835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3125267Z outputs = self.model( 2025-08-26T20:34:32.3125667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3126108Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3126537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3126974Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3127365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3127765Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3128188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3128678Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3128871Z 2025-08-26T20:34:32.3128994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3129399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3129770Z return mod(**inputs) 2025-08-26T20:34:32.3130188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3130619Z outputs = self.model( 2025-08-26T20:34:32.3131037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3131454Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3131868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3132288Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3132665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3133057Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3133492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3133966Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3134391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3134760Z return self.act(input) 2025-08-26T20:34:32.3134880Z 2025-08-26T20:34:32.3135000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3135385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3135762Z return mod(**inputs) 2025-08-26T20:34:32.3136156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3136570Z outputs = self.model( 2025-08-26T20:34:32.3136963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3137386Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3137804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3138232Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3138618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3139013Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3139449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3139900Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3140082Z 2025-08-26T20:34:32.3140210Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3140603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3140972Z return mod(**inputs) 2025-08-26T20:34:32.3141378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3141809Z outputs = self.model( 2025-08-26T20:34:32.3142220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3142648Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3143087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3143515Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3143898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3144304Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3144731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3145192Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3145631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3146155Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3146380Z 2025-08-26T20:34:32.3146497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3146875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3147228Z return mod(**inputs) 2025-08-26T20:34:32.3147623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3148041Z outputs = self.model( 2025-08-26T20:34:32.3148465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3148913Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3149369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3149804Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3150187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3150571Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3150989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3151456Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3151890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3152314Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3152475Z 2025-08-26T20:34:32.3152588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3152979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3153332Z return mod(**inputs) 2025-08-26T20:34:32.3153726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3154134Z outputs = self.model( 2025-08-26T20:34:32.3154528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3154950Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3155375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3155834Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3156208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3156617Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3157058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3157514Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3157955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3158400Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3158570Z 2025-08-26T20:34:32.3158661Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3158898Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3159137Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3159363Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3159729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3160148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3160536Z return mod(**inputs) 2025-08-26T20:34:32.3160938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3161367Z outputs = self.model( 2025-08-26T20:34:32.3161802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3162247Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3162667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3163096Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3163492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3163913Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3164347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3164844Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3165303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3165762Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3166263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3166821Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3167033Z 2025-08-26T20:34:32.3167161Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3167569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3167944Z return mod(**inputs) 2025-08-26T20:34:32.3168379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3168814Z outputs = self.model( 2025-08-26T20:34:32.3169246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3169703Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3170138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3170581Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3170975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3171386Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3171820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3172266Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3172709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3173157Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3173653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3174170Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3174350Z 2025-08-26T20:34:32.3174475Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3174880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3175242Z return mod(**inputs) 2025-08-26T20:34:32.3175656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3176074Z outputs = self.model( 2025-08-26T20:34:32.3176477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3176892Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3177316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3177736Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3178130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3178536Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3178954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3179395Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3179822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3180258Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3180429Z 2025-08-26T20:34:32.3180549Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3180930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3181284Z return mod(**inputs) 2025-08-26T20:34:32.3181678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3182112Z outputs = self.model( 2025-08-26T20:34:32.3182505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3182922Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3183332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3183752Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3184135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3184513Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3184947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3185425Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3185620Z 2025-08-26T20:34:32.3185733Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3186126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3186514Z return mod(**inputs) 2025-08-26T20:34:32.3186908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3187324Z outputs = self.model( 2025-08-26T20:34:32.3187721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3188161Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3188583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3188999Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3189372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3189765Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3190190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3190671Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3191101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3191489Z return self.act(input) 2025-08-26T20:34:32.3191620Z 2025-08-26T20:34:32.3191741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3192123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3192478Z return mod(**inputs) 2025-08-26T20:34:32.3192881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3193297Z outputs = self.model( 2025-08-26T20:34:32.3193706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3194132Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3194557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3194983Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3195373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3195788Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3196370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3196831Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3196985Z 2025-08-26T20:34:32.3197110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3197518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3197931Z return mod(**inputs) 2025-08-26T20:34:32.3198336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3198772Z outputs = self.model( 2025-08-26T20:34:32.3199176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3199674Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3200100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3200531Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3200914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3201314Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3201740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-26T20:34:32.3202186Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3202379Z 2025-08-26T20:34:32.3202497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3202893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3203252Z return mod(**inputs) 2025-08-26T20:34:32.3203670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3204092Z outputs = self.model( 2025-08-26T20:34:32.3204497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3204928Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3205348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3205768Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3206152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3206553Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3206979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3207420Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3207868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3208406Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3208641Z 2025-08-26T20:34:32.3208770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3209181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3209544Z return mod(**inputs) 2025-08-26T20:34:32.3209946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3210383Z outputs = self.model( 2025-08-26T20:34:32.3210771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3211196Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3211633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3212055Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3212428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3212819Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3213224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3213677Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3214109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3214540Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3214686Z 2025-08-26T20:34:32.3214803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3215183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3215535Z return mod(**inputs) 2025-08-26T20:34:32.3215923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3216336Z outputs = self.model( 2025-08-26T20:34:32.3216725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3217135Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3217550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3217986Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3218363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3218749Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3219171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3219607Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3220059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3220500Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3220655Z 2025-08-26T20:34:32.3220744Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3220978Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3221206Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3221429Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3221675Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3222064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3222428Z return mod(**inputs) 2025-08-26T20:34:32.3222821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3223243Z outputs = self.model( 2025-08-26T20:34:32.3223656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3224096Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3224550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3224980Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3225358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3225759Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3226176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3226640Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3227092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3227539Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3228015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3228580Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3228778Z 2025-08-26T20:34:32.3228899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3229283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3229622Z return mod(**inputs) 2025-08-26T20:34:32.3229993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3230380Z outputs = self.model( 2025-08-26T20:34:32.3230748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3231135Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3231520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3231910Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3232264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3232656Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3233044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3233451Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3233860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3234281Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3234752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3235237Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3235417Z 2025-08-26T20:34:32.3235530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3235918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3236272Z return mod(**inputs) 2025-08-26T20:34:32.3236655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3237081Z outputs = self.model( 2025-08-26T20:34:32.3237476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3237908Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3238348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3238775Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3239169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3239649Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3240093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3240546Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3240987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3241391Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3241556Z 2025-08-26T20:34:32.3241664Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3242027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3242354Z return mod(**inputs) 2025-08-26T20:34:32.3242721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3243128Z outputs = self.model( 2025-08-26T20:34:32.3243500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3243894Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3244278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3244668Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3245025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3245390Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3245785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3246232Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3246412Z 2025-08-26T20:34:32.3246515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3246875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3247225Z return mod(**inputs) 2025-08-26T20:34:32.3247600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3248009Z outputs = self.model( 2025-08-26T20:34:32.3248400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3248817Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3249224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3249627Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3249978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3250346Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3250737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3251175Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3251561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3251913Z return self.act(input) 2025-08-26T20:34:32.3252032Z 2025-08-26T20:34:32.3252138Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3252504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3252825Z return mod(**inputs) 2025-08-26T20:34:32.3253212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3253614Z outputs = self.model( 2025-08-26T20:34:32.3254007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3254427Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3254811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3255213Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3255587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3255958Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3256375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3256805Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3256959Z 2025-08-26T20:34:32.3257071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3257460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3257836Z return mod(**inputs) 2025-08-26T20:34:32.3258221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3258637Z outputs = self.model( 2025-08-26T20:34:32.3259029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3259447Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3259837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3260222Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3260581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3260948Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3261343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3261750Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3262178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3262661Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3262868Z 2025-08-26T20:34:32.3262984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3263352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3263674Z return mod(**inputs) 2025-08-26T20:34:32.3264042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3264457Z outputs = self.model( 2025-08-26T20:34:32.3264854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3265248Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3265631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3266021Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3266372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3266742Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3267127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3267584Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3268028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3268463Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3268610Z 2025-08-26T20:34:32.3268729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3269109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3269459Z return mod(**inputs) 2025-08-26T20:34:32.3269862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3270268Z outputs = self.model( 2025-08-26T20:34:32.3270652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3271043Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3271437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3271831Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3272188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3272563Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3272961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3273371Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3273781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3274195Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3274341Z 2025-08-26T20:34:32.3274424Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3274645Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3274861Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3275073Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3275302Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3275696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3276072Z return mod(**inputs) 2025-08-26T20:34:32.3276465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3276886Z outputs = self.model( 2025-08-26T20:34:32.3277275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3277708Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3278135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3278567Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3278942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3279352Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3279884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3280342Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3280789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3281241Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3281738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3282273Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3282502Z 2025-08-26T20:34:32.3282628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3283025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3283381Z return mod(**inputs) 2025-08-26T20:34:32.3283781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3284208Z outputs = self.model( 2025-08-26T20:34:32.3284606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3285023Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3285459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3285889Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3286279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3286681Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3287109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3287581Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3288028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3288488Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3288976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3289458Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3289628Z 2025-08-26T20:34:32.3289734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3290104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3290438Z return mod(**inputs) 2025-08-26T20:34:32.3290800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3291201Z outputs = self.model( 2025-08-26T20:34:32.3291594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3292033Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3292451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3292836Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3293193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3293563Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3293963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3294375Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3294775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3295181Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3295329Z 2025-08-26T20:34:32.3295435Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3295800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3296121Z return mod(**inputs) 2025-08-26T20:34:32.3296677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3297071Z outputs = self.model( 2025-08-26T20:34:32.3297493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3297967Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3298353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3298751Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3299107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3299479Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3299878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3300345Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3300533Z 2025-08-26T20:34:32.3300639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3301004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3301334Z return mod(**inputs) 2025-08-26T20:34:32.3301702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3302139Z outputs = self.model( 2025-08-26T20:34:32.3302511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3302893Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3303265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3303635Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3303985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3304341Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3304727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3305152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3305531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3305936Z return self.act(input) 2025-08-26T20:34:32.3306086Z 2025-08-26T20:34:32.3306192Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3306554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3306871Z return mod(**inputs) 2025-08-26T20:34:32.3307237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3307619Z outputs = self.model( 2025-08-26T20:34:32.3307984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3308369Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3308752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3309156Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3309502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3309860Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3310241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3310643Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3310790Z 2025-08-26T20:34:32.3310896Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3311262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3311589Z return mod(**inputs) 2025-08-26T20:34:32.3311969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3312367Z outputs = self.model( 2025-08-26T20:34:32.3312728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3313116Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3313495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3313872Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3314245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3314613Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3315008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-26T20:34:32.3315401Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3315548Z 2025-08-26T20:34:32.3315655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3316044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3316418Z return mod(**inputs) 2025-08-26T20:34:32.3316809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3317212Z outputs = self.model( 2025-08-26T20:34:32.3317602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3318033Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3318443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3318861Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3319242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3319716Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3320143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3320635Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3321066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3321574Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3321807Z 2025-08-26T20:34:32.3321925Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3322320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3322676Z return mod(**inputs) 2025-08-26T20:34:32.3323071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3323494Z outputs = self.model( 2025-08-26T20:34:32.3323903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3324328Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3324746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3325167Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3325553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3325958Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3326384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3326834Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3327281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3327708Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3327854Z 2025-08-26T20:34:32.3327972Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3328361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3328704Z return mod(**inputs) 2025-08-26T20:34:32.3329092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3329518Z outputs = self.model( 2025-08-26T20:34:32.3329927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3330316Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3330708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3331099Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3331453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3331836Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3332226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3332642Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3333055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3333458Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3333602Z 2025-08-26T20:34:32.3333692Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3333905Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3334121Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3334333Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3334568Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3334928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3335284Z return mod(**inputs) 2025-08-26T20:34:32.3335655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3336053Z outputs = self.model( 2025-08-26T20:34:32.3336419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3336819Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3337221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3337645Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3338029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3338431Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3338860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3339312Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3339727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3340147Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3340600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3341096Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3341312Z 2025-08-26T20:34:32.3341421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3341804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3342157Z return mod(**inputs) 2025-08-26T20:34:32.3342537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3342930Z outputs = self.model( 2025-08-26T20:34:32.3343304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3343698Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3344098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3344495Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3344862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3345227Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3345620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3346042Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3346445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3346870Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3347303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3347752Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3347911Z 2025-08-26T20:34:32.3348013Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3348371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3348692Z return mod(**inputs) 2025-08-26T20:34:32.3349046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3349419Z outputs = self.model( 2025-08-26T20:34:32.3349788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3350213Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3350620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3351029Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3351404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3351769Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3352173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3352569Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3352961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3353390Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3353535Z 2025-08-26T20:34:32.3353639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3354001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3354328Z return mod(**inputs) 2025-08-26T20:34:32.3354690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3355131Z outputs = self.model( 2025-08-26T20:34:32.3355550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3355981Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3356406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3356825Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3357211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3357608Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3358043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3358541Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3358728Z 2025-08-26T20:34:32.3358840Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3359225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3359664Z return mod(**inputs) 2025-08-26T20:34:32.3360069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3360524Z outputs = self.model( 2025-08-26T20:34:32.3360989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3361393Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3361787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3362177Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3362522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3362887Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3363283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3363754Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3364162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3364529Z return self.act(input) 2025-08-26T20:34:32.3364677Z 2025-08-26T20:34:32.3364789Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3365179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3365525Z return mod(**inputs) 2025-08-26T20:34:32.3365894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3366322Z outputs = self.model( 2025-08-26T20:34:32.3366722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3367155Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3367569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3367985Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3368370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3368764Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3369183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3369598Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3369754Z 2025-08-26T20:34:32.3369868Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3370255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3370606Z return mod(**inputs) 2025-08-26T20:34:32.3371026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3371445Z outputs = self.model( 2025-08-26T20:34:32.3371847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3372295Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3372702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3373113Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3373495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3373886Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3374305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3374741Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3375168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3376489Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3376699Z 2025-08-26T20:34:32.3376804Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3377164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3377487Z return mod(**inputs) 2025-08-26T20:34:32.3377844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3378224Z outputs = self.model( 2025-08-26T20:34:32.3378588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3378986Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3379379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3379766Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3380123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3380521Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3380925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3381335Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3381756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3382160Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3382301Z 2025-08-26T20:34:32.3382420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3382798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3383130Z return mod(**inputs) 2025-08-26T20:34:32.3383512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3383911Z outputs = self.model( 2025-08-26T20:34:32.3384291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3384683Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3385079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3385477Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3385841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3386235Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3386622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3387031Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3387439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3387845Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3387988Z 2025-08-26T20:34:32.3388077Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3388296Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3388506Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3388726Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3388961Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3389313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3389645Z return mod(**inputs) 2025-08-26T20:34:32.3390021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3390451Z outputs = self.model( 2025-08-26T20:34:32.3390839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3391266Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3391680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3392094Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3392456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3392818Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3393236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3393676Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3394111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3394563Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3395058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3395577Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3395785Z 2025-08-26T20:34:32.3395899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3396419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3396781Z return mod(**inputs) 2025-08-26T20:34:32.3397169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3397584Z outputs = self.model( 2025-08-26T20:34:32.3397980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3398397Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3398802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3399231Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3399676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3400088Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3400521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3400961Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3401437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3401854Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3402301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3402784Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3402958Z 2025-08-26T20:34:32.3403071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3403461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3403816Z return mod(**inputs) 2025-08-26T20:34:32.3404235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3404643Z outputs = self.model( 2025-08-26T20:34:32.3405037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3405458Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3405867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3406305Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3406682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3407078Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3407503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3407935Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3408375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3408805Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3408963Z 2025-08-26T20:34:32.3409068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3409430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3409761Z return mod(**inputs) 2025-08-26T20:34:32.3410144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3410596Z outputs = self.model( 2025-08-26T20:34:32.3410998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3411435Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3411846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3412259Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3412647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3413046Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3413469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3413934Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3414133Z 2025-08-26T20:34:32.3414245Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3414636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3414986Z return mod(**inputs) 2025-08-26T20:34:32.3415381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3415788Z outputs = self.model( 2025-08-26T20:34:32.3416203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3416620Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3417025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3417452Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3417820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3418208Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3418624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3419101Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3419515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3419882Z return self.act(input) 2025-08-26T20:34:32.3420010Z 2025-08-26T20:34:32.3420122Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3420507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3420887Z return mod(**inputs) 2025-08-26T20:34:32.3421279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3421705Z outputs = self.model( 2025-08-26T20:34:32.3422111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3422543Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3422961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3423376Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3423759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3424152Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3424577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3425003Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3425181Z 2025-08-26T20:34:32.3425291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3425677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3426002Z return mod(**inputs) 2025-08-26T20:34:32.3426379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3426784Z outputs = self.model( 2025-08-26T20:34:32.3427168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3427560Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3427945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3428328Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3428680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3429043Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3429434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-26T20:34:32.3429833Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3429968Z 2025-08-26T20:34:32.3430075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3430440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3430778Z return mod(**inputs) 2025-08-26T20:34:32.3431185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3431602Z outputs = self.model( 2025-08-26T20:34:32.3432004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3432405Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3432797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3433194Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3433579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3433950Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3434345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3434780Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3435211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3435716Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3435943Z 2025-08-26T20:34:32.3436058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3436441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3436791Z return mod(**inputs) 2025-08-26T20:34:32.3437184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3437588Z outputs = self.model( 2025-08-26T20:34:32.3437979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3438410Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3438823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3439226Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3439686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3440120Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3440558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3441006Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3441437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3441865Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3442020Z 2025-08-26T20:34:32.3442145Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3442518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3442850Z return mod(**inputs) 2025-08-26T20:34:32.3443215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3443630Z outputs = self.model( 2025-08-26T20:34:32.3444017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3444446Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3444848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3445265Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3445640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3446052Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3446467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3446894Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3447338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3447777Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3447928Z 2025-08-26T20:34:32.3448024Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3448255Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3448478Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3448721Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3448976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3449367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3449727Z return mod(**inputs) 2025-08-26T20:34:32.3450131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3450575Z outputs = self.model( 2025-08-26T20:34:32.3450973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3451465Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3451877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3452296Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3452671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3453067Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3453478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3453916Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3454352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3454810Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3455309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3455827Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3456036Z 2025-08-26T20:34:32.3456153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3456548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3456908Z return mod(**inputs) 2025-08-26T20:34:32.3457301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3457721Z outputs = self.model( 2025-08-26T20:34:32.3458122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3458536Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3458933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3459325Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3459687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3460056Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3460456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3460870Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3461298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3461714Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3462163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3462626Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3462790Z 2025-08-26T20:34:32.3462902Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3463259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3463596Z return mod(**inputs) 2025-08-26T20:34:32.3464020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3464414Z outputs = self.model( 2025-08-26T20:34:32.3464778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3465190Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3465601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3466037Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3466414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3466792Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3467211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3467634Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3468039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3468433Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3468582Z 2025-08-26T20:34:32.3468693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3469086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3469442Z return mod(**inputs) 2025-08-26T20:34:32.3469827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3470252Z outputs = self.model( 2025-08-26T20:34:32.3470643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3471032Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3471420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3471813Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3472181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3472568Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3472981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3473453Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3473639Z 2025-08-26T20:34:32.3473751Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3474135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3474481Z return mod(**inputs) 2025-08-26T20:34:32.3474870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3475282Z outputs = self.model( 2025-08-26T20:34:32.3475683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3476104Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3476513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3476933Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3477321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3477710Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3478140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3478632Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3479064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3479433Z return self.act(input) 2025-08-26T20:34:32.3479641Z 2025-08-26T20:34:32.3479761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3480172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3480563Z return mod(**inputs) 2025-08-26T20:34:32.3480963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3481373Z outputs = self.model( 2025-08-26T20:34:32.3481735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3482122Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3482376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3482460Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3482678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3482767Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3483014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3483098Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3483119Z 2025-08-26T20:34:32.3483230Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3483426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3483500Z return mod(**inputs) 2025-08-26T20:34:32.3483748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3483816Z outputs = self.model( 2025-08-26T20:34:32.3484069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3484144Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3484399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3484472Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3484693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3484775Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3485022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3485121Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3485367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3485528Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3485558Z 2025-08-26T20:34:32.3485664Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3485862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3485939Z return mod(**inputs) 2025-08-26T20:34:32.3486195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3486276Z outputs = self.model( 2025-08-26T20:34:32.3486537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3486632Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3486900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3486973Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3487200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3487277Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3487528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3487638Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3487885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3487985Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3487988Z 2025-08-26T20:34:32.3488089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3488295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3488360Z return mod(**inputs) 2025-08-26T20:34:32.3488618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3488694Z outputs = self.model( 2025-08-26T20:34:32.3488950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3489033Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3489287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3489383Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3489608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3489687Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3489951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3490045Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3490311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3490399Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3490403Z 2025-08-26T20:34:32.3490485Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3490575Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3490654Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3490739Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3490843Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3491045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3491121Z return mod(**inputs) 2025-08-26T20:34:32.3491380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3491455Z outputs = self.model( 2025-08-26T20:34:32.3491740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3491815Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3492071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3492146Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3492379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3492460Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3492724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3492832Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3493085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3493194Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3493488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3493630Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3493651Z 2025-08-26T20:34:32.3493754Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3493965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3494031Z return mod(**inputs) 2025-08-26T20:34:32.3494290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3494366Z outputs = self.model( 2025-08-26T20:34:32.3494648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3494731Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3495001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3495079Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3495321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3495408Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3495701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3495796Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3496064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3496284Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3496590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3496711Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3496715Z 2025-08-26T20:34:32.3496819Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3497034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3497104Z return mod(**inputs) 2025-08-26T20:34:32.3497361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3497440Z outputs = self.model( 2025-08-26T20:34:32.3497694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3497777Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3498031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3498154Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3498388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3498467Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3498731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3498824Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3499079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3499170Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3499173Z 2025-08-26T20:34:32.3499306Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3499519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3499587Z return mod(**inputs) 2025-08-26T20:34:32.3499854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3499923Z outputs = self.model( 2025-08-26T20:34:32.3500178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3500291Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3500550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3500632Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3500855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3500936Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3501201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3501323Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3501327Z 2025-08-26T20:34:32.3501437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3501641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3501718Z return mod(**inputs) 2025-08-26T20:34:32.3502003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3502071Z outputs = self.model( 2025-08-26T20:34:32.3502334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3502409Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3502673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3502747Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3502969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3503056Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3503312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3503442Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3503657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3503730Z return self.act(input) 2025-08-26T20:34:32.3503739Z 2025-08-26T20:34:32.3503843Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3504047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3504120Z return mod(**inputs) 2025-08-26T20:34:32.3504393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3504472Z outputs = self.model( 2025-08-26T20:34:32.3504728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3504803Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3505069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3505141Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3505370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3505467Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3505721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3505813Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3505818Z 2025-08-26T20:34:32.3505921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3506129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3506246Z return mod(**inputs) 2025-08-26T20:34:32.3506511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3506580Z outputs = self.model( 2025-08-26T20:34:32.3506838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3506920Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3507179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3507264Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3507503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3507586Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3507870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-26T20:34:32.3507953Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3507979Z 2025-08-26T20:34:32.3508090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3508293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3508364Z return mod(**inputs) 2025-08-26T20:34:32.3508656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3508729Z outputs = self.model( 2025-08-26T20:34:32.3509015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3509093Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3509370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3509449Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3509688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3509781Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3510047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3510151Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3510423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3510584Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3510617Z 2025-08-26T20:34:32.3510728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3510940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3511020Z return mod(**inputs) 2025-08-26T20:34:32.3511316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3511397Z outputs = self.model( 2025-08-26T20:34:32.3511688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3511767Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3512059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3512138Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3512393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3512477Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3512746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3512869Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3513142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3513234Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3513237Z 2025-08-26T20:34:32.3513346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3513566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3513637Z return mod(**inputs) 2025-08-26T20:34:32.3513930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3514010Z outputs = self.model( 2025-08-26T20:34:32.3514292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3514378Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3514647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3514744Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3514986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3515072Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3515348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3515445Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3515715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3515814Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3515818Z 2025-08-26T20:34:32.3515904Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3516000Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3516083Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3516168Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3516286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3516501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3516578Z return mod(**inputs) 2025-08-26T20:34:32.3516871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3516949Z outputs = self.model( 2025-08-26T20:34:32.3517256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3517336Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3517642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3517724Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3517982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3518071Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3518346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3518470Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3518749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3518866Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3519190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3519337Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3519363Z 2025-08-26T20:34:32.3519546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3519775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3519853Z return mod(**inputs) 2025-08-26T20:34:32.3520128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3520210Z outputs = self.model( 2025-08-26T20:34:32.3520489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3520568Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3520863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3520940Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3521183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3521271Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3521566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3521671Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3521943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3522060Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3522375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3522505Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3522509Z 2025-08-26T20:34:32.3522619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3522834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3522918Z return mod(**inputs) 2025-08-26T20:34:32.3523190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3523270Z outputs = self.model( 2025-08-26T20:34:32.3523540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3523618Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3523897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3523994Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3524239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3524324Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3524604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3524704Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3524959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3525049Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3525053Z 2025-08-26T20:34:32.3525174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3525387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3525454Z return mod(**inputs) 2025-08-26T20:34:32.3525710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3525786Z outputs = self.model( 2025-08-26T20:34:32.3526039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3526138Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3526398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3526472Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3526706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3526786Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3527054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3527176Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3527179Z 2025-08-26T20:34:32.3527288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3527490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3527562Z return mod(**inputs) 2025-08-26T20:34:32.3527845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3527913Z outputs = self.model( 2025-08-26T20:34:32.3528176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3528252Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3528505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3528587Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3528807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3528893Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3529147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3529268Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3529489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3529560Z return self.act(input) 2025-08-26T20:34:32.3529563Z 2025-08-26T20:34:32.3529674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3529875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3529949Z return mod(**inputs) 2025-08-26T20:34:32.3530224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3530295Z outputs = self.model( 2025-08-26T20:34:32.3530556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3530631Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3530893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3530965Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3531190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3531292Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3531549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3531641Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3531645Z 2025-08-26T20:34:32.3531748Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3531956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3532049Z return mod(**inputs) 2025-08-26T20:34:32.3532317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3532396Z outputs = self.model( 2025-08-26T20:34:32.3532659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3532741Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3533003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3533076Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3533314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3533393Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3533662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3533755Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3534024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3534183Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3534188Z 2025-08-26T20:34:32.3534290Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3534499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3534566Z return mod(**inputs) 2025-08-26T20:34:32.3534832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3534901Z outputs = self.model( 2025-08-26T20:34:32.3535155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3535239Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3535492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3535574Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3535797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3535875Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3536136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3536232Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3536526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3536615Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3536620Z 2025-08-26T20:34:32.3536735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3536952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3537019Z return mod(**inputs) 2025-08-26T20:34:32.3537283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3537352Z outputs = self.model( 2025-08-26T20:34:32.3537631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3537707Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3537961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3538042Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3538263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3538367Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3538619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3538712Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3538972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3539060Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3539064Z 2025-08-26T20:34:32.3539151Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3539232Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3539317Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3539393Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3539497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3539706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3539772Z return mod(**inputs) 2025-08-26T20:34:32.3540054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3540123Z outputs = self.model( 2025-08-26T20:34:32.3540378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3540468Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3540708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3540784Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3540997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3541074Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3541326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3541418Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3541672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3541768Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3542063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3542193Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3542197Z 2025-08-26T20:34:32.3542312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3542516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3542581Z return mod(**inputs) 2025-08-26T20:34:32.3542841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3542919Z outputs = self.model( 2025-08-26T20:34:32.3543161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3543240Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3543498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3543584Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3543793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3543869Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3544118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3544220Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3544467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3544561Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3544862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3544976Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3544982Z 2025-08-26T20:34:32.3545086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3545295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3545366Z return mod(**inputs) 2025-08-26T20:34:32.3545630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3545702Z outputs = self.model( 2025-08-26T20:34:32.3545956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3546057Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3546315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3546395Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3546623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3546711Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3546968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3547061Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3547324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3547406Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3547411Z 2025-08-26T20:34:32.3547519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3547730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3547797Z return mod(**inputs) 2025-08-26T20:34:32.3548066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3548137Z outputs = self.model( 2025-08-26T20:34:32.3548401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3548493Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3548753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3548829Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3549058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3549144Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3549388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3549511Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3549514Z 2025-08-26T20:34:32.3549634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3549832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3549905Z return mod(**inputs) 2025-08-26T20:34:32.3550154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3550227Z outputs = self.model( 2025-08-26T20:34:32.3550490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3550564Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3550819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3550891Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3551118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3551197Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3551460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3551580Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3551797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3551877Z return self.act(input) 2025-08-26T20:34:32.3551880Z 2025-08-26T20:34:32.3551982Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3552209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3552275Z return mod(**inputs) 2025-08-26T20:34:32.3552529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3552607Z outputs = self.model( 2025-08-26T20:34:32.3552859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3552938Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3553194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3553265Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3553493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3553573Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3553832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3553915Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3553918Z 2025-08-26T20:34:32.3554027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3554228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3554292Z return mod(**inputs) 2025-08-26T20:34:32.3554568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3554638Z outputs = self.model( 2025-08-26T20:34:32.3554902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3554980Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3555251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3555335Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3555567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3555682Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3555952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-26T20:34:32.3556047Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3556051Z 2025-08-26T20:34:32.3556743Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3556964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3557063Z return mod(**inputs) 2025-08-26T20:34:32.3557344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3557429Z outputs = self.model( 2025-08-26T20:34:32.3557709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3557790Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3558090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3558168Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3558414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3558499Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3558774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3558873Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3559160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3559329Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3559333Z 2025-08-26T20:34:32.3559443Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3559746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3559821Z return mod(**inputs) 2025-08-26T20:34:32.3560097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3560181Z outputs = self.model( 2025-08-26T20:34:32.3560458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3560548Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3560830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3560909Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3561151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3561235Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3561514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3561614Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3561912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3562001Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3562006Z 2025-08-26T20:34:32.3562116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3562338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3562411Z return mod(**inputs) 2025-08-26T20:34:32.3562693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3562767Z outputs = self.model( 2025-08-26T20:34:32.3563055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3563145Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3563420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3563506Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3563746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3563859Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3564129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3564228Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3564504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3564597Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3564601Z 2025-08-26T20:34:32.3564695Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3564781Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3564864Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3564954Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3565064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3565285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3565359Z return mod(**inputs) 2025-08-26T20:34:32.3565647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3565726Z outputs = self.model( 2025-08-26T20:34:32.3565995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3566080Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3566350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3566427Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3566672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3566758Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3567039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3567137Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3567415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3567520Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3567833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3567983Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3567987Z 2025-08-26T20:34:32.3568115Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3568341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3568408Z return mod(**inputs) 2025-08-26T20:34:32.3568670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3568748Z outputs = self.model( 2025-08-26T20:34:32.3569012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3569097Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3569384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3569468Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3569707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3569788Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3570052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3570159Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3570422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3570522Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3570820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3570946Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3570952Z 2025-08-26T20:34:32.3571062Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3571284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3571356Z return mod(**inputs) 2025-08-26T20:34:32.3571634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3571708Z outputs = self.model( 2025-08-26T20:34:32.3571996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3572103Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3572375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3572458Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3572704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3572790Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3573074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3573170Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3573450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3573539Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3573545Z 2025-08-26T20:34:32.3573660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3573876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3573946Z return mod(**inputs) 2025-08-26T20:34:32.3574242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3574317Z outputs = self.model( 2025-08-26T20:34:32.3574618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3574748Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3575019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3575104Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3575348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3575441Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3575709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3575836Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3575840Z 2025-08-26T20:34:32.3575979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3576194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3576273Z return mod(**inputs) 2025-08-26T20:34:32.3576563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3576644Z outputs = self.model( 2025-08-26T20:34:32.3576945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3577023Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3577301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3577379Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3577633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3577716Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3577989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3578123Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3578351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3578432Z return self.act(input) 2025-08-26T20:34:32.3578436Z 2025-08-26T20:34:32.3578544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3578774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3578851Z return mod(**inputs) 2025-08-26T20:34:32.3579132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3579212Z outputs = self.model( 2025-08-26T20:34:32.3579500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3579585Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3579854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3579931Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3580172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3580258Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3580531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3580619Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3580623Z 2025-08-26T20:34:32.3580731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3580953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3581023Z return mod(**inputs) 2025-08-26T20:34:32.3581347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3581421Z outputs = self.model( 2025-08-26T20:34:32.3581692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3581779Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3582051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3582140Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3582376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3582482Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3582751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3582852Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3583129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3583289Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3583310Z 2025-08-26T20:34:32.3583424Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3583637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3583707Z return mod(**inputs) 2025-08-26T20:34:32.3583985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3584057Z outputs = self.model( 2025-08-26T20:34:32.3584337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3584414Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3584690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3584766Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3585004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3585094Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3585382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3585483Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3585755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3585840Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3585844Z 2025-08-26T20:34:32.3585959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3586171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3586249Z return mod(**inputs) 2025-08-26T20:34:32.3586523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3586602Z outputs = self.model( 2025-08-26T20:34:32.3586873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3586951Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3587225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3587302Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3587545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3587629Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3587909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3588016Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3588290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3588391Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3588395Z 2025-08-26T20:34:32.3588481Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3588567Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3588656Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3588736Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3588866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3589080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3589152Z return mod(**inputs) 2025-08-26T20:34:32.3589434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3589508Z outputs = self.model( 2025-08-26T20:34:32.3589804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3589884Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3590161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3590238Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3590476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3590569Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3590841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3590948Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3591218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3591326Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3591645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3591805Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3591809Z 2025-08-26T20:34:32.3591927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3592144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3592223Z return mod(**inputs) 2025-08-26T20:34:32.3592500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3592574Z outputs = self.model( 2025-08-26T20:34:32.3592854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3592934Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3593213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3593291Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3593528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3593618Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3593887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3593991Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3594277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3594384Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3594700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3594821Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3594827Z 2025-08-26T20:34:32.3594943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3595154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3595233Z return mod(**inputs) 2025-08-26T20:34:32.3595537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3595613Z outputs = self.model( 2025-08-26T20:34:32.3595896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3595974Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3596374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3596500Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3596738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3596834Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3597113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-26T20:34:32.3597220Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:34:32.3597490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3597585Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3597591Z 2025-08-26T20:34:32.3597702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3597918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3598000Z return mod(**inputs) 2025-08-26T20:34:32.3598276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3598384Z outputs = self.model( 2025-08-26T20:34:32.3598664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3598745Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3599039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3599119Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3599377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3599511Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3599799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3599940Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3599946Z 2025-08-26T20:34:32.3600059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3600284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3600355Z return mod(**inputs) 2025-08-26T20:34:32.3600643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3600719Z outputs = self.model( 2025-08-26T20:34:32.3601043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3601131Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3601406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3601496Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3601742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3601832Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3602119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-26T20:34:32.3602249Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3602520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3602599Z return self.act(input) 2025-08-26T20:34:32.3602603Z 2025-08-26T20:34:32.3602725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3602945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3603017Z return mod(**inputs) 2025-08-26T20:34:32.3603320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3603397Z outputs = self.model( 2025-08-26T20:34:32.3603681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3603761Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3604039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3604127Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3604372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3604470Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3604745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-26T20:34:32.3604836Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3604846Z 2025-08-26T20:34:32.3604959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3605206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3605286Z return mod(**inputs) 2025-08-26T20:34:32.3605568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3605651Z outputs = self.model( 2025-08-26T20:34:32.3605929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-26T20:34:32.3606010Z encoder_outputs = self.encoder( 2025-08-26T20:34:32.3606298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-26T20:34:32.3606377Z layer_outputs = encoder_layer( 2025-08-26T20:34:32.3606627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3606718Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3606995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-26T20:34:32.3607090Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3607094Z 2025-08-26T20:34:32.3607206Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3607436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3607508Z return mod(**inputs) 2025-08-26T20:34:32.3607804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3607888Z outputs = self.model( 2025-08-26T20:34:32.3608165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3608247Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3608503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3608582Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3608806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3608902Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3609172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3609276Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3609533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3609685Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3609707Z 2025-08-26T20:34:32.3609819Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3610022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3610089Z return mod(**inputs) 2025-08-26T20:34:32.3610350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3610421Z outputs = self.model( 2025-08-26T20:34:32.3610681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3610755Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3611009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3611090Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3611314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3611402Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3611672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3611775Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3612038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3612119Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3612123Z 2025-08-26T20:34:32.3612232Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3612434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3612507Z return mod(**inputs) 2025-08-26T20:34:32.3612762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3612834Z outputs = self.model( 2025-08-26T20:34:32.3613096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3613170Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3613433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3613506Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3613729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3613837Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3614094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3614199Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3614455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3614544Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3614554Z 2025-08-26T20:34:32.3614636Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3614716Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3614802Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3614879Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3614996Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3615209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3615277Z return mod(**inputs) 2025-08-26T20:34:32.3615538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3615606Z outputs = self.model( 2025-08-26T20:34:32.3615893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3615969Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3616226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3616306Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3616527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3616614Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3616869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3616968Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3617229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3617330Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3617629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3617787Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3617791Z 2025-08-26T20:34:32.3617899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3618105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3618171Z return mod(**inputs) 2025-08-26T20:34:32.3618440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3618510Z outputs = self.model( 2025-08-26T20:34:32.3618784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3618860Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3619120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3619203Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3619429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3619515Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3619774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3619873Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3620158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3620257Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3620561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3620677Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3620681Z 2025-08-26T20:34:32.3620792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3620996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3621063Z return mod(**inputs) 2025-08-26T20:34:32.3621350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3621423Z outputs = self.model( 2025-08-26T20:34:32.3621689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3621762Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3622015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3622115Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3622349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3622441Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3622716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3622829Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3623102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3623187Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3623190Z 2025-08-26T20:34:32.3623302Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3623505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3623581Z return mod(**inputs) 2025-08-26T20:34:32.3623838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3623928Z outputs = self.model( 2025-08-26T20:34:32.3624190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3624262Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3624525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3624597Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3624821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3624906Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3625163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3625285Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3625545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3625711Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3625714Z 2025-08-26T20:34:32.3625822Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3626026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3626100Z return mod(**inputs) 2025-08-26T20:34:32.3626384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3626463Z outputs = self.model( 2025-08-26T20:34:32.3626716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3626790Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3627054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3627125Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3627354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3627453Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3627716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3627836Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3628086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3628174Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3628202Z 2025-08-26T20:34:32.3628302Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3628504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3628568Z return mod(**inputs) 2025-08-26T20:34:32.3628814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3628889Z outputs = self.model( 2025-08-26T20:34:32.3629135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3629216Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3629467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3629545Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3629765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3629843Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3630123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3630232Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3630501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3630585Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3630589Z 2025-08-26T20:34:32.3630667Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3630758Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3630833Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3630917Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3631019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3631219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3631296Z return mod(**inputs) 2025-08-26T20:34:32.3631549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3631624Z outputs = self.model( 2025-08-26T20:34:32.3631880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3631953Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3632217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3632307Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3632539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3632617Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3632878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3632988Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3633241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3633348Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3633655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3633800Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3633806Z 2025-08-26T20:34:32.3633909Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3634119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3634215Z return mod(**inputs) 2025-08-26T20:34:32.3634507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3634590Z outputs = self.model( 2025-08-26T20:34:32.3634878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3634963Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3635255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3635335Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3635581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3635664Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3635937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3636054Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3636367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3636477Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3636785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3636909Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3636913Z 2025-08-26T20:34:32.3637023Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3637247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3637317Z return mod(**inputs) 2025-08-26T20:34:32.3637610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3637692Z outputs = self.model( 2025-08-26T20:34:32.3637984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3638072Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3638350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3638427Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3638681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3638765Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3639056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3639171Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3639440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3639803Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3639809Z 2025-08-26T20:34:32.3639922Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3640148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3640221Z return mod(**inputs) 2025-08-26T20:34:32.3640522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3640598Z outputs = self.model( 2025-08-26T20:34:32.3640905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3640988Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3641239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3641342Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3641561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3641637Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3641892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3642013Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3642017Z 2025-08-26T20:34:32.3642124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3642321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3642393Z return mod(**inputs) 2025-08-26T20:34:32.3642643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3642711Z outputs = self.model( 2025-08-26T20:34:32.3642968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3643061Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3643320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3643389Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3643608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3643693Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3643941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3644068Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3644276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3644346Z return self.act(input) 2025-08-26T20:34:32.3644359Z 2025-08-26T20:34:32.3644459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3644652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3644725Z return mod(**inputs) 2025-08-26T20:34:32.3644972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3645046Z outputs = self.model( 2025-08-26T20:34:32.3645310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3645382Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3645638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3645710Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3645933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3646015Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3646268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3646359Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3646379Z 2025-08-26T20:34:32.3646482Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3646689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3646758Z return mod(**inputs) 2025-08-26T20:34:32.3647013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3647088Z outputs = self.model( 2025-08-26T20:34:32.3647360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3647443Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3647699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3647776Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3648012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3648098Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3648378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3648495Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3648755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3648910Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3648931Z 2025-08-26T20:34:32.3649042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3649243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3649310Z return mod(**inputs) 2025-08-26T20:34:32.3649576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3649645Z outputs = self.model( 2025-08-26T20:34:32.3649909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3649984Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3650237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3650320Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3650543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3650632Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3650886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3650986Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3651251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3651333Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3651337Z 2025-08-26T20:34:32.3651466Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3651668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3651742Z return mod(**inputs) 2025-08-26T20:34:32.3651997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3652066Z outputs = self.model( 2025-08-26T20:34:32.3652327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3652400Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3652686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3652761Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3652987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3653074Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3653327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3653452Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3653707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3653794Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3653806Z 2025-08-26T20:34:32.3653887Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3653967Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3654055Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3654131Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3654233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3654442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3654509Z return mod(**inputs) 2025-08-26T20:34:32.3654772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3654841Z outputs = self.model( 2025-08-26T20:34:32.3655099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3655189Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3655444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3655524Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3655749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3655835Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3656091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3656191Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3656458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3656566Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3656888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3657032Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3657036Z 2025-08-26T20:34:32.3657158Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3657362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3657429Z return mod(**inputs) 2025-08-26T20:34:32.3657732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3657802Z outputs = self.model( 2025-08-26T20:34:32.3658061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3658136Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3658393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3658477Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3658709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3658821Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3659097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3659197Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3659457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3659553Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3659886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3660003Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3660007Z 2025-08-26T20:34:32.3660121Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3660343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3660410Z return mod(**inputs) 2025-08-26T20:34:32.3660670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3660739Z outputs = self.model( 2025-08-26T20:34:32.3661000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3661075Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3661332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3661429Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3661660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3661752Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3662027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3662139Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3662410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3662497Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3662500Z 2025-08-26T20:34:32.3662618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3662834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3662913Z return mod(**inputs) 2025-08-26T20:34:32.3663186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3663259Z outputs = self.model( 2025-08-26T20:34:32.3663540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3663616Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3663882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3663968Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3664191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3664276Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3664532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-26T20:34:32.3664627Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3664631Z 2025-08-26T20:34:32.3664732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3664940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3665007Z return mod(**inputs) 2025-08-26T20:34:32.3665280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3665359Z outputs = self.model( 2025-08-26T20:34:32.3665616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3665697Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3665953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3666044Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3666280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3666359Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3666624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3666737Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3667001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3667158Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3667162Z 2025-08-26T20:34:32.3667271Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3667497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3667567Z return mod(**inputs) 2025-08-26T20:34:32.3667869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3667939Z outputs = self.model( 2025-08-26T20:34:32.3668194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3668278Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3668532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3668612Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3668841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3668928Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3669183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3669293Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3669552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3669633Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3669636Z 2025-08-26T20:34:32.3669746Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3669946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3670013Z return mod(**inputs) 2025-08-26T20:34:32.3670293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3670364Z outputs = self.model( 2025-08-26T20:34:32.3670629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3670706Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3670960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3671041Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3671279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3671367Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3671623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3671742Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3671997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3672101Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3672105Z 2025-08-26T20:34:32.3672197Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3672279Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3672363Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3672441Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3672544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3672759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3672829Z return mod(**inputs) 2025-08-26T20:34:32.3673107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3673184Z outputs = self.model( 2025-08-26T20:34:32.3673455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3673543Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3673810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3673951Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3674187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3674270Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3674549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3674664Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3674945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3675050Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3675367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3675512Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3675517Z 2025-08-26T20:34:32.3675628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3675847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3675917Z return mod(**inputs) 2025-08-26T20:34:32.3676211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3695801Z outputs = self.model( 2025-08-26T20:34:32.3696647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3696758Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3697042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3697139Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3697382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3697480Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3697744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3697909Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3698179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3698288Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3698597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3698759Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3698765Z 2025-08-26T20:34:32.3698886Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3699125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3699205Z return mod(**inputs) 2025-08-26T20:34:32.3699491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3699575Z outputs = self.model( 2025-08-26T20:34:32.3699855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3699939Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3700213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3700304Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3700548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3700697Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3700968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3701087Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3701371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3701468Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3701473Z 2025-08-26T20:34:32.3701591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3701801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3701870Z return mod(**inputs) 2025-08-26T20:34:32.3702134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3702209Z outputs = self.model( 2025-08-26T20:34:32.3702475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3702552Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3702818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3702895Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3703120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3703226Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3703484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3703619Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3703625Z 2025-08-26T20:34:32.3703734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3703944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3704021Z return mod(**inputs) 2025-08-26T20:34:32.3704276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3704351Z outputs = self.model( 2025-08-26T20:34:32.3704886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3704972Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3705230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3705303Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3705532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3705631Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3705893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3706017Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3706237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3706315Z return self.act(input) 2025-08-26T20:34:32.3706319Z 2025-08-26T20:34:32.3706424Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3706644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3706715Z return mod(**inputs) 2025-08-26T20:34:32.3706982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3707053Z outputs = self.model( 2025-08-26T20:34:32.3707313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3707420Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3707688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3707771Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3708006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3708092Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3708374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3708463Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3708468Z 2025-08-26T20:34:32.3708588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3708800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3708878Z return mod(**inputs) 2025-08-26T20:34:32.3709146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3709230Z outputs = self.model( 2025-08-26T20:34:32.3709489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3709564Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3709845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3709925Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3710162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3710253Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3710524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3710640Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3710913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3711097Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3711108Z 2025-08-26T20:34:32.3711222Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3711437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3711517Z return mod(**inputs) 2025-08-26T20:34:32.3711789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3711891Z outputs = self.model( 2025-08-26T20:34:32.3712160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3712240Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3712515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3712592Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3712833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3712917Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3713187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3713301Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3713568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3713662Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3713685Z 2025-08-26T20:34:32.3713796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3714015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3714085Z return mod(**inputs) 2025-08-26T20:34:32.3714356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3714437Z outputs = self.model( 2025-08-26T20:34:32.3714714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3714798Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3715066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3715144Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3715387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3715473Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3715745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3715852Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3716118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3716219Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3716243Z 2025-08-26T20:34:32.3716336Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3716434Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3716517Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3716608Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3716719Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3716936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3717014Z return mod(**inputs) 2025-08-26T20:34:32.3717288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3717367Z outputs = self.model( 2025-08-26T20:34:32.3717654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3717734Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3718011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3718088Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3718330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3718444Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3718724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3718840Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3719115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3719230Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3719648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3719816Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3719821Z 2025-08-26T20:34:32.3719937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3720160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3720267Z return mod(**inputs) 2025-08-26T20:34:32.3720550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3720643Z outputs = self.model( 2025-08-26T20:34:32.3720915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3720994Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3721275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3721354Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3721595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3721679Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3721948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3722064Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3722331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3722445Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3722761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3722888Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3722911Z 2025-08-26T20:34:32.3723023Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3723239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3723319Z return mod(**inputs) 2025-08-26T20:34:32.3723595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3723671Z outputs = self.model( 2025-08-26T20:34:32.3723920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3723993Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3724264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3724340Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3724573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3724653Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3724914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3725031Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3725285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3725380Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3725384Z 2025-08-26T20:34:32.3725486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3725696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3725763Z return mod(**inputs) 2025-08-26T20:34:32.3726019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3726094Z outputs = self.model( 2025-08-26T20:34:32.3726360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3726447Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3726714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3726817Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3727044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3727122Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3727381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3727491Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3727751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3727903Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3727907Z 2025-08-26T20:34:32.3728020Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3728222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3728288Z return mod(**inputs) 2025-08-26T20:34:32.3728540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3728606Z outputs = self.model( 2025-08-26T20:34:32.3728852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3728930Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3729192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3729273Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3729491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3729580Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3729828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3729938Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3730197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3730293Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3730297Z 2025-08-26T20:34:32.3730406Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3730604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3730672Z return mod(**inputs) 2025-08-26T20:34:32.3730928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3731014Z outputs = self.model( 2025-08-26T20:34:32.3731272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3731344Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3731599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3731671Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3731891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3731977Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3732227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3732340Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3732589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3732679Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3732699Z 2025-08-26T20:34:32.3732789Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3732866Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3732950Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3733025Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3733129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3733333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3733398Z return mod(**inputs) 2025-08-26T20:34:32.3733655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3733723Z outputs = self.model( 2025-08-26T20:34:32.3733971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3734052Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3734301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3734379Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3734595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3734681Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3734926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3735046Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3735303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3735403Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3735702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3735840Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3735844Z 2025-08-26T20:34:32.3735947Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3736153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3736236Z return mod(**inputs) 2025-08-26T20:34:32.3736499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3736572Z outputs = self.model( 2025-08-26T20:34:32.3736835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3736908Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3737194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3737276Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3737491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3737575Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3737826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3737931Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3738189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3738285Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3738575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3738684Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3738705Z 2025-08-26T20:34:32.3738815Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3739010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3739076Z return mod(**inputs) 2025-08-26T20:34:32.3739333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3739400Z outputs = self.model( 2025-08-26T20:34:32.3739657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3739729Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3739978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3740059Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3740275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3740361Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3740607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3740709Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3740966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3741049Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3741052Z 2025-08-26T20:34:32.3741174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3741370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3741443Z return mod(**inputs) 2025-08-26T20:34:32.3741692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3741761Z outputs = self.model( 2025-08-26T20:34:32.3742017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3742089Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3742378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3742453Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3742682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3742767Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3743030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-26T20:34:32.3743140Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3743143Z 2025-08-26T20:34:32.3743248Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3743455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3743521Z return mod(**inputs) 2025-08-26T20:34:32.3743778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3743855Z outputs = self.model( 2025-08-26T20:34:32.3744112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3744196Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3744449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3744521Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3744752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3744848Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3745109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3745233Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3745237Z 2025-08-26T20:34:32.3745342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3745553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3745619Z return mod(**inputs) 2025-08-26T20:34:32.3745888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3745958Z outputs = self.model( 2025-08-26T20:34:32.3746227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3746303Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3746562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3746644Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3746874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3746962Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3747221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3747358Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3747585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3747659Z return self.act(input) 2025-08-26T20:34:32.3747663Z 2025-08-26T20:34:32.3747772Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3747973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3748044Z return mod(**inputs) 2025-08-26T20:34:32.3748300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3748370Z outputs = self.model( 2025-08-26T20:34:32.3748648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3748723Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3748985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3749058Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3749278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3749384Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3749643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3749739Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3749743Z 2025-08-26T20:34:32.3749848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3750053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3750130Z return mod(**inputs) 2025-08-26T20:34:32.3750393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3750475Z outputs = self.model( 2025-08-26T20:34:32.3750732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3750818Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3751077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3751168Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3751397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3751475Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3751739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3751840Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3752096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3752255Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3752261Z 2025-08-26T20:34:32.3752363Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3752570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3752636Z return mod(**inputs) 2025-08-26T20:34:32.3752897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3752965Z outputs = self.model( 2025-08-26T20:34:32.3753220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3753301Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3753571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3753653Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3753877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3753959Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3754224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3754324Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3754604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3754688Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3754691Z 2025-08-26T20:34:32.3754794Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3755001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3755067Z return mod(**inputs) 2025-08-26T20:34:32.3755327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3755417Z outputs = self.model( 2025-08-26T20:34:32.3755693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3755773Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3756039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3756124Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3756358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3756449Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3756716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3756821Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3757098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3757207Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3757211Z 2025-08-26T20:34:32.3757308Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3757396Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3757481Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3757573Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3757686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3757910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3757983Z return mod(**inputs) 2025-08-26T20:34:32.3758259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3758344Z outputs = self.model( 2025-08-26T20:34:32.3758621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3758712Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3758990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3759078Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3759318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3759405Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3759768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3759902Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3760190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3760298Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3760619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3760780Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3760785Z 2025-08-26T20:34:32.3760898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3761151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3761225Z return mod(**inputs) 2025-08-26T20:34:32.3761507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3761582Z outputs = self.model( 2025-08-26T20:34:32.3761850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3761965Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3762232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3762321Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3762556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3762641Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3762917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3763021Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3763298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3763400Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3763717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3763835Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3763861Z 2025-08-26T20:34:32.3763971Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3764192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3764261Z return mod(**inputs) 2025-08-26T20:34:32.3764540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3764614Z outputs = self.model( 2025-08-26T20:34:32.3764885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3764971Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3765239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3765324Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3765557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3765644Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3765919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3766024Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3766298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3766410Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3766415Z 2025-08-26T20:34:32.3766531Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3766748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3766820Z return mod(**inputs) 2025-08-26T20:34:32.3767097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3767171Z outputs = self.model( 2025-08-26T20:34:32.3767444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3767522Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3767814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3767892Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3768127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3768218Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3768482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3768624Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3768892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3769052Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3769063Z 2025-08-26T20:34:32.3769174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3769390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3769466Z return mod(**inputs) 2025-08-26T20:34:32.3769738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3769820Z outputs = self.model( 2025-08-26T20:34:32.3770093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3770167Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3770448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3770521Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3770748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3770828Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3771082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3771202Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3771455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3771546Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3771551Z 2025-08-26T20:34:32.3771655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3771865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3771932Z return mod(**inputs) 2025-08-26T20:34:32.3772187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3772262Z outputs = self.model( 2025-08-26T20:34:32.3772529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3772613Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3772900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3772978Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3773219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3773303Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3773581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3773703Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3773958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3774071Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3774075Z 2025-08-26T20:34:32.3774159Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3774245Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3774326Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3774404Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3774518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3774736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3774809Z return mod(**inputs) 2025-08-26T20:34:32.3775065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3775140Z outputs = self.model( 2025-08-26T20:34:32.3775396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3775472Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3775734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3775807Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3776038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3776119Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3776373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3776508Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3776761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3776867Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3777164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3777299Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3777310Z 2025-08-26T20:34:32.3777415Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3777617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3777696Z return mod(**inputs) 2025-08-26T20:34:32.3777953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3778032Z outputs = self.model( 2025-08-26T20:34:32.3778283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3778358Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3778622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3778694Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3778938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3779018Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3779277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3779396Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3779651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3779755Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3780047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3780180Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3780184Z 2025-08-26T20:34:32.3780288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3780491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3780567Z return mod(**inputs) 2025-08-26T20:34:32.3780828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3780920Z outputs = self.model( 2025-08-26T20:34:32.3781177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3781253Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3781520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3781592Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3781823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3781903Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3782167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3782276Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3782535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3782627Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3782647Z 2025-08-26T20:34:32.3782753Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3782967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3783037Z return mod(**inputs) 2025-08-26T20:34:32.3783298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3783380Z outputs = self.model( 2025-08-26T20:34:32.3783641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3783727Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3783986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3784065Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3784302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3784387Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3784654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3784781Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3784786Z 2025-08-26T20:34:32.3784900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3785162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3785231Z return mod(**inputs) 2025-08-26T20:34:32.3785495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3785566Z outputs = self.model( 2025-08-26T20:34:32.3785828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3785904Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3786158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3786237Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3786476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3786566Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3786824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3786955Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3787174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3787265Z return self.act(input) 2025-08-26T20:34:32.3787270Z 2025-08-26T20:34:32.3787379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3787592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3787670Z return mod(**inputs) 2025-08-26T20:34:32.3787965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3788038Z outputs = self.model( 2025-08-26T20:34:32.3788329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3788406Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3788692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3788763Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3788979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3789092Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3789336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3789426Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3789430Z 2025-08-26T20:34:32.3789532Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3789731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3789798Z return mod(**inputs) 2025-08-26T20:34:32.3790049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3790126Z outputs = self.model( 2025-08-26T20:34:32.3790379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3790461Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3790712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3790783Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3791013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3791093Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3791369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:34:32.3791452Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3791456Z 2025-08-26T20:34:32.3791565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3791766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3791835Z return mod(**inputs) 2025-08-26T20:34:32.3792095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3792164Z outputs = self.model( 2025-08-26T20:34:32.3792424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3792515Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3792774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3792854Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3793078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3793165Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3793435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3793537Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3793809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3793956Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3793959Z 2025-08-26T20:34:32.3794068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3794265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3794336Z return mod(**inputs) 2025-08-26T20:34:32.3794585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3794649Z outputs = self.model( 2025-08-26T20:34:32.3794905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3795003Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3795260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3795332Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3795561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3795651Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3795918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3796031Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3796431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3796534Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3796538Z 2025-08-26T20:34:32.3796648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3796861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3796940Z return mod(**inputs) 2025-08-26T20:34:32.3797211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3797293Z outputs = self.model( 2025-08-26T20:34:32.3797678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3797817Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3798102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3798182Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3798433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3798522Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3798801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3798919Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3799223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3799329Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3799333Z 2025-08-26T20:34:32.3799424Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3799580Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3799674Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3799758Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3799881Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3800135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3800216Z return mod(**inputs) 2025-08-26T20:34:32.3800493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3800568Z outputs = self.model( 2025-08-26T20:34:32.3800855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3800940Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3801196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3801268Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3801484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3801572Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3801819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3801953Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3802202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3802297Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3802591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3802723Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3802727Z 2025-08-26T20:34:32.3802834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3803030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3803104Z return mod(**inputs) 2025-08-26T20:34:32.3803354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3803421Z outputs = self.model( 2025-08-26T20:34:32.3803676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3803748Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3804003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3804073Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3804306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3804394Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3804643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3804747Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3804998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3805100Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3805405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3805513Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3805517Z 2025-08-26T20:34:32.3805625Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3805823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3805896Z return mod(**inputs) 2025-08-26T20:34:32.3806144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3806229Z outputs = self.model( 2025-08-26T20:34:32.3806491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3806563Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3806829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3806903Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3807141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3807219Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3807469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3807572Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3807829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3807936Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3807940Z 2025-08-26T20:34:32.3808042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3808241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3808314Z return mod(**inputs) 2025-08-26T20:34:32.3808573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3808650Z outputs = self.model( 2025-08-26T20:34:32.3808904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3808976Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3809235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3809308Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3809536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3809614Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3809873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3809985Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3810237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3810411Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3810416Z 2025-08-26T20:34:32.3810521Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3810730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3810798Z return mod(**inputs) 2025-08-26T20:34:32.3811056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3811133Z outputs = self.model( 2025-08-26T20:34:32.3811390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3811488Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3811745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3811825Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3812045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3812124Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3812414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3812523Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3812783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3812863Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3812867Z 2025-08-26T20:34:32.3812970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3813178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3813244Z return mod(**inputs) 2025-08-26T20:34:32.3813505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3813574Z outputs = self.model( 2025-08-26T20:34:32.3813834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3813909Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3814183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3814264Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3814486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3814574Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3814827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3814936Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3815197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3815288Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3815292Z 2025-08-26T20:34:32.3815382Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3815466Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3815544Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3815629Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3815734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3815942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3816009Z return mod(**inputs) 2025-08-26T20:34:32.3816265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3816365Z outputs = self.model( 2025-08-26T20:34:32.3816648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3816733Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3817004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3817081Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3817325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3817408Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3817714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3817824Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3818088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3818188Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3818484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3818642Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3818648Z 2025-08-26T20:34:32.3818752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3818960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3819026Z return mod(**inputs) 2025-08-26T20:34:32.3819283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3819359Z outputs = self.model( 2025-08-26T20:34:32.3819618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3819700Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3819958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3820040Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3820264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3820361Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3820622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3820732Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3821005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3821112Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3821424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3821545Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3821550Z 2025-08-26T20:34:32.3821659Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3821883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3821952Z return mod(**inputs) 2025-08-26T20:34:32.3822227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3822299Z outputs = self.model( 2025-08-26T20:34:32.3822566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3822651Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3822935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3823018Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3823239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3823319Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3823582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3823687Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3823962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3824047Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3824051Z 2025-08-26T20:34:32.3824160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3824362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3824428Z return mod(**inputs) 2025-08-26T20:34:32.3824691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3824776Z outputs = self.model( 2025-08-26T20:34:32.3825041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3825113Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3825368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3825448Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3825673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3825759Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3826015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3826138Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3826151Z 2025-08-26T20:34:32.3826255Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3826470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3826543Z return mod(**inputs) 2025-08-26T20:34:32.3826803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3826883Z outputs = self.model( 2025-08-26T20:34:32.3827149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3827226Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3827500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3827575Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3827813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3827897Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3828160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3828289Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3828512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3828595Z return self.act(input) 2025-08-26T20:34:32.3828599Z 2025-08-26T20:34:32.3828706Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3828940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3829011Z return mod(**inputs) 2025-08-26T20:34:32.3829265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3829344Z outputs = self.model( 2025-08-26T20:34:32.3829600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3829681Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3829938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3830009Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3830259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3830341Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3830604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3830687Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3830691Z 2025-08-26T20:34:32.3830797Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3831020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3831089Z return mod(**inputs) 2025-08-26T20:34:32.3831349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3831417Z outputs = self.model( 2025-08-26T20:34:32.3831676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3831749Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3832002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3832080Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3832300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3832390Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3832645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3832762Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3833023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3833176Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3833179Z 2025-08-26T20:34:32.3833287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3833489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3833563Z return mod(**inputs) 2025-08-26T20:34:32.3833822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3833894Z outputs = self.model( 2025-08-26T20:34:32.3834156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3834231Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3834500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3834575Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3834811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3834903Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3835192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3835308Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3835580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3835669Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3835683Z 2025-08-26T20:34:32.3835793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3836004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3836082Z return mod(**inputs) 2025-08-26T20:34:32.3836379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3836459Z outputs = self.model( 2025-08-26T20:34:32.3836730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3836808Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3837087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3837185Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3837426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3837511Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3837778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3837891Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3838160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3838262Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3838268Z 2025-08-26T20:34:32.3838354Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3838446Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3838527Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3838608Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3838725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3838975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3839046Z return mod(**inputs) 2025-08-26T20:34:32.3839318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3839391Z outputs = self.model( 2025-08-26T20:34:32.3839746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3839830Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3840117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3840198Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3840440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3840537Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3840815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3840929Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3841205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3841316Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3841677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3841823Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3841828Z 2025-08-26T20:34:32.3841946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3842161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3842241Z return mod(**inputs) 2025-08-26T20:34:32.3842515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3842587Z outputs = self.model( 2025-08-26T20:34:32.3842866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3842960Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3843238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3843321Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3843560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3843656Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3843939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3844049Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3844317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3844425Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3844736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3844852Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3844856Z 2025-08-26T20:34:32.3844972Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3845182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3845258Z return mod(**inputs) 2025-08-26T20:34:32.3845527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3845620Z outputs = self.model( 2025-08-26T20:34:32.3845898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3845976Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3846254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3846331Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3846569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3846662Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3846929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3847042Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3847315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3847411Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3847415Z 2025-08-26T20:34:32.3847524Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3847739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3847818Z return mod(**inputs) 2025-08-26T20:34:32.3848087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3848187Z outputs = self.model( 2025-08-26T20:34:32.3848460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3848537Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3848814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3848892Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3849133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3849216Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3849510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-26T20:34:32.3849599Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3849603Z 2025-08-26T20:34:32.3849714Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3849933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3850003Z return mod(**inputs) 2025-08-26T20:34:32.3850277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3850368Z outputs = self.model( 2025-08-26T20:34:32.3850637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3850723Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3850991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3851076Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3851311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3851396Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3851672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3851789Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3852065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3852242Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3852247Z 2025-08-26T20:34:32.3852364Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3852579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3852650Z return mod(**inputs) 2025-08-26T20:34:32.3852929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3853004Z outputs = self.model( 2025-08-26T20:34:32.3853279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3853351Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3853609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3853688Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3853923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3854013Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3854284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3854406Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3854696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3854785Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3854789Z 2025-08-26T20:34:32.3854903Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3855117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3855196Z return mod(**inputs) 2025-08-26T20:34:32.3855462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3855534Z outputs = self.model( 2025-08-26T20:34:32.3855825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3855903Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3856182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3856261Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3856498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3856588Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3856875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3857000Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3857269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3857369Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3857373Z 2025-08-26T20:34:32.3857460Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3857546Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3857638Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3857720Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3857836Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3858057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3858125Z return mod(**inputs) 2025-08-26T20:34:32.3858389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3858475Z outputs = self.model( 2025-08-26T20:34:32.3858739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3858812Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3859071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3859150Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3859372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3859460Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3859714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3859832Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3860091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3860192Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3860496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3860632Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3860636Z 2025-08-26T20:34:32.3860748Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3860978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3861048Z return mod(**inputs) 2025-08-26T20:34:32.3861328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3861404Z outputs = self.model( 2025-08-26T20:34:32.3861679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3861757Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3862033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3862129Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3862363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3862455Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3862722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3862843Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3863140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3863240Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3863541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3863648Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3863652Z 2025-08-26T20:34:32.3863762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3863961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3864036Z return mod(**inputs) 2025-08-26T20:34:32.3864293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3864361Z outputs = self.model( 2025-08-26T20:34:32.3864624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3864721Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3864985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3865056Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3865282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3865369Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3865619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3865736Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3865989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3866075Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3866087Z 2025-08-26T20:34:32.3866189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3866392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3866466Z return mod(**inputs) 2025-08-26T20:34:32.3866718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3866793Z outputs = self.model( 2025-08-26T20:34:32.3867050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3867141Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3867408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3867483Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3867726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3867811Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3868091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3868219Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3868223Z 2025-08-26T20:34:32.3868341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3868552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3868617Z return mod(**inputs) 2025-08-26T20:34:32.3868878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3868947Z outputs = self.model( 2025-08-26T20:34:32.3869201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3869303Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3869574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3869659Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3869892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3869976Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3870251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3870379Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3870613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3870687Z return self.act(input) 2025-08-26T20:34:32.3870692Z 2025-08-26T20:34:32.3870799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3871035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3871106Z return mod(**inputs) 2025-08-26T20:34:32.3871380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3871454Z outputs = self.model( 2025-08-26T20:34:32.3871736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3871818Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3872093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3872182Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3872420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3872518Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3872792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3872885Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3872890Z 2025-08-26T20:34:32.3873010Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3873227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3873310Z return mod(**inputs) 2025-08-26T20:34:32.3873602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3873677Z outputs = self.model( 2025-08-26T20:34:32.3873957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3874037Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3874312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3874391Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3874635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3874720Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3875011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3875127Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3875399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3875568Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3875588Z 2025-08-26T20:34:32.3875699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3875912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3875989Z return mod(**inputs) 2025-08-26T20:34:32.3876289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3876370Z outputs = self.model( 2025-08-26T20:34:32.3876651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3876738Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3877019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3877096Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3877348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3877436Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3877752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3877860Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3878137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3878236Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3878240Z 2025-08-26T20:34:32.3878351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3878577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3878648Z return mod(**inputs) 2025-08-26T20:34:32.3878923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3879004Z outputs = self.model( 2025-08-26T20:34:32.3879292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3879380Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3879722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3879811Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3880052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3880137Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3880446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3880554Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3880836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3880931Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3880936Z 2025-08-26T20:34:32.3881023Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3881117Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3881202Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3881292Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3881420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3881636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3881714Z return mod(**inputs) 2025-08-26T20:34:32.3881988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3882071Z outputs = self.model( 2025-08-26T20:34:32.3882353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3882461Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3882751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3882828Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3883081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3883162Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3883419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3883519Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3883771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3883878Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3884171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3884375Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3884379Z 2025-08-26T20:34:32.3884483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3884698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3884764Z return mod(**inputs) 2025-08-26T20:34:32.3885023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3885101Z outputs = self.model( 2025-08-26T20:34:32.3885356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3885436Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3885691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3885764Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3885995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3886074Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3886341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3886440Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3886709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3886818Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3887107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3887222Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3887227Z 2025-08-26T20:34:32.3887332Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3887551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3887621Z return mod(**inputs) 2025-08-26T20:34:32.3887904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3887988Z outputs = self.model( 2025-08-26T20:34:32.3888269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3888356Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3888637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3888733Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3888986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3889072Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3889356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3889456Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3889718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3889800Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3889805Z 2025-08-26T20:34:32.3889908Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3890119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3890188Z return mod(**inputs) 2025-08-26T20:34:32.3890451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3890545Z outputs = self.model( 2025-08-26T20:34:32.3890804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3890885Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3891145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3891226Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3891455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3891535Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3891797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3891908Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3892168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3892318Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3892322Z 2025-08-26T20:34:32.3892430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3892633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3892700Z return mod(**inputs) 2025-08-26T20:34:32.3892981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3893050Z outputs = self.model( 2025-08-26T20:34:32.3893316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3893391Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3893643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3893723Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3893945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3894032Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3894299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3894418Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3894672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3894754Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3894774Z 2025-08-26T20:34:32.3894885Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3895084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3895161Z return mod(**inputs) 2025-08-26T20:34:32.3895417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3895485Z outputs = self.model( 2025-08-26T20:34:32.3895746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3895821Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3896084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3896156Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3896602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3896715Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3897010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3897128Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3897381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3897479Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3897483Z 2025-08-26T20:34:32.3897565Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3897646Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3897736Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3897813Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3897924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3898125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3898191Z return mod(**inputs) 2025-08-26T20:34:32.3898456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3898526Z outputs = self.model( 2025-08-26T20:34:32.3898788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3898864Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3899120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3899235Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3899471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3899561Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3899834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3899955Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3900221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3900325Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3900668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3900812Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3900816Z 2025-08-26T20:34:32.3900934Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3901150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3901216Z return mod(**inputs) 2025-08-26T20:34:32.3901506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3901579Z outputs = self.model( 2025-08-26T20:34:32.3901858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3901938Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3902217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3902294Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3902532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3902623Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3902896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3903018Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3903289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3903410Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3903728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3903844Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3903848Z 2025-08-26T20:34:32.3903964Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3904180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3904258Z return mod(**inputs) 2025-08-26T20:34:32.3904531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3904607Z outputs = self.model( 2025-08-26T20:34:32.3904885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3904963Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3905241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3905318Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3905554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3905645Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3905936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3906060Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3906330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3906418Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3906431Z 2025-08-26T20:34:32.3906540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3906751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3906830Z return mod(**inputs) 2025-08-26T20:34:32.3907115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3907198Z outputs = self.model( 2025-08-26T20:34:32.3907470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3907547Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3907827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3907931Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3908173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3908257Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3908527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-26T20:34:32.3908620Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3908626Z 2025-08-26T20:34:32.3908732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3908951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3909023Z return mod(**inputs) 2025-08-26T20:34:32.3909293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3909371Z outputs = self.model( 2025-08-26T20:34:32.3909642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3909746Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3910021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3910104Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3910345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3910428Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3910714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3910842Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3910846Z 2025-08-26T20:34:32.3910960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3911177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3911248Z return mod(**inputs) 2025-08-26T20:34:32.3911541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3911610Z outputs = self.model( 2025-08-26T20:34:32.3911876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3911949Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3912217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3912305Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3912530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3912619Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3912876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3913005Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3913219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3913290Z return self.act(input) 2025-08-26T20:34:32.3913293Z 2025-08-26T20:34:32.3913421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3913620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3913696Z return mod(**inputs) 2025-08-26T20:34:32.3913951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3914019Z outputs = self.model( 2025-08-26T20:34:32.3914281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3914376Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3914639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3914711Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3914939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3915019Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3915277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3915367Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3915371Z 2025-08-26T20:34:32.3915473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3915679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3915746Z return mod(**inputs) 2025-08-26T20:34:32.3916004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3916100Z outputs = self.model( 2025-08-26T20:34:32.3916354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3916438Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3916707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3916783Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3917022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3917105Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3917377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3917484Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3917757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3917918Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3917922Z 2025-08-26T20:34:32.3918032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3918251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3918320Z return mod(**inputs) 2025-08-26T20:34:32.3918615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3918689Z outputs = self.model( 2025-08-26T20:34:32.3918963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3919049Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3919322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3919408Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3919701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3919817Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3920087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3920195Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3920472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3920579Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3920582Z 2025-08-26T20:34:32.3920699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3920910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3920979Z return mod(**inputs) 2025-08-26T20:34:32.3921257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3921331Z outputs = self.model( 2025-08-26T20:34:32.3921616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3921692Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3921954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3922026Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3922249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3922340Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3922613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3922719Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3922974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3923063Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3923067Z 2025-08-26T20:34:32.3923157Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3923239Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3923324Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3923399Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3923501Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3923711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3923779Z return mod(**inputs) 2025-08-26T20:34:32.3924044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3924113Z outputs = self.model( 2025-08-26T20:34:32.3924369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3924449Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3924705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3924811Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3925048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3925140Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3925409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3925522Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3925783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3925881Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3926199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3926337Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3926343Z 2025-08-26T20:34:32.3926445Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3926653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3926738Z return mod(**inputs) 2025-08-26T20:34:32.3927003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3927073Z outputs = self.model( 2025-08-26T20:34:32.3927333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3927407Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3927657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3927738Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3927960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3928046Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3928296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3928395Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3928676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3928775Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3929074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3929186Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3929190Z 2025-08-26T20:34:32.3929299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3929500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3929568Z return mod(**inputs) 2025-08-26T20:34:32.3929828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3929898Z outputs = self.model( 2025-08-26T20:34:32.3930158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3930232Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3930486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3930567Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3930786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3930873Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3931154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3931249Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3931502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3931584Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3931588Z 2025-08-26T20:34:32.3931697Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3931896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3931969Z return mod(**inputs) 2025-08-26T20:34:32.3932249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3932321Z outputs = self.model( 2025-08-26T20:34:32.3932587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3932660Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3932917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3933021Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3933251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3933340Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3933608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3933725Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3933977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3934133Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3934137Z 2025-08-26T20:34:32.3934237Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3934439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3934515Z return mod(**inputs) 2025-08-26T20:34:32.3934782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3934857Z outputs = self.model( 2025-08-26T20:34:32.3935104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3935176Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3935431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3935502Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3935727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3935804Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3936050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3936165Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3936411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3936500Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3936504Z 2025-08-26T20:34:32.3936606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3936807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3936871Z return mod(**inputs) 2025-08-26T20:34:32.3937134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3937212Z outputs = self.model( 2025-08-26T20:34:32.3937460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3937538Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3937787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3937857Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3938085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3938178Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3938436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3938543Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3938793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3938886Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3938911Z 2025-08-26T20:34:32.3938998Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3939094Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3939176Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3939263Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3939373Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3939588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3939663Z return mod(**inputs) 2025-08-26T20:34:32.3939917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3939994Z outputs = self.model( 2025-08-26T20:34:32.3940250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3940323Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3940586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3940678Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3940905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3940986Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3941243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3941358Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3941619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3941724Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3942010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3942150Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3942155Z 2025-08-26T20:34:32.3942256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3942448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3942520Z return mod(**inputs) 2025-08-26T20:34:32.3942772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3942848Z outputs = self.model( 2025-08-26T20:34:32.3943117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3943194Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3943454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3943528Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3943756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3943841Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3944127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3944257Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3944529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3944632Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3944926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3945041Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3945063Z 2025-08-26T20:34:32.3945167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3945367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3945442Z return mod(**inputs) 2025-08-26T20:34:32.3945699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3945781Z outputs = self.model( 2025-08-26T20:34:32.3946031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3946109Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3946405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3946478Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3946708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3946788Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3947072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3947178Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3947433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3947524Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3947528Z 2025-08-26T20:34:32.3947629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3947838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3947906Z return mod(**inputs) 2025-08-26T20:34:32.3948166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3948244Z outputs = self.model( 2025-08-26T20:34:32.3948501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3948584Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3948840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3948919Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3949139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3949218Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3949737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3949870Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3949876Z 2025-08-26T20:34:32.3949993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3950217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3950292Z return mod(**inputs) 2025-08-26T20:34:32.3950590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3950666Z outputs = self.model( 2025-08-26T20:34:32.3950966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3951047Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3951385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3951459Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3951683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3951789Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3952047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3952175Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3952393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3952468Z return self.act(input) 2025-08-26T20:34:32.3952471Z 2025-08-26T20:34:32.3952585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3952792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3952867Z return mod(**inputs) 2025-08-26T20:34:32.3953124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3953195Z outputs = self.model( 2025-08-26T20:34:32.3953457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3953550Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3953813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3953885Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3954116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3954195Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3954449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3954541Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3954545Z 2025-08-26T20:34:32.3954647Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3954857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3954926Z return mod(**inputs) 2025-08-26T20:34:32.3955179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3955257Z outputs = self.model( 2025-08-26T20:34:32.3955514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3955595Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3955886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3955964Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3956206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3956291Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3956568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:34:32.3956657Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.3956660Z 2025-08-26T20:34:32.3956776Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3956989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3957079Z return mod(**inputs) 2025-08-26T20:34:32.3957359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3957430Z outputs = self.model( 2025-08-26T20:34:32.3957708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3957786Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3958074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3958160Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3958395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3958488Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3958758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3958873Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3959143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3959303Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3959307Z 2025-08-26T20:34:32.3959422Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3959708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3959816Z return mod(**inputs) 2025-08-26T20:34:32.3960094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3960167Z outputs = self.model( 2025-08-26T20:34:32.3960453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3960532Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3960819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3960895Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3961131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3961211Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3961470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3961580Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3961840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3961928Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3961932Z 2025-08-26T20:34:32.3962038Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3962241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3962339Z return mod(**inputs) 2025-08-26T20:34:32.3962596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3962672Z outputs = self.model( 2025-08-26T20:34:32.3962928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3963002Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3963263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3963335Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3963581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3963662Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3963925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3964026Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3964281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3964401Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3964405Z 2025-08-26T20:34:32.3964491Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3964585Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3964667Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3964747Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3964862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3965067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3965144Z return mod(**inputs) 2025-08-26T20:34:32.3965405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3965479Z outputs = self.model( 2025-08-26T20:34:32.3965748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3965827Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3966092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3966187Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3966412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3966502Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3966773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3966886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3967160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3967272Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3967586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3967733Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3967737Z 2025-08-26T20:34:32.3967852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3968064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3968142Z return mod(**inputs) 2025-08-26T20:34:32.3968414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3968488Z outputs = self.model( 2025-08-26T20:34:32.3968782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3968863Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3969138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3969217Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3969457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3969541Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3969810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3969939Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3970207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3970321Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3970633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3970768Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3970779Z 2025-08-26T20:34:32.3970887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3971103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3971183Z return mod(**inputs) 2025-08-26T20:34:32.3971457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3971538Z outputs = self.model( 2025-08-26T20:34:32.3971807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3971888Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3972164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3972239Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3972482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3972585Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3972853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3972964Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3973234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3973328Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3973332Z 2025-08-26T20:34:32.3973440Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3973662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3973731Z return mod(**inputs) 2025-08-26T20:34:32.3974000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3974081Z outputs = self.model( 2025-08-26T20:34:32.3974349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3974432Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3974700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3974776Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3975017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3975118Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3975396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3975515Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3975787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3975956Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3975961Z 2025-08-26T20:34:32.3976069Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3976289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3976376Z return mod(**inputs) 2025-08-26T20:34:32.3976652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3976726Z outputs = self.model( 2025-08-26T20:34:32.3976994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3977078Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3977363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3977446Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3977679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3977762Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3978044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3978160Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3978437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.3978524Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.3978528Z 2025-08-26T20:34:32.3978643Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3978856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3978927Z return mod(**inputs) 2025-08-26T20:34:32.3979239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3979313Z outputs = self.model( 2025-08-26T20:34:32.3979597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3979680Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3979960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3980045Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3980278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3980367Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3980638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3980755Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3981028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.3981120Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.3981124Z 2025-08-26T20:34:32.3981222Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3981309Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3981400Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3981498Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.3981610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3981832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3981903Z return mod(**inputs) 2025-08-26T20:34:32.3982182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3982256Z outputs = self.model( 2025-08-26T20:34:32.3982524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3982618Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3982886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3982967Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3983192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3983271Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3983533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3983694Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3983955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3984054Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3984353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.3984489Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.3984493Z 2025-08-26T20:34:32.3984597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3984808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3984876Z return mod(**inputs) 2025-08-26T20:34:32.3985137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3985208Z outputs = self.model( 2025-08-26T20:34:32.3985462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3985564Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3985817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3985897Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3986121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3986203Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3986467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3986577Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3986842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.3986942Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.3987240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.3987348Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.3987352Z 2025-08-26T20:34:32.3987456Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3987666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3987733Z return mod(**inputs) 2025-08-26T20:34:32.3988014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3988085Z outputs = self.model( 2025-08-26T20:34:32.3988344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3988429Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3988686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3988767Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3988990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3989092Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3989349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.3989458Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.3989722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.3989826Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.3989830Z 2025-08-26T20:34:32.3989943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3990157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3990227Z return mod(**inputs) 2025-08-26T20:34:32.3990515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3990589Z outputs = self.model( 2025-08-26T20:34:32.3990877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3990955Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3991243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3991318Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3991556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3991671Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3991949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3992076Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3992080Z 2025-08-26T20:34:32.3992184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3992385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3992457Z return mod(**inputs) 2025-08-26T20:34:32.3992715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3992792Z outputs = self.model( 2025-08-26T20:34:32.3993052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3993127Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3993390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3993462Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3993692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3993772Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3994033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.3994171Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.3994399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.3994482Z return self.act(input) 2025-08-26T20:34:32.3994487Z 2025-08-26T20:34:32.3994596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3994820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3994893Z return mod(**inputs) 2025-08-26T20:34:32.3995173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3995253Z outputs = self.model( 2025-08-26T20:34:32.3995556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3995644Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3995916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3996001Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3996442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3996578Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3996862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.3996954Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.3996959Z 2025-08-26T20:34:32.3997077Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.3997289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.3997361Z return mod(**inputs) 2025-08-26T20:34:32.3997641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.3997713Z outputs = self.model( 2025-08-26T20:34:32.3997988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.3998067Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.3998336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.3998451Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.3998696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.3998800Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.3999078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.3999195Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.3999522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.3999702Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.3999708Z 2025-08-26T20:34:32.3999831Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4000049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4000127Z return mod(**inputs) 2025-08-26T20:34:32.4000405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4000480Z outputs = self.model( 2025-08-26T20:34:32.4000769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4000846Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4001161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4001239Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4001484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4001570Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4001846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4001959Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4002233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.4002352Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.4002357Z 2025-08-26T20:34:32.4002467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4002681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4002761Z return mod(**inputs) 2025-08-26T20:34:32.4003031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4003138Z outputs = self.model( 2025-08-26T20:34:32.4003419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4003498Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4003784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4003860Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4004114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4004201Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4004483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4004587Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4004864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.4004964Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.4004986Z 2025-08-26T20:34:32.4005077Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4005169Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4005251Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4005332Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4005452Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4005666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4005742Z return mod(**inputs) 2025-08-26T20:34:32.4006015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4006086Z outputs = self.model( 2025-08-26T20:34:32.4006363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4006442Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4006717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4006794Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4007030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4007124Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4007399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4007523Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4007778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4007880Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4008179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.4008316Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.4008320Z 2025-08-26T20:34:32.4008431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4008633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4008724Z return mod(**inputs) 2025-08-26T20:34:32.4008981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4009053Z outputs = self.model( 2025-08-26T20:34:32.4009314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4009387Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4009664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4009740Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4009972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4010055Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4010314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4010424Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4010681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4010789Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4011084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.4011199Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.4011230Z 2025-08-26T20:34:32.4011336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4011537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4011610Z return mod(**inputs) 2025-08-26T20:34:32.4011871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4011949Z outputs = self.model( 2025-08-26T20:34:32.4012204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4012279Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4012541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4012618Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4012845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4012928Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4013180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4013286Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4013547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.4013638Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.4013658Z 2025-08-26T20:34:32.4013763Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4013972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4014039Z return mod(**inputs) 2025-08-26T20:34:32.4014294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4014373Z outputs = self.model( 2025-08-26T20:34:32.4014629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4014710Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4014982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4015058Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4015289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4015368Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4015631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-26T20:34:32.4015730Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.4015734Z 2025-08-26T20:34:32.4015837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4016046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4016112Z return mod(**inputs) 2025-08-26T20:34:32.4016378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4016447Z outputs = self.model( 2025-08-26T20:34:32.4016710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4016786Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4017040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4017119Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4017343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4017451Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4017703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4017813Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4018078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.4018232Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.4018235Z 2025-08-26T20:34:32.4018346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4018547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4018621Z return mod(**inputs) 2025-08-26T20:34:32.4018879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4018948Z outputs = self.model( 2025-08-26T20:34:32.4019209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4019280Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4019542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4019614Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4019851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4019939Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4020195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4020310Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4020563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.4020648Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.4020659Z 2025-08-26T20:34:32.4020761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4020997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4021072Z return mod(**inputs) 2025-08-26T20:34:32.4021324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4021401Z outputs = self.model( 2025-08-26T20:34:32.4021659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4021750Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4022013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4022088Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4022314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4022394Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4022647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4022762Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4023013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.4023105Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.4023109Z 2025-08-26T20:34:32.4023190Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4023280Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4023358Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4023455Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4023566Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4023769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4023844Z return mod(**inputs) 2025-08-26T20:34:32.4024104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4024175Z outputs = self.model( 2025-08-26T20:34:32.4024438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4024515Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4024775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4024850Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4025070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4025158Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4025413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4025531Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4025783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4025900Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4026202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.4026338Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.4026344Z 2025-08-26T20:34:32.4026457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4026670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4026747Z return mod(**inputs) 2025-08-26T20:34:32.4027018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4027115Z outputs = self.model( 2025-08-26T20:34:32.4027393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4027472Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4027749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4027826Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4028082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4028178Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4028455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4028571Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4028826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4028932Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4029226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.4029336Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.4029340Z 2025-08-26T20:34:32.4029451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4029654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4029746Z return mod(**inputs) 2025-08-26T20:34:32.4029999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4030067Z outputs = self.model( 2025-08-26T20:34:32.4030331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4030405Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4030665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4030738Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4030972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4031057Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4031324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4031445Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4031713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.4031808Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.4031814Z 2025-08-26T20:34:32.4031921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4032132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4032230Z return mod(**inputs) 2025-08-26T20:34:32.4032502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4032581Z outputs = self.model( 2025-08-26T20:34:32.4032855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4032935Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4033216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4033292Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4033554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4033641Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4033921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.4034049Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.4034053Z 2025-08-26T20:34:32.4034160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4034396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4034468Z return mod(**inputs) 2025-08-26T20:34:32.4034746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4034818Z outputs = self.model( 2025-08-26T20:34:32.4035087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4035173Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4035442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4035525Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4035760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4035852Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4036122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.4036267Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.4036500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.4036573Z return self.act(input) 2025-08-26T20:34:32.4036580Z 2025-08-26T20:34:32.4036696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4036905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4036974Z return mod(**inputs) 2025-08-26T20:34:32.4037251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4037322Z outputs = self.model( 2025-08-26T20:34:32.4037596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4037676Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4037943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4038027Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4038259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4038351Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4038619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.4038732Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.4038736Z 2025-08-26T20:34:32.4038848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4039060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4039142Z return mod(**inputs) 2025-08-26T20:34:32.4039411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4039557Z outputs = self.model( 2025-08-26T20:34:32.4039844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4039923Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4040255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4040346Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4040598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4040682Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4040955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4041093Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4041346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.4041502Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.4041507Z 2025-08-26T20:34:32.4041609Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4041813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4041880Z return mod(**inputs) 2025-08-26T20:34:32.4042138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4042214Z outputs = self.model( 2025-08-26T20:34:32.4042476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4042563Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4042855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4042931Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4043174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4043255Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4043515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4043622Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4043901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.4043988Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.4043993Z 2025-08-26T20:34:32.4044101Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4044321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4044392Z return mod(**inputs) 2025-08-26T20:34:32.4044668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4044739Z outputs = self.model( 2025-08-26T20:34:32.4045012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4045098Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4045386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4045471Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4045703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4045790Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4046063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4046168Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4046463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.4046559Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.4046562Z 2025-08-26T20:34:32.4046656Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4046741Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4046823Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4046915Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4047174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4047426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4047499Z return mod(**inputs) 2025-08-26T20:34:32.4047769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4047849Z outputs = self.model( 2025-08-26T20:34:32.4048118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4048207Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4048477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4048555Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4048802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4048886Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4049163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4049290Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4049569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4049674Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4049984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.4050135Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.4050140Z 2025-08-26T20:34:32.4050248Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4050465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4050536Z return mod(**inputs) 2025-08-26T20:34:32.4050804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4050884Z outputs = self.model( 2025-08-26T20:34:32.4051149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4051235Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4051505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4051587Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4051844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4051930Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4052208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4052314Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4052586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4052689Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4053004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.4053136Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.4053140Z 2025-08-26T20:34:32.4053245Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4053456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4053525Z return mod(**inputs) 2025-08-26T20:34:32.4053796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4053879Z outputs = self.model( 2025-08-26T20:34:32.4054124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4054206Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4054451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4054532Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4054749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4054825Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4055077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4055173Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4055425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.4055507Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.4055527Z 2025-08-26T20:34:32.4055627Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4055829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4055892Z return mod(**inputs) 2025-08-26T20:34:32.4056145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4056213Z outputs = self.model( 2025-08-26T20:34:32.4056466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4056539Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4056785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4056865Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4057078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4057164Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4057408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4057517Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4057774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.4057941Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.4057945Z 2025-08-26T20:34:32.4058058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4058257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4058333Z return mod(**inputs) 2025-08-26T20:34:32.4058598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4058672Z outputs = self.model( 2025-08-26T20:34:32.4058948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4059026Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4059318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4059394Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4059615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4059703Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4059953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4060087Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4060342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.4060422Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.4060432Z 2025-08-26T20:34:32.4060537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4060741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4060816Z return mod(**inputs) 2025-08-26T20:34:32.4061086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4061162Z outputs = self.model( 2025-08-26T20:34:32.4061411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4061484Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4061758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4061828Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4062051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4062130Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4062387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4062507Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4062760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.4062855Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.4062861Z 2025-08-26T20:34:32.4062943Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4063031Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4063111Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4063188Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4063299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4063501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4063576Z return mod(**inputs) 2025-08-26T20:34:32.4063832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4063916Z outputs = self.model( 2025-08-26T20:34:32.4064187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4064259Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4064514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4064587Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4064802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4064886Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4065148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4065262Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4065509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4065606Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4065898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.4066048Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.4066053Z 2025-08-26T20:34:32.4066157Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4066351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4066423Z return mod(**inputs) 2025-08-26T20:34:32.4066678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4066747Z outputs = self.model( 2025-08-26T20:34:32.4067011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4067085Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4067346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4067420Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4067641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4067744Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4067997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4068111Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4068366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4068474Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4068768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.4068876Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.4068881Z 2025-08-26T20:34:32.4068990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4069190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4069266Z return mod(**inputs) 2025-08-26T20:34:32.4069523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4069593Z outputs = self.model( 2025-08-26T20:34:32.4069855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4069929Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4070206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4070281Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4070510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4070589Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4070845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4070958Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4071212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.4071317Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.4071321Z 2025-08-26T20:34:32.4071425Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4071626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4071701Z return mod(**inputs) 2025-08-26T20:34:32.4071953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4072051Z outputs = self.model( 2025-08-26T20:34:32.4072304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4072378Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4072640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4072718Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4072958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4073042Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4073316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-26T20:34:32.4073402Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.4073406Z 2025-08-26T20:34:32.4073515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4073733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4073831Z return mod(**inputs) 2025-08-26T20:34:32.4074104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4074172Z outputs = self.model( 2025-08-26T20:34:32.4074426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4074507Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4074759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4074837Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4075061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4075146Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4075419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.4075550Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.4075554Z 2025-08-26T20:34:32.4075669Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4075882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4075958Z return mod(**inputs) 2025-08-26T20:34:32.4076226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4076319Z outputs = self.model( 2025-08-26T20:34:32.4076599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4076677Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4076953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4077031Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4077265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4077356Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4077644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.4077782Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.4078011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.4078094Z return self.act(input) 2025-08-26T20:34:32.4078098Z 2025-08-26T20:34:32.4078208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4078435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4078515Z return mod(**inputs) 2025-08-26T20:34:32.4078783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4078861Z outputs = self.model( 2025-08-26T20:34:32.4079127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4079204Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4079546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4079637Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4079885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4079970Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4080247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.4080374Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.4080378Z 2025-08-26T20:34:32.4080492Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4080720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4080803Z return mod(**inputs) 2025-08-26T20:34:32.4081062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4081130Z outputs = self.model( 2025-08-26T20:34:32.4081386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4081468Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4081723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4081805Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4082028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4082106Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4082373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4082475Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4082765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.4082920Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.4082924Z 2025-08-26T20:34:32.4083032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4083235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4083301Z return mod(**inputs) 2025-08-26T20:34:32.4083565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4083634Z outputs = self.model( 2025-08-26T20:34:32.4083896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4084011Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4084267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4084349Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4084569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4084655Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4084930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4085031Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4085293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.4085375Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.4085379Z 2025-08-26T20:34:32.4085489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4085692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4085765Z return mod(**inputs) 2025-08-26T20:34:32.4086023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4086092Z outputs = self.model( 2025-08-26T20:34:32.4086359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4086451Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4086709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4086782Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4087004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4087093Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4087345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4087452Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4087704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.4087799Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.4087802Z 2025-08-26T20:34:32.4087885Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4087968Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4088056Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4088132Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4088243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4088445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4088511Z return mod(**inputs) 2025-08-26T20:34:32.4088791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4088861Z outputs = self.model( 2025-08-26T20:34:32.4089128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4089202Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4089448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4089528Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4089741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4089828Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4090092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4090190Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4090451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4090546Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4090844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.4090994Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.4090999Z 2025-08-26T20:34:32.4091105Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4091303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4091369Z return mod(**inputs) 2025-08-26T20:34:32.4091632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4091701Z outputs = self.model( 2025-08-26T20:34:32.4091961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4092034Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4092286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4092365Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4092584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4092687Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4092931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4093035Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4093283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4093383Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4093682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.4093791Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.4093797Z 2025-08-26T20:34:32.4093906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4094106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4094173Z return mod(**inputs) 2025-08-26T20:34:32.4094435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4094504Z outputs = self.model( 2025-08-26T20:34:32.4094769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4094841Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4095113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4095184Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4095397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4095484Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4095732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-26T20:34:32.4095832Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:34:32.4096091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.4096323Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.4096329Z 2025-08-26T20:34:32.4096443Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4096643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4096715Z return mod(**inputs) 2025-08-26T20:34:32.4096964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4097084Z outputs = self.model( 2025-08-26T20:34:32.4097344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4097416Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4097677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4097751Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4097984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4098065Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4098321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4098441Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4098696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-26T20:34:32.4098887Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:34:32.4098891Z 2025-08-26T20:34:32.4098995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4099197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4099275Z return mod(**inputs) 2025-08-26T20:34:32.4099534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4099612Z outputs = self.model( 2025-08-26T20:34:32.4099868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4099948Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4100206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4100280Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4100510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4100601Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4100859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4100965Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4101243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-26T20:34:32.4101332Z key_states = self.k_proj(current_states) 2025-08-26T20:34:32.4101336Z 2025-08-26T20:34:32.4101436Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4101638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4101704Z return mod(**inputs) 2025-08-26T20:34:32.4101962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4102028Z outputs = self.model( 2025-08-26T20:34:32.4102278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4102378Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4102625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4102704Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4102919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4102996Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4103268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4103376Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4103628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-26T20:34:32.4103713Z value_states = self.v_proj(current_states) 2025-08-26T20:34:32.4103717Z 2025-08-26T20:34:32.4103807Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4103886Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4103960Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4104041Z cudagraph partition due to non gpu ops 2025-08-26T20:34:32.4104144Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4104337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4104410Z return mod(**inputs) 2025-08-26T20:34:32.4104660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4104752Z outputs = self.model( 2025-08-26T20:34:32.4105009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4105089Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4105346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4105418Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4105653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4105733Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4106000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4106116Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4106386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4106500Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4106814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:34:32.4106968Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:34:32.4106972Z 2025-08-26T20:34:32.4107081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4110024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4110127Z return mod(**inputs) 2025-08-26T20:34:32.4110418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4110505Z outputs = self.model( 2025-08-26T20:34:32.4110780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4110863Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4111143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4111220Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4111488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4111578Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4111884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4111994Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4112260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-26T20:34:32.4112378Z attn_output, attn_weights = attention_interface( 2025-08-26T20:34:32.4112679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:34:32.4112789Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:34:32.4112792Z 2025-08-26T20:34:32.4112897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4113106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4113171Z return mod(**inputs) 2025-08-26T20:34:32.4113434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4113503Z outputs = self.model( 2025-08-26T20:34:32.4113756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4113838Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4114109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4114189Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4114410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4114500Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4114763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-26T20:34:32.4114876Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:34:32.4115149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-26T20:34:32.4115237Z attn_output = self.out_proj(attn_output) 2025-08-26T20:34:32.4115242Z 2025-08-26T20:34:32.4115357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4115571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4115641Z return mod(**inputs) 2025-08-26T20:34:32.4115916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4115988Z outputs = self.model( 2025-08-26T20:34:32.4116264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4116342Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4116684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4116774Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4117019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4117117Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4117396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.4117536Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.4117540Z 2025-08-26T20:34:32.4117653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4117897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4117979Z return mod(**inputs) 2025-08-26T20:34:32.4118253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4118332Z outputs = self.model( 2025-08-26T20:34:32.4118605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4118706Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4118991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4119069Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4119321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4119408Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4119766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-26T20:34:32.4119903Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:34:32.4120141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:34:32.4120225Z return self.act(input) 2025-08-26T20:34:32.4120231Z 2025-08-26T20:34:32.4120342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4120567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4120663Z return mod(**inputs) 2025-08-26T20:34:32.4120946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4121034Z outputs = self.model( 2025-08-26T20:34:32.4121306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4121393Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4121675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4121753Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4121986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4122082Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4122356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-26T20:34:32.4122456Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:34:32.4122460Z 2025-08-26T20:34:32.4122574Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4122775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4122849Z return mod(**inputs) 2025-08-26T20:34:32.4123102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-26T20:34:32.4123220Z outputs = self.model( 2025-08-26T20:34:32.4123481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-26T20:34:32.4123565Z decoder_outputs = self.decoder( 2025-08-26T20:34:32.4123835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-26T20:34:32.4123913Z layer_outputs = decoder_layer( 2025-08-26T20:34:32.4124154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:34:32.4124239Z return super().__call__(*args, **kwargs) 2025-08-26T20:34:32.4124533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-26T20:34:32.4124621Z hidden_states = residual + hidden_states 2025-08-26T20:34:32.4124625Z 2025-08-26T20:34:32.4124734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4124955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4125026Z return mod(**inputs) 2025-08-26T20:34:32.4125319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1456, in forward 2025-08-26T20:34:32.4125452Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-26T20:34:32.4125456Z 2025-08-26T20:34:32.4125572Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:34:32.4125783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:34:32.4125853Z return mod(**inputs) 2025-08-26T20:34:32.4126130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1461, in forward 2025-08-26T20:34:32.4126312Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:34:32.4126317Z 2025-08-26T20:34:45.6344060Z Compilation time (from dynamo_timed): 29.052555415 2025-08-26T20:34:45.6435996Z pass 2025-08-26T20:34:45.6436403Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:34:45.6437322Z TIMING: _recursive_pre_grad_passes:0.01562 _recursive_joint_graph_passes:1.18437 _recursive_post_grad_passes:0.18069 async_compile.wait:0.75926 code_gen:11.71708 inductor_compile:15.12653 backend_compile:22.9037 gc:0.00056 entire_frame_compile:29.05256 total_wall_time:29.05256 2025-08-26T20:34:45.6438709Z STATS: call_* op count: 986 | FakeTensorMode.__torch_dispatch__:33710 | FakeTensor.__torch_dispatch__:11299 | ProxyTorchDispatchMode.__torch_dispatch__:12456 2025-08-26T20:34:45.6439296Z Dynamo produced 1 graphs covering 986 ops with 0 graph breaks (0 unique) 2025-08-26T20:34:51.6520728Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:34:51.6521658Z from pkg_resources import resource_filename 2025-08-26T20:34:52.2643751Z 2025-08-26T20:34:55.4025554Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:34:55.4025901Z loading model: 0it [00:03, ?it/s] 2025-08-26T20:34:55.4043804Z cpu eval MT5ForConditionalGeneration 2025-08-26T20:34:56.0417284Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:34:56.3107619Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:34:56.5781519Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:35:10.0146945Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0147380Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0148437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0148924Z return mod(**inputs) 2025-08-26T20:35:10.0149359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0149816Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0150234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0150652Z layer_outputs = layer_module( 2025-08-26T20:35:10.0151109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0151529Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0151948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0152399Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0152813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0153306Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0153736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 421, in forward 2025-08-26T20:35:10.0154174Z position_bias = position_bias + causal_mask 2025-08-26T20:35:10.0154341Z 2025-08-26T20:35:10.0154477Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0154982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0155366Z return mod(**inputs) 2025-08-26T20:35:10.0155778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0156208Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0156636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0157059Z layer_outputs = layer_module( 2025-08-26T20:35:10.0157459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0157927Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0158362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0158788Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0159209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0159861Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0160330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0160778Z return self.weight * hidden_states 2025-08-26T20:35:10.0160931Z 2025-08-26T20:35:10.0161059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0161467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0161843Z return mod(**inputs) 2025-08-26T20:35:10.0162250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0162740Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0163161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0163596Z layer_outputs = layer_module( 2025-08-26T20:35:10.0163985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0164389Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0164849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0165277Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0165711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0166141Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0166565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0166996Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0167147Z 2025-08-26T20:35:10.0167289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0167694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0168072Z return mod(**inputs) 2025-08-26T20:35:10.0168464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0168876Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0169298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0169825Z layer_outputs = layer_module( 2025-08-26T20:35:10.0170220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0170626Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0171025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0171446Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0171875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0172293Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0172708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0173109Z key_states = self.k(current_states) 2025-08-26T20:35:10.0173262Z 2025-08-26T20:35:10.0173375Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0173762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0174142Z return mod(**inputs) 2025-08-26T20:35:10.0174511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0174943Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0175348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0175748Z layer_outputs = layer_module( 2025-08-26T20:35:10.0176113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0176505Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0176907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0177315Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0177724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0178137Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0178544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0179013Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0179211Z 2025-08-26T20:35:10.0179331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0180555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0180912Z return mod(**inputs) 2025-08-26T20:35:10.0181295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0181711Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0182121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0182530Z layer_outputs = layer_module( 2025-08-26T20:35:10.0182903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0183296Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0183725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0184149Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0184566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0184995Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0185417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0185942Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0186172Z 2025-08-26T20:35:10.0186292Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0186674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0187026Z return mod(**inputs) 2025-08-26T20:35:10.0187404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0187816Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0188206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0188611Z layer_outputs = layer_module( 2025-08-26T20:35:10.0188992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0189390Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0189832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0190254Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0190678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0191096Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0191518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0191937Z value_states = self.v(current_states) 2025-08-26T20:35:10.0192087Z 2025-08-26T20:35:10.0192205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0192611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0192984Z return mod(**inputs) 2025-08-26T20:35:10.0193379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0193800Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0194219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0194645Z layer_outputs = layer_module( 2025-08-26T20:35:10.0195040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0195446Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0195893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0196654Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0197075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0197523Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0197949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0198399Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0198589Z 2025-08-26T20:35:10.0198704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0199146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0199577Z return mod(**inputs) 2025-08-26T20:35:10.0199973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0200398Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0200807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0201282Z layer_outputs = layer_module( 2025-08-26T20:35:10.0201666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0202059Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0202473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0202901Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0203318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0203746Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0204171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0204620Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0204807Z 2025-08-26T20:35:10.0204924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0205321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0205710Z return mod(**inputs) 2025-08-26T20:35:10.0206099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0206603Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0207021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0207432Z layer_outputs = layer_module( 2025-08-26T20:35:10.0207807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0208208Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0208625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0209056Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0209465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0209869Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0210277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0210720Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0210894Z 2025-08-26T20:35:10.0211013Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0211390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0211776Z return mod(**inputs) 2025-08-26T20:35:10.0212158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0212564Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0212973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0213405Z layer_outputs = layer_module( 2025-08-26T20:35:10.0213789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0214176Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0214603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0215017Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0215415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0215824Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0216241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0216676Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0216824Z 2025-08-26T20:35:10.0216940Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0217336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0217699Z return mod(**inputs) 2025-08-26T20:35:10.0218094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0218510Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0218911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0219333Z layer_outputs = layer_module( 2025-08-26T20:35:10.0219722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0220125Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0220533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0220962Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0221370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0221836Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0222250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0222646Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0222801Z 2025-08-26T20:35:10.0222913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0223298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0223647Z return mod(**inputs) 2025-08-26T20:35:10.0224027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0224430Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0224829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0225241Z layer_outputs = layer_module( 2025-08-26T20:35:10.0226128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0226519Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0226925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0227386Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0227794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0228206Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0228601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0229020Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0229175Z 2025-08-26T20:35:10.0229290Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0229688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0230058Z return mod(**inputs) 2025-08-26T20:35:10.0230984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0231419Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0231841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0232255Z layer_outputs = layer_module( 2025-08-26T20:35:10.0232660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0233051Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0233494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0233927Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0234356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0234778Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0235207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0235632Z key_states = self.k(current_states) 2025-08-26T20:35:10.0235778Z 2025-08-26T20:35:10.0235901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0236304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0236664Z return mod(**inputs) 2025-08-26T20:35:10.0237077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0237506Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0237938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0238345Z layer_outputs = layer_module( 2025-08-26T20:35:10.0238731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0239129Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0239609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0240037Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0240445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0240864Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0241277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0241760Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0241960Z 2025-08-26T20:35:10.0242082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0242470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0242831Z return mod(**inputs) 2025-08-26T20:35:10.0243251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0243673Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0244073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0244490Z layer_outputs = layer_module( 2025-08-26T20:35:10.0244880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0245280Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0245696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0246141Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0246564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0246988Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0247406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0247908Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0248160Z 2025-08-26T20:35:10.0248285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0248684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0249043Z return mod(**inputs) 2025-08-26T20:35:10.0249424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0249833Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0250223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0250630Z layer_outputs = layer_module( 2025-08-26T20:35:10.0251013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0251409Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0251824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0252267Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0252681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0253103Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0253526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0254021Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0254261Z 2025-08-26T20:35:10.0254377Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0254784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0255147Z return mod(**inputs) 2025-08-26T20:35:10.0255543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0255950Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0256366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0256791Z layer_outputs = layer_module( 2025-08-26T20:35:10.0257185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0257587Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0258011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0258450Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0258907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0259322Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0259741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0260243Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0260470Z 2025-08-26T20:35:10.0260580Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0260968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0261360Z return mod(**inputs) 2025-08-26T20:35:10.0261746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0262185Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0262587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0262985Z layer_outputs = layer_module( 2025-08-26T20:35:10.0263369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0263759Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0264161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0264571Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0264981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0265385Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0265791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0266208Z value_states = self.v(current_states) 2025-08-26T20:35:10.0266355Z 2025-08-26T20:35:10.0266475Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0266863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0267207Z return mod(**inputs) 2025-08-26T20:35:10.0267606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0268024Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0268418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0268817Z layer_outputs = layer_module( 2025-08-26T20:35:10.0269196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0269601Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0270017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0270447Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0270855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0271265Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0271665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0272109Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0272283Z 2025-08-26T20:35:10.0272400Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0272780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0273127Z return mod(**inputs) 2025-08-26T20:35:10.0273532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0273949Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0274356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0274788Z layer_outputs = layer_module( 2025-08-26T20:35:10.0275176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0275578Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0276005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0276447Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0276878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0277321Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0277746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0278195Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0278393Z 2025-08-26T20:35:10.0278506Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0278906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0279279Z return mod(**inputs) 2025-08-26T20:35:10.0279766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0280185Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0280601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0281021Z layer_outputs = layer_module( 2025-08-26T20:35:10.0281410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0281814Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0282229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0282657Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0283100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0283540Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0283961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0284423Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0284609Z 2025-08-26T20:35:10.0284725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0285126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0285492Z return mod(**inputs) 2025-08-26T20:35:10.0285873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0286342Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0286755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0287173Z layer_outputs = layer_module( 2025-08-26T20:35:10.0287566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0287967Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0288385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0288817Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0289263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0289684Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0290113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0290539Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0290686Z 2025-08-26T20:35:10.0290787Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0291055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0291449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0291831Z return mod(**inputs) 2025-08-26T20:35:10.0292235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0292663Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0293076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0293481Z layer_outputs = layer_module( 2025-08-26T20:35:10.0293863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0294275Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0294683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0295099Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0295519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0295952Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0296632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0297053Z return self.weight * hidden_states 2025-08-26T20:35:10.0297198Z 2025-08-26T20:35:10.0297309Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0297697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0298050Z return mod(**inputs) 2025-08-26T20:35:10.0298489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0298894Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0299292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0299710Z layer_outputs = layer_module( 2025-08-26T20:35:10.0300096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0300489Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0300901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0301330Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0301762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0302227Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0302683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0303121Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0303296Z 2025-08-26T20:35:10.0303412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0303814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0304175Z return mod(**inputs) 2025-08-26T20:35:10.0304588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0304999Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0305394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0305795Z layer_outputs = layer_module( 2025-08-26T20:35:10.0306169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0306551Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0306968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0307420Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0307842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0308304Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0308746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0309222Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0309375Z 2025-08-26T20:35:10.0309486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0309876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0310219Z return mod(**inputs) 2025-08-26T20:35:10.0310599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0311006Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0311404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0311813Z layer_outputs = layer_module( 2025-08-26T20:35:10.0312181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0312568Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0312978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0313409Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0313854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0314296Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0314747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0315175Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0315328Z 2025-08-26T20:35:10.0315446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0315829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0316180Z return mod(**inputs) 2025-08-26T20:35:10.0316576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0317009Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0317428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0317847Z layer_outputs = layer_module( 2025-08-26T20:35:10.0318230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0318663Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0319087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0319582Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0320057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0320519Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0320975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0321399Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0321550Z 2025-08-26T20:35:10.0321643Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0321913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0322334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0322703Z return mod(**inputs) 2025-08-26T20:35:10.0323099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0323515Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0323922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0324336Z layer_outputs = layer_module( 2025-08-26T20:35:10.0324740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0325132Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0325549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0325974Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0326395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0326846Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0327292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0327715Z return self.weight * hidden_states 2025-08-26T20:35:10.0327864Z 2025-08-26T20:35:10.0327976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0328365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0328736Z return mod(**inputs) 2025-08-26T20:35:10.0329101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0329504Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0329906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0330308Z layer_outputs = layer_module( 2025-08-26T20:35:10.0330675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0331064Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0331609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0332025Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0332434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0332847Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0333261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0333667Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0333812Z 2025-08-26T20:35:10.0333933Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0334320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0334665Z return mod(**inputs) 2025-08-26T20:35:10.0335067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0335470Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0335867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0336260Z layer_outputs = layer_module( 2025-08-26T20:35:10.0336631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0337016Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0337438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0337845Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0338243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0338653Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0339040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0339459Z key_states = self.k(current_states) 2025-08-26T20:35:10.0339602Z 2025-08-26T20:35:10.0339722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0340107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0340460Z return mod(**inputs) 2025-08-26T20:35:10.0340845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0341255Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0341639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0342022Z layer_outputs = layer_module( 2025-08-26T20:35:10.0342377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0342759Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0343167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0343562Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0343960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0344349Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0344738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0345164Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0345353Z 2025-08-26T20:35:10.0345459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0345826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0346156Z return mod(**inputs) 2025-08-26T20:35:10.0346527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0346927Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0347328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0347727Z layer_outputs = layer_module( 2025-08-26T20:35:10.0348098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0348484Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0348863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0349267Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0349689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0350100Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0350489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0350949Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0351168Z 2025-08-26T20:35:10.0351273Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0351654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0352018Z return mod(**inputs) 2025-08-26T20:35:10.0352394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0352800Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0353197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0353600Z layer_outputs = layer_module( 2025-08-26T20:35:10.0353969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0354375Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0354785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0355195Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0355603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0356011Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0356423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0356931Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0357168Z 2025-08-26T20:35:10.0357291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0357696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0358050Z return mod(**inputs) 2025-08-26T20:35:10.0358468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0358896Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0359314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0359817Z layer_outputs = layer_module( 2025-08-26T20:35:10.0360203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0360612Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0361041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0361480Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0361897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0362326Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0362741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0363241Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0363465Z 2025-08-26T20:35:10.0363585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0363973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0364333Z return mod(**inputs) 2025-08-26T20:35:10.0364776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0365185Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0365585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0365984Z layer_outputs = layer_module( 2025-08-26T20:35:10.0366357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0366756Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0367177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0367579Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0367985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0368407Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0368815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0369242Z value_states = self.v(current_states) 2025-08-26T20:35:10.0369388Z 2025-08-26T20:35:10.0369500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0369889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0370237Z return mod(**inputs) 2025-08-26T20:35:10.0370617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0371021Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0371415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0371817Z layer_outputs = layer_module( 2025-08-26T20:35:10.0372191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0372582Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0372980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0373392Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0373846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0374268Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0374685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0375133Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0375315Z 2025-08-26T20:35:10.0375426Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0375815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0376184Z return mod(**inputs) 2025-08-26T20:35:10.0376560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0376986Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0377392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0377803Z layer_outputs = layer_module( 2025-08-26T20:35:10.0378179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0378574Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0378995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0379415Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0379870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0380293Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0380718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0381185Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0381372Z 2025-08-26T20:35:10.0381486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0381883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0382266Z return mod(**inputs) 2025-08-26T20:35:10.0382711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0383126Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0383537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0384015Z layer_outputs = layer_module( 2025-08-26T20:35:10.0384392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0384816Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0385237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0385670Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0386086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0386510Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0386927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0387385Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0387566Z 2025-08-26T20:35:10.0387689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0388079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0388441Z return mod(**inputs) 2025-08-26T20:35:10.0388827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0389267Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0389678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0390097Z layer_outputs = layer_module( 2025-08-26T20:35:10.0390480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0390880Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0391299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0391722Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0392134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0392561Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0392982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0393411Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0393560Z 2025-08-26T20:35:10.0393673Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0394078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0394442Z return mod(**inputs) 2025-08-26T20:35:10.0394834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0395286Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0395688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0396106Z layer_outputs = layer_module( 2025-08-26T20:35:10.0396693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0397102Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0397523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0397971Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0398470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0398909Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0399345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0399823Z return self.weight * hidden_states 2025-08-26T20:35:10.0400028Z 2025-08-26T20:35:10.0400142Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0400540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0400909Z return mod(**inputs) 2025-08-26T20:35:10.0401303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0401719Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0402136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0402555Z layer_outputs = layer_module( 2025-08-26T20:35:10.0402938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0403336Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0403758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0404200Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0404667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0405147Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0405603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0406047Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0406222Z 2025-08-26T20:35:10.0406341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0406744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0407108Z return mod(**inputs) 2025-08-26T20:35:10.0407492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0407917Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0408327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0408731Z layer_outputs = layer_module( 2025-08-26T20:35:10.0409098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0409499Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0409911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0410329Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0410786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0411226Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0411676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0412094Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0412246Z 2025-08-26T20:35:10.0412368Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0412775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0413117Z return mod(**inputs) 2025-08-26T20:35:10.0413517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0413933Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0414339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0414737Z layer_outputs = layer_module( 2025-08-26T20:35:10.0415109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0415514Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0415926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0416356Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0416773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0417224Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0417676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0418091Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0418246Z 2025-08-26T20:35:10.0418367Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0418748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0419114Z return mod(**inputs) 2025-08-26T20:35:10.0419491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0419918Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0420326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0420730Z layer_outputs = layer_module( 2025-08-26T20:35:10.0421103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0421498Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0421908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0422331Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0422756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0423207Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0423656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0424062Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0424207Z 2025-08-26T20:35:10.0424297Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0424558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0424946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0425310Z return mod(**inputs) 2025-08-26T20:35:10.0425704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0426112Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0426508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0426916Z layer_outputs = layer_module( 2025-08-26T20:35:10.0427289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0427670Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0428070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0428508Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0428915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0429341Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0429771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0430175Z return self.weight * hidden_states 2025-08-26T20:35:10.0430354Z 2025-08-26T20:35:10.0430474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0430864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0431209Z return mod(**inputs) 2025-08-26T20:35:10.0431590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0432000Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0432399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0432804Z layer_outputs = layer_module( 2025-08-26T20:35:10.0433177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0433570Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0433976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0434391Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0434848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0435263Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0435680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0436085Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0436231Z 2025-08-26T20:35:10.0436351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0436737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0437085Z return mod(**inputs) 2025-08-26T20:35:10.0437468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0437893Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0438307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0438721Z layer_outputs = layer_module( 2025-08-26T20:35:10.0439107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0439582Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0440012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0440430Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0440881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0441307Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0441729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0442153Z key_states = self.k(current_states) 2025-08-26T20:35:10.0442305Z 2025-08-26T20:35:10.0442420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0442823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0443189Z return mod(**inputs) 2025-08-26T20:35:10.0443596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0444009Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0444419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0444831Z layer_outputs = layer_module( 2025-08-26T20:35:10.0445215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0445631Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0446039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0446463Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0446886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0447311Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0447734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0448201Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0448406Z 2025-08-26T20:35:10.0448521Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0448909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0449263Z return mod(**inputs) 2025-08-26T20:35:10.0449633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0450056Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0450450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0450849Z layer_outputs = layer_module( 2025-08-26T20:35:10.0451221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0451597Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0451999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0452405Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0452805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0453215Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0453620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0454103Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0454332Z 2025-08-26T20:35:10.0454441Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0454827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0455166Z return mod(**inputs) 2025-08-26T20:35:10.0455540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0455946Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0456320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0456700Z layer_outputs = layer_module( 2025-08-26T20:35:10.0457049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0457436Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0457846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0458254Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0458677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0459088Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0459464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0459950Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0460194Z 2025-08-26T20:35:10.0460312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0460706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0461071Z return mod(**inputs) 2025-08-26T20:35:10.0461461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0461881Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0462288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0462674Z layer_outputs = layer_module( 2025-08-26T20:35:10.0463037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0463400Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0463779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0464169Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0464565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0464954Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0465340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0465800Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0466021Z 2025-08-26T20:35:10.0466139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0466528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0466891Z return mod(**inputs) 2025-08-26T20:35:10.0467279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0467685Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0468075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0468484Z layer_outputs = layer_module( 2025-08-26T20:35:10.0468859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0469250Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0469659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0470057Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0470463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0470853Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0471233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0471613Z value_states = self.v(current_states) 2025-08-26T20:35:10.0471765Z 2025-08-26T20:35:10.0471875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0472258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0472605Z return mod(**inputs) 2025-08-26T20:35:10.0473000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0473405Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0473805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0474211Z layer_outputs = layer_module( 2025-08-26T20:35:10.0474583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0474989Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0475383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0475791Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0476197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0476604Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0477004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0477446Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0477629Z 2025-08-26T20:35:10.0477743Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0478138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0478496Z return mod(**inputs) 2025-08-26T20:35:10.0478877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0479318Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0479817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0480239Z layer_outputs = layer_module( 2025-08-26T20:35:10.0480619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0481021Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0481443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0481835Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0482220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0482604Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0482993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0483436Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0483609Z 2025-08-26T20:35:10.0483729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0484115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0484462Z return mod(**inputs) 2025-08-26T20:35:10.0484820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0485230Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0485612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0485990Z layer_outputs = layer_module( 2025-08-26T20:35:10.0486364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0486759Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0487168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0487583Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0488020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0488437Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0488845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0489287Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0489459Z 2025-08-26T20:35:10.0489597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0489981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0490335Z return mod(**inputs) 2025-08-26T20:35:10.0490709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0491164Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0491563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0491968Z layer_outputs = layer_module( 2025-08-26T20:35:10.0492344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0492736Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0493142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0493546Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0493956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0494389Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0494795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0495205Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0495349Z 2025-08-26T20:35:10.0495439Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0495699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0496086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0496599Z return mod(**inputs) 2025-08-26T20:35:10.0496977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0497387Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0497791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0498204Z layer_outputs = layer_module( 2025-08-26T20:35:10.0498578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0498966Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0499376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0499804Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0500277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0500700Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0501123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0501528Z return self.weight * hidden_states 2025-08-26T20:35:10.0501673Z 2025-08-26T20:35:10.0501790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0502178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0502516Z return mod(**inputs) 2025-08-26T20:35:10.0502924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0503334Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0503733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0504136Z layer_outputs = layer_module( 2025-08-26T20:35:10.0504502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0504915Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0505300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0505699Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0506085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0506541Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0506981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0507410Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0507574Z 2025-08-26T20:35:10.0507693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0508074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0508423Z return mod(**inputs) 2025-08-26T20:35:10.0508789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0509242Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0509639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0510038Z layer_outputs = layer_module( 2025-08-26T20:35:10.0510395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0510767Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0511151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0511562Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0511984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0512410Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0512830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0513240Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0513383Z 2025-08-26T20:35:10.0513494Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0513884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0514234Z return mod(**inputs) 2025-08-26T20:35:10.0514607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0515045Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0515440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0515853Z layer_outputs = layer_module( 2025-08-26T20:35:10.0516244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0516630Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0517027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0517442Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0517876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0518333Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0518785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0519207Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0519396Z 2025-08-26T20:35:10.0519578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0519990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0520356Z return mod(**inputs) 2025-08-26T20:35:10.0520743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0521153Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0521564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0521984Z layer_outputs = layer_module( 2025-08-26T20:35:10.0522370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0522766Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0523185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0523617Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0524072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0524534Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0524984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0525406Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0525577Z 2025-08-26T20:35:10.0525670Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0525936Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0526332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0526697Z return mod(**inputs) 2025-08-26T20:35:10.0527084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0527505Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0527919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0528327Z layer_outputs = layer_module( 2025-08-26T20:35:10.0528712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0529115Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0529537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0529949Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0530376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0530796Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0531205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0531586Z return self.weight * hidden_states 2025-08-26T20:35:10.0531721Z 2025-08-26T20:35:10.0531825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0532196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0532541Z return mod(**inputs) 2025-08-26T20:35:10.0532932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0533335Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0533723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0534120Z layer_outputs = layer_module( 2025-08-26T20:35:10.0560123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0560927Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0561392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0561820Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0562243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0562668Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0563082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0563494Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0563651Z 2025-08-26T20:35:10.0563767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0564153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0564489Z return mod(**inputs) 2025-08-26T20:35:10.0564849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0565298Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0565690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0566077Z layer_outputs = layer_module( 2025-08-26T20:35:10.0566438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0566811Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0567201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0567592Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0567983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0568378Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0568761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0569143Z key_states = self.k(current_states) 2025-08-26T20:35:10.0569284Z 2025-08-26T20:35:10.0569395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0569771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0570106Z return mod(**inputs) 2025-08-26T20:35:10.0570466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0570883Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0571292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0571709Z layer_outputs = layer_module( 2025-08-26T20:35:10.0572085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0572470Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0572878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0573273Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0573688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0574071Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0574460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0574901Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0575109Z 2025-08-26T20:35:10.0575226Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0575593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0575926Z return mod(**inputs) 2025-08-26T20:35:10.0576307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0576723Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0577124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0577527Z layer_outputs = layer_module( 2025-08-26T20:35:10.0577894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0578285Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0578692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0579111Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0579507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0579936Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0580342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0580831Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0581063Z 2025-08-26T20:35:10.0581185Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0581570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0581926Z return mod(**inputs) 2025-08-26T20:35:10.0582311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0582719Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0583116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0583517Z layer_outputs = layer_module( 2025-08-26T20:35:10.0583887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0584278Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0584689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0585092Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0585518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0585926Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0586330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0586496Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0586503Z 2025-08-26T20:35:10.0586623Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0586839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0586912Z return mod(**inputs) 2025-08-26T20:35:10.0587195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0587275Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0587543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0587622Z layer_outputs = layer_module( 2025-08-26T20:35:10.0587863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0587970Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0588224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0588320Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0588572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0588666Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0588924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0589088Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0589094Z 2025-08-26T20:35:10.0589216Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0589432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0589512Z return mod(**inputs) 2025-08-26T20:35:10.0589773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0589882Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0590141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0590219Z layer_outputs = layer_module( 2025-08-26T20:35:10.0590464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0590550Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0590811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0590898Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0591151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0591250Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0591505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0591597Z value_states = self.v(current_states) 2025-08-26T20:35:10.0591601Z 2025-08-26T20:35:10.0591711Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0591927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0592006Z return mod(**inputs) 2025-08-26T20:35:10.0592266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0592368Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0592626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0592713Z layer_outputs = layer_module( 2025-08-26T20:35:10.0592949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0593035Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0593299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0593385Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0593660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0593750Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0594007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0594138Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0594159Z 2025-08-26T20:35:10.0594272Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0594493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0594567Z return mod(**inputs) 2025-08-26T20:35:10.0594825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0594911Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0595171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0595256Z layer_outputs = layer_module( 2025-08-26T20:35:10.0595492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0595587Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0595842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0595928Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0596363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0596527Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0596796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0596918Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0596923Z 2025-08-26T20:35:10.0597035Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0597256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0597333Z return mod(**inputs) 2025-08-26T20:35:10.0597611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0597694Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0597967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0598051Z layer_outputs = layer_module( 2025-08-26T20:35:10.0598292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0598389Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0598652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0598751Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0599055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0599149Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0599421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0599602Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0599613Z 2025-08-26T20:35:10.0599742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0599960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0600033Z return mod(**inputs) 2025-08-26T20:35:10.0600338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0600421Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0600699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0600780Z layer_outputs = layer_module( 2025-08-26T20:35:10.0601032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0601153Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0601416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0601516Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0601781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0601877Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0602145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0602231Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0602235Z 2025-08-26T20:35:10.0602360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0602578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0602656Z return mod(**inputs) 2025-08-26T20:35:10.0602924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0603032Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0603294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0603371Z layer_outputs = layer_module( 2025-08-26T20:35:10.0603626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0603711Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0603980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0604069Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0604328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-26T20:35:10.0604485Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:35:10.0604489Z 2025-08-26T20:35:10.0604580Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0604702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0604917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0604989Z return mod(**inputs) 2025-08-26T20:35:10.0605259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0605338Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0605608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0605722Z layer_outputs = layer_module( 2025-08-26T20:35:10.0605976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0606065Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0606339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0606451Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0606715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0606847Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0607123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0607209Z return self.weight * hidden_states 2025-08-26T20:35:10.0607214Z 2025-08-26T20:35:10.0607334Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0607552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0607654Z return mod(**inputs) 2025-08-26T20:35:10.0607922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0608005Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0608280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0608359Z layer_outputs = layer_module( 2025-08-26T20:35:10.0608616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0608703Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0608974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0609079Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0609366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0609510Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0609802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0609922Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0609926Z 2025-08-26T20:35:10.0610045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0610248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0610324Z return mod(**inputs) 2025-08-26T20:35:10.0610569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0610651Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0610902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0610987Z layer_outputs = layer_module( 2025-08-26T20:35:10.0611223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0611311Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0611575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0611671Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0611937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0612068Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0612350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0612447Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0612451Z 2025-08-26T20:35:10.0612565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0612799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0612872Z return mod(**inputs) 2025-08-26T20:35:10.0613136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0613216Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0613490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0613576Z layer_outputs = layer_module( 2025-08-26T20:35:10.0613812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0613905Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0614161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0614290Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0614559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0614678Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0614923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0615017Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0615021Z 2025-08-26T20:35:10.0615134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0615333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0615402Z return mod(**inputs) 2025-08-26T20:35:10.0615648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0615726Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0615985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0616083Z layer_outputs = layer_module( 2025-08-26T20:35:10.0616323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0616407Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0616666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0616771Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0617029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0617160Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0617416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0617505Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0617511Z 2025-08-26T20:35:10.0617607Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0617721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0617939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0618012Z return mod(**inputs) 2025-08-26T20:35:10.0618273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0618358Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0618637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0618724Z layer_outputs = layer_module( 2025-08-26T20:35:10.0618961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0619057Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0619313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0619400Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0619661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0619793Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0620057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0620141Z return self.weight * hidden_states 2025-08-26T20:35:10.0620147Z 2025-08-26T20:35:10.0620259Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0620476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0620566Z return mod(**inputs) 2025-08-26T20:35:10.0620832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0620913Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0621183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0621260Z layer_outputs = layer_module( 2025-08-26T20:35:10.0621496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0621588Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0621843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0621938Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0622190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0622283Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0622576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0622660Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0622664Z 2025-08-26T20:35:10.0622783Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0622999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0623067Z return mod(**inputs) 2025-08-26T20:35:10.0623315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0623393Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0623644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0623719Z layer_outputs = layer_module( 2025-08-26T20:35:10.0623944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0624026Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0624266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0624355Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0624595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0624686Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0624946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0625034Z key_states = self.k(current_states) 2025-08-26T20:35:10.0625038Z 2025-08-26T20:35:10.0625159Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0625370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0625450Z return mod(**inputs) 2025-08-26T20:35:10.0625706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0625784Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0626065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0626144Z layer_outputs = layer_module( 2025-08-26T20:35:10.0626386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0626471Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0626738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0626877Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0627118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0627212Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0627468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0627617Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0627621Z 2025-08-26T20:35:10.0627732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0627945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0628025Z return mod(**inputs) 2025-08-26T20:35:10.0628284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0628372Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0628629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0628732Z layer_outputs = layer_module( 2025-08-26T20:35:10.0628966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0629048Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0629309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0629395Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0629654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0629743Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0629995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0630173Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0630179Z 2025-08-26T20:35:10.0630291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0630506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0630576Z return mod(**inputs) 2025-08-26T20:35:10.0630841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0630919Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0631176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0631275Z layer_outputs = layer_module( 2025-08-26T20:35:10.0631515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0631607Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0631863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0631952Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0632213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0632301Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0632582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0632749Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0632753Z 2025-08-26T20:35:10.0632871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0633084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0633177Z return mod(**inputs) 2025-08-26T20:35:10.0633442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0633521Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0633784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0633863Z layer_outputs = layer_module( 2025-08-26T20:35:10.0634100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0634192Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0634445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0634540Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0634792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0634881Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0635161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0635323Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0635327Z 2025-08-26T20:35:10.0635444Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0635657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0635734Z return mod(**inputs) 2025-08-26T20:35:10.0635993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0636070Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0636334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0636413Z layer_outputs = layer_module( 2025-08-26T20:35:10.0636656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0636742Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0636999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0637094Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0637349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0637443Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0637717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0637803Z value_states = self.v(current_states) 2025-08-26T20:35:10.0637815Z 2025-08-26T20:35:10.0637925Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0638136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0638217Z return mod(**inputs) 2025-08-26T20:35:10.0638473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0638556Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0638829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0638907Z layer_outputs = layer_module( 2025-08-26T20:35:10.0639157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0639240Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0639589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0639710Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0639974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0640078Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0640343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0640475Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0640480Z 2025-08-26T20:35:10.0640593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0640818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0640893Z return mod(**inputs) 2025-08-26T20:35:10.0641168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0641257Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0641512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0641616Z layer_outputs = layer_module( 2025-08-26T20:35:10.0641852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0641936Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0642200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0642287Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0642552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0642640Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0642895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0643025Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0643031Z 2025-08-26T20:35:10.0643143Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0643366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0643437Z return mod(**inputs) 2025-08-26T20:35:10.0643703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0643781Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0644035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0644134Z layer_outputs = layer_module( 2025-08-26T20:35:10.0644373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0644465Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0644718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0644807Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0645069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0645157Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0645437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0645566Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0645570Z 2025-08-26T20:35:10.0645682Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0645879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0645945Z return mod(**inputs) 2025-08-26T20:35:10.0646220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0646295Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0646543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0646615Z layer_outputs = layer_module( 2025-08-26T20:35:10.0646841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0646929Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0647168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0647260Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0647508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0647597Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0647856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0647960Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0647964Z 2025-08-26T20:35:10.0648068Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0648173Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0648380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0648446Z return mod(**inputs) 2025-08-26T20:35:10.0648689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0648773Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0649016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0649097Z layer_outputs = layer_module( 2025-08-26T20:35:10.0649319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0649400Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0649651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0649744Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0649990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0650090Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0650341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0650431Z return self.weight * hidden_states 2025-08-26T20:35:10.0650434Z 2025-08-26T20:35:10.0650540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0650752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0650822Z return mod(**inputs) 2025-08-26T20:35:10.0651072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0651147Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0651407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0651490Z layer_outputs = layer_module( 2025-08-26T20:35:10.0651713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0651801Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0652040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0652151Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0652401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0652523Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0652770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0652875Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0652878Z 2025-08-26T20:35:10.0652994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0653206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0653278Z return mod(**inputs) 2025-08-26T20:35:10.0653545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0653625Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0653889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0653984Z layer_outputs = layer_module( 2025-08-26T20:35:10.0654232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0654319Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0654564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0654662Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0654908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0655027Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0655277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0655359Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0655364Z 2025-08-26T20:35:10.0655474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0655673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0655749Z return mod(**inputs) 2025-08-26T20:35:10.0655996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0656069Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0656323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0656410Z layer_outputs = layer_module( 2025-08-26T20:35:10.0656643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0656726Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0656967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0657068Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0657321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0657470Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0657726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0657828Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0657832Z 2025-08-26T20:35:10.0657941Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0658152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0658258Z return mod(**inputs) 2025-08-26T20:35:10.0658519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0658608Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0658869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0658946Z layer_outputs = layer_module( 2025-08-26T20:35:10.0659195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0659279Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0659547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0659640Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0659899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0660025Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0660284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0660371Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0660375Z 2025-08-26T20:35:10.0660457Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0660568Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0660767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0660834Z return mod(**inputs) 2025-08-26T20:35:10.0661087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0661162Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0661407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0661480Z layer_outputs = layer_module( 2025-08-26T20:35:10.0661704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0661794Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0662032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0662125Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0662369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0662493Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0662765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0662851Z return self.weight * hidden_states 2025-08-26T20:35:10.0662857Z 2025-08-26T20:35:10.0662977Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0663191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0663272Z return mod(**inputs) 2025-08-26T20:35:10.0663529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0663608Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0663891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0663969Z layer_outputs = layer_module( 2025-08-26T20:35:10.0664215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0664297Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0664551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0664666Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0664922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0665021Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0665273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0665366Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0665370Z 2025-08-26T20:35:10.0665480Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0665690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0665772Z return mod(**inputs) 2025-08-26T20:35:10.0666028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0666114Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0666370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0666478Z layer_outputs = layer_module( 2025-08-26T20:35:10.0666724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0666807Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0667072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0667158Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0667422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0667517Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0667781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0667871Z key_states = self.k(current_states) 2025-08-26T20:35:10.0667877Z 2025-08-26T20:35:10.0667988Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0668204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0668275Z return mod(**inputs) 2025-08-26T20:35:10.0668548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0668633Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0668890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0668989Z layer_outputs = layer_module( 2025-08-26T20:35:10.0669228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0669311Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0669585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0669672Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0669932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0670019Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0670312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0670463Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0670467Z 2025-08-26T20:35:10.0670577Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0670797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0670887Z return mod(**inputs) 2025-08-26T20:35:10.0671171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0671251Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0671523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0671607Z layer_outputs = layer_module( 2025-08-26T20:35:10.0671849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0671941Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0672219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0672304Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0672574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0672664Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0672936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0673760Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0673765Z 2025-08-26T20:35:10.0673881Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0674095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0674167Z return mod(**inputs) 2025-08-26T20:35:10.0674434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0674513Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0674775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0674851Z layer_outputs = layer_module( 2025-08-26T20:35:10.0675086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0675179Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0675436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0675529Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0675783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0675865Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0676133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0676300Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0676305Z 2025-08-26T20:35:10.0676420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0676633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0676712Z return mod(**inputs) 2025-08-26T20:35:10.0676969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0677046Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0677330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0677408Z layer_outputs = layer_module( 2025-08-26T20:35:10.0677652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0677736Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0677989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0678135Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0678388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0678483Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0678733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0678904Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0678907Z 2025-08-26T20:35:10.0679017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0679232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0679313Z return mod(**inputs) 2025-08-26T20:35:10.0679674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0679771Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0680035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0680135Z layer_outputs = layer_module( 2025-08-26T20:35:10.0680386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0680472Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0680748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0680837Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0681108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0681200Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0681457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0681549Z value_states = self.v(current_states) 2025-08-26T20:35:10.0681555Z 2025-08-26T20:35:10.0681660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0681869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0681939Z return mod(**inputs) 2025-08-26T20:35:10.0682185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0682268Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0682511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0682606Z layer_outputs = layer_module( 2025-08-26T20:35:10.0682827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0682908Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0683159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0683242Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0683487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0683569Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0683831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0683944Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0683948Z 2025-08-26T20:35:10.0684054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0684262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0684346Z return mod(**inputs) 2025-08-26T20:35:10.0684598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0684674Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0684923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0685006Z layer_outputs = layer_module( 2025-08-26T20:35:10.0685241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0685331Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0685584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0685671Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0685932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0686020Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0686282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0686417Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0686421Z 2025-08-26T20:35:10.0686536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0686745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0686815Z return mod(**inputs) 2025-08-26T20:35:10.0687078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0687155Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0687404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0687478Z layer_outputs = layer_module( 2025-08-26T20:35:10.0687698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0687785Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0688026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0688114Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0688353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0688441Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0688694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0688807Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0688810Z 2025-08-26T20:35:10.0688921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0689121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0689195Z return mod(**inputs) 2025-08-26T20:35:10.0689438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0689511Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0689779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0689854Z layer_outputs = layer_module( 2025-08-26T20:35:10.0690082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0690163Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0690402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0690507Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0690747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0690837Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0691075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0691162Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0691166Z 2025-08-26T20:35:10.0691270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0691471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0691546Z return mod(**inputs) 2025-08-26T20:35:10.0691795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0691878Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0692123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0692214Z layer_outputs = layer_module( 2025-08-26T20:35:10.0692443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0692524Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0692771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0692853Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0693111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-26T20:35:10.0693256Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:35:10.0693260Z 2025-08-26T20:35:10.0693347Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0693465Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0693678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0693757Z return mod(**inputs) 2025-08-26T20:35:10.0694015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0694093Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0694356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0694434Z layer_outputs = layer_module( 2025-08-26T20:35:10.0694675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0694788Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0695045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0695152Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0695409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0695522Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0695775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0695886Z return self.weight * hidden_states 2025-08-26T20:35:10.0695890Z 2025-08-26T20:35:10.0696001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0696419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0696507Z return mod(**inputs) 2025-08-26T20:35:10.0696768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0696926Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0697186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0697264Z layer_outputs = layer_module( 2025-08-26T20:35:10.0697510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0697597Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0697866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0697966Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0698221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0698356Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0698611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0698729Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0698760Z 2025-08-26T20:35:10.0698871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0699092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0699166Z return mod(**inputs) 2025-08-26T20:35:10.0699423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0699514Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0699768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0699856Z layer_outputs = layer_module( 2025-08-26T20:35:10.0700091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0700179Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0700439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0700537Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0700797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0700922Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0701186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0701271Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0701275Z 2025-08-26T20:35:10.0701412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0701633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0701706Z return mod(**inputs) 2025-08-26T20:35:10.0701971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0702050Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0702309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0702393Z layer_outputs = layer_module( 2025-08-26T20:35:10.0702652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0702744Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0702996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0703099Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0703353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0703495Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0703756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0703850Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0703854Z 2025-08-26T20:35:10.0703967Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0704176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0704246Z return mod(**inputs) 2025-08-26T20:35:10.0704510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0704589Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0704849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0704926Z layer_outputs = layer_module( 2025-08-26T20:35:10.0705161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0705273Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0705526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0705629Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0705884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0706013Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0706265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0706351Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0706355Z 2025-08-26T20:35:10.0706453Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0706561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0706779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0706848Z return mod(**inputs) 2025-08-26T20:35:10.0707104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0707189Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0707446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0707530Z layer_outputs = layer_module( 2025-08-26T20:35:10.0707790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0707876Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0708148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0708238Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0708503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0708617Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0708885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0708994Z return self.weight * hidden_states 2025-08-26T20:35:10.0708998Z 2025-08-26T20:35:10.0709103Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0709310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0709377Z return mod(**inputs) 2025-08-26T20:35:10.0709628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0709722Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0709968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0710049Z layer_outputs = layer_module( 2025-08-26T20:35:10.0710271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0710358Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0710600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0710690Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0710936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0711021Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0711268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0711347Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0711371Z 2025-08-26T20:35:10.0711485Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0711685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0711753Z return mod(**inputs) 2025-08-26T20:35:10.0712013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0712092Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0712359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0712436Z layer_outputs = layer_module( 2025-08-26T20:35:10.0712669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0712762Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0713027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0713122Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0713385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0713479Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0713742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0713824Z key_states = self.k(current_states) 2025-08-26T20:35:10.0713828Z 2025-08-26T20:35:10.0713962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0714174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0714251Z return mod(**inputs) 2025-08-26T20:35:10.0714519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0714599Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0714875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0714952Z layer_outputs = layer_module( 2025-08-26T20:35:10.0715227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0715313Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0715585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0715674Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0715940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0716066Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0716339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0716493Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0716497Z 2025-08-26T20:35:10.0716609Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0716827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0716908Z return mod(**inputs) 2025-08-26T20:35:10.0717187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0717273Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0717527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0717604Z layer_outputs = layer_module( 2025-08-26T20:35:10.0717846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0717957Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0718221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0718308Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0718571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0718662Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0718922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0719102Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0719107Z 2025-08-26T20:35:10.0719222Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0719503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0719585Z return mod(**inputs) 2025-08-26T20:35:10.0719841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0719926Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0720193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0720277Z layer_outputs = layer_module( 2025-08-26T20:35:10.0720515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0720629Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0720901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0720990Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0721252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0721340Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0721601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0721783Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0721787Z 2025-08-26T20:35:10.0721899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0722123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0722196Z return mod(**inputs) 2025-08-26T20:35:10.0722464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0722561Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0722820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0722905Z layer_outputs = layer_module( 2025-08-26T20:35:10.0723138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0723230Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0723489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0723579Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0723819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0723901Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0724158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0724320Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0724342Z 2025-08-26T20:35:10.0724461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0724672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0724742Z return mod(**inputs) 2025-08-26T20:35:10.0725008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0725088Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0725350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0725428Z layer_outputs = layer_module( 2025-08-26T20:35:10.0725673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0725759Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0726014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0726111Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0726364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0726460Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0726716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0726800Z value_states = self.v(current_states) 2025-08-26T20:35:10.0726804Z 2025-08-26T20:35:10.0726939Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0727154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0727233Z return mod(**inputs) 2025-08-26T20:35:10.0727490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0727570Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0727835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0727912Z layer_outputs = layer_module( 2025-08-26T20:35:10.0728182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0728271Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0728538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0728630Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0728885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0728998Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0729253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0729380Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0729384Z 2025-08-26T20:35:10.0729494Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0729706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0729785Z return mod(**inputs) 2025-08-26T20:35:10.0730045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0730131Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0730388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0730473Z layer_outputs = layer_module( 2025-08-26T20:35:10.0730708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0730811Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0731075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0731161Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0731423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0731510Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0731761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0731886Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0731890Z 2025-08-26T20:35:10.0731999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0732219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0732291Z return mod(**inputs) 2025-08-26T20:35:10.0732554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0732632Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0732888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0732972Z layer_outputs = layer_module( 2025-08-26T20:35:10.0733203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0733311Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0733569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0733655Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0733915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0734002Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0734263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0734378Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0734399Z 2025-08-26T20:35:10.0734510Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0734726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0734796Z return mod(**inputs) 2025-08-26T20:35:10.0735060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0735139Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0735418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0735495Z layer_outputs = layer_module( 2025-08-26T20:35:10.0735728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0735819Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0736074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0736166Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0736418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0736507Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0736765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0736847Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0736851Z 2025-08-26T20:35:10.0736946Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0737077Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0737287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0737363Z return mod(**inputs) 2025-08-26T20:35:10.0737621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0737706Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0737963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0738047Z layer_outputs = layer_module( 2025-08-26T20:35:10.0738285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0738371Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0738636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0738735Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0738995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0739100Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0739355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0739446Z return self.weight * hidden_states 2025-08-26T20:35:10.0739450Z 2025-08-26T20:35:10.0739583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0739788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0739856Z return mod(**inputs) 2025-08-26T20:35:10.0740106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0740179Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0740420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0740502Z layer_outputs = layer_module( 2025-08-26T20:35:10.0740743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0740833Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0741076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0741169Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0741418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0741561Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0741807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0741909Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0741912Z 2025-08-26T20:35:10.0742015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0742223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0742289Z return mod(**inputs) 2025-08-26T20:35:10.0742538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0742614Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0742862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0742936Z layer_outputs = layer_module( 2025-08-26T20:35:10.0743164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0743274Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0743526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0743628Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0743883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0744006Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0744270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0744357Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0744361Z 2025-08-26T20:35:10.0744478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0744690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0744769Z return mod(**inputs) 2025-08-26T20:35:10.0745027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0745105Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0745373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0745450Z layer_outputs = layer_module( 2025-08-26T20:35:10.0745690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0745793Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0746048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0746156Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0746408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0746539Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0746790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0746900Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0746912Z 2025-08-26T20:35:10.0747023Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0747236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0747317Z return mod(**inputs) 2025-08-26T20:35:10.0747575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0747680Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0747938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0748017Z layer_outputs = layer_module( 2025-08-26T20:35:10.0748259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0748343Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0748605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0748702Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0748958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0749088Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0749342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0749436Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0749459Z 2025-08-26T20:35:10.0749548Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0749664Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0749879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0749947Z return mod(**inputs) 2025-08-26T20:35:10.0750218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0750292Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0750547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0750620Z layer_outputs = layer_module( 2025-08-26T20:35:10.0750844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0750930Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0751171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0751260Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0751502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0751611Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0751861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0751940Z return self.weight * hidden_states 2025-08-26T20:35:10.0751962Z 2025-08-26T20:35:10.0752075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0752274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0752349Z return mod(**inputs) 2025-08-26T20:35:10.0752594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0752670Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0752920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0752991Z layer_outputs = layer_module( 2025-08-26T20:35:10.0753237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0753320Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0753568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0753663Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0753922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0754036Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0754293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0754376Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0754388Z 2025-08-26T20:35:10.0754498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0754713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0754793Z return mod(**inputs) 2025-08-26T20:35:10.0755048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0755134Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0755390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0755468Z layer_outputs = layer_module( 2025-08-26T20:35:10.0755709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0755814Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0756084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0756171Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0756432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0756530Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0756793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0756884Z key_states = self.k(current_states) 2025-08-26T20:35:10.0756888Z 2025-08-26T20:35:10.0756998Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0757218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0757291Z return mod(**inputs) 2025-08-26T20:35:10.0757547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0757632Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0757899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0757981Z layer_outputs = layer_module( 2025-08-26T20:35:10.0758215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0758316Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0758595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0758683Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0758951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0759041Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0759296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0759741Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0759749Z 2025-08-26T20:35:10.0759870Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0760103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0760180Z return mod(**inputs) 2025-08-26T20:35:10.0760464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0760585Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0760842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0760929Z layer_outputs = layer_module( 2025-08-26T20:35:10.0761166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0761261Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0761517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0761603Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0761868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0761956Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0762217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0762386Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0762412Z 2025-08-26T20:35:10.0762531Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0762746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0762816Z return mod(**inputs) 2025-08-26T20:35:10.0763086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0763164Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0763428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0763507Z layer_outputs = layer_module( 2025-08-26T20:35:10.0763743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0763836Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0764090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0764185Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0764439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0764528Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0764792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0764956Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0764960Z 2025-08-26T20:35:10.0765092Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0765308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0765387Z return mod(**inputs) 2025-08-26T20:35:10.0765644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0765723Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0765985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0766059Z layer_outputs = layer_module( 2025-08-26T20:35:10.0766325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0766412Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0766671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0766766Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0767024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0767143Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0767398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0767563Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0767567Z 2025-08-26T20:35:10.0767676Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0767886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0767966Z return mod(**inputs) 2025-08-26T20:35:10.0768222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0768308Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0768564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0768642Z layer_outputs = layer_module( 2025-08-26T20:35:10.0768883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0768982Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0769228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0769310Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0769553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0769645Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0769891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0769978Z value_states = self.v(current_states) 2025-08-26T20:35:10.0769982Z 2025-08-26T20:35:10.0770087Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0770295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0770363Z return mod(**inputs) 2025-08-26T20:35:10.0770607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0770690Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0770950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0771035Z layer_outputs = layer_module( 2025-08-26T20:35:10.0771271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0771381Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0771632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0771716Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0771974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0772065Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0772336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0772470Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0772474Z 2025-08-26T20:35:10.0772585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0772806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0772876Z return mod(**inputs) 2025-08-26T20:35:10.0773151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0773247Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0773508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0773595Z layer_outputs = layer_module( 2025-08-26T20:35:10.0773832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0773923Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0774191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0774278Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0774552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0774640Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0774906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0775023Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0775047Z 2025-08-26T20:35:10.0775165Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0775374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0775444Z return mod(**inputs) 2025-08-26T20:35:10.0775725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0775803Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0776067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0776140Z layer_outputs = layer_module( 2025-08-26T20:35:10.0776362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0776453Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0776693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0776785Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0777024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0777113Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0777358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0777466Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0777470Z 2025-08-26T20:35:10.0777600Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0777802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0777877Z return mod(**inputs) 2025-08-26T20:35:10.0778123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0778201Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0778454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0778526Z layer_outputs = layer_module( 2025-08-26T20:35:10.0778801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0778883Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0779127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0779217Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0779459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0779577Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0779843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0779935Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0779939Z 2025-08-26T20:35:10.0780049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0780261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0780339Z return mod(**inputs) 2025-08-26T20:35:10.0780598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0780681Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0780954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0781029Z layer_outputs = layer_module( 2025-08-26T20:35:10.0781276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0781382Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0781642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0781725Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0781966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-26T20:35:10.0782107Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:35:10.0782111Z 2025-08-26T20:35:10.0782194Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0782307Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0782506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0782580Z return mod(**inputs) 2025-08-26T20:35:10.0782819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0782893Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0783140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0783211Z layer_outputs = layer_module( 2025-08-26T20:35:10.0783441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0783521Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0783759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0783875Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0784121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0784228Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0784470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0784562Z return self.weight * hidden_states 2025-08-26T20:35:10.0784566Z 2025-08-26T20:35:10.0784674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0784905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0784984Z return mod(**inputs) 2025-08-26T20:35:10.0785242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0785327Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0785587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0785682Z layer_outputs = layer_module( 2025-08-26T20:35:10.0785937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0786021Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0786274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0786368Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0786612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0786739Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0786988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0787099Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0787103Z 2025-08-26T20:35:10.0787211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0787435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0787523Z return mod(**inputs) 2025-08-26T20:35:10.0787780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0787866Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0788127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0788220Z layer_outputs = layer_module( 2025-08-26T20:35:10.0788454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0788541Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0788804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0788902Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0789165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0789291Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0789550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0789633Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0789639Z 2025-08-26T20:35:10.0789749Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0789969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0790038Z return mod(**inputs) 2025-08-26T20:35:10.0790332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0790413Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0790670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0790756Z layer_outputs = layer_module( 2025-08-26T20:35:10.0790989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0791078Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0791346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0791442Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0791704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0791828Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0792091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0792204Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0792210Z 2025-08-26T20:35:10.0792326Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0792539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0792610Z return mod(**inputs) 2025-08-26T20:35:10.0792874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0792953Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0793217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0793296Z layer_outputs = layer_module( 2025-08-26T20:35:10.0793528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0793620Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0793871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0793990Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0794249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0794376Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0794635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0794721Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0794725Z 2025-08-26T20:35:10.0794822Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0794931Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0795152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0795224Z return mod(**inputs) 2025-08-26T20:35:10.0795482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-26T20:35:10.0795567Z encoder_outputs = self.encoder( 2025-08-26T20:35:10.0795827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-08-26T20:35:10.0795946Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-26T20:35:10.0796432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0796523Z return self.weight * hidden_states 2025-08-26T20:35:10.0796535Z 2025-08-26T20:35:10.0796691Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0796909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0796991Z return mod(**inputs) 2025-08-26T20:35:10.0797251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0797337Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0797593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0797671Z layer_outputs = layer_module( 2025-08-26T20:35:10.0797941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0798028Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0798291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0798380Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0798637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0798770Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0799027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0799121Z key_states = self.k(current_states) 2025-08-26T20:35:10.0799125Z 2025-08-26T20:35:10.0799235Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0799499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0799579Z return mod(**inputs) 2025-08-26T20:35:10.0799840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0799924Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0800190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0800275Z layer_outputs = layer_module( 2025-08-26T20:35:10.0800521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0800652Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0800897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0800977Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0801225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0801309Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0801549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0801692Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0801695Z 2025-08-26T20:35:10.0801800Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0802014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0802084Z return mod(**inputs) 2025-08-26T20:35:10.0802332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0802407Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0802651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0802736Z layer_outputs = layer_module( 2025-08-26T20:35:10.0802959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0803061Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0803304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0803385Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0803633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0803719Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0803966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0804122Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0804142Z 2025-08-26T20:35:10.0804254Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0804456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0804521Z return mod(**inputs) 2025-08-26T20:35:10.0804774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0804847Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0805115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0805189Z layer_outputs = layer_module( 2025-08-26T20:35:10.0805410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0805495Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0805736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0805825Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0806063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0806149Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0806399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0806480Z value_states = self.v(current_states) 2025-08-26T20:35:10.0806484Z 2025-08-26T20:35:10.0806613Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0806811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0806885Z return mod(**inputs) 2025-08-26T20:35:10.0807132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0807205Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0807461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0807534Z layer_outputs = layer_module( 2025-08-26T20:35:10.0807768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0807848Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0808091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0808182Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0808423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0808515Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0808761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0808878Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0808882Z 2025-08-26T20:35:10.0808989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0809235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0809314Z return mod(**inputs) 2025-08-26T20:35:10.0809560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0809642Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0809883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0809956Z layer_outputs = layer_module( 2025-08-26T20:35:10.0810201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0810282Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0810528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0810614Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0810853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0810959Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0811201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0811322Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0811327Z 2025-08-26T20:35:10.0811432Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0811640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0811722Z return mod(**inputs) 2025-08-26T20:35:10.0811963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0812048Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0812290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0812373Z layer_outputs = layer_module( 2025-08-26T20:35:10.0812600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0812711Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0812960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0813041Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0813299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0813384Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0813644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0813762Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0813766Z 2025-08-26T20:35:10.0813877Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0814102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0814173Z return mod(**inputs) 2025-08-26T20:35:10.0814459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0814539Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0814810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0814897Z layer_outputs = layer_module( 2025-08-26T20:35:10.0815135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0815227Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0815507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0815597Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0815874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0815965Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0816234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0816321Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0816326Z 2025-08-26T20:35:10.0816444Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0816560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0816778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0816860Z return mod(**inputs) 2025-08-26T20:35:10.0817125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0817213Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0817508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0817589Z layer_outputs = layer_module( 2025-08-26T20:35:10.0817837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0817924Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0818198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0818297Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0818568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0818687Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0818956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0819042Z return self.weight * hidden_states 2025-08-26T20:35:10.0819046Z 2025-08-26T20:35:10.0819166Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0819372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0819438Z return mod(**inputs) 2025-08-26T20:35:10.0819685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0819766Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0820010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0820090Z layer_outputs = layer_module( 2025-08-26T20:35:10.0820315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0820395Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0820647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0820740Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0821015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0821145Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0821430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0821539Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0821543Z 2025-08-26T20:35:10.0821674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0821904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0821975Z return mod(**inputs) 2025-08-26T20:35:10.0822254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0822346Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0822615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0822702Z layer_outputs = layer_module( 2025-08-26T20:35:10.0822952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0823054Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0823296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0823388Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0823633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0823767Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0824014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0824097Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0824101Z 2025-08-26T20:35:10.0824211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0824413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0824478Z return mod(**inputs) 2025-08-26T20:35:10.0824730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0824807Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0825064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0825141Z layer_outputs = layer_module( 2025-08-26T20:35:10.0825375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0825486Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0825740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0825840Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0826094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0826223Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0826488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0826583Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0826587Z 2025-08-26T20:35:10.0826704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0826929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0827015Z return mod(**inputs) 2025-08-26T20:35:10.0827258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0827332Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0827582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0827655Z layer_outputs = layer_module( 2025-08-26T20:35:10.0827881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0827978Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0828220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0828321Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0828574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0828706Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0828961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0829053Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0829075Z 2025-08-26T20:35:10.0829188Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0829400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0829477Z return mod(**inputs) 2025-08-26T20:35:10.0829738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0829828Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0830117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0830198Z layer_outputs = layer_module( 2025-08-26T20:35:10.0830457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0830542Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0830810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0830900Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0831165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0831283Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0831541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0831635Z return self.weight * hidden_states 2025-08-26T20:35:10.0831639Z 2025-08-26T20:35:10.0831769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0831989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0832059Z return mod(**inputs) 2025-08-26T20:35:10.0832323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0832406Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0832654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0832732Z layer_outputs = layer_module( 2025-08-26T20:35:10.0832955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0833034Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0833283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0833367Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0833615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0833701Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0833947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0834027Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0834031Z 2025-08-26T20:35:10.0834135Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0834359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0834427Z return mod(**inputs) 2025-08-26T20:35:10.0834677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0834753Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0834995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0835076Z layer_outputs = layer_module( 2025-08-26T20:35:10.0835296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0835401Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0835647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0835736Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0835988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0836079Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0836368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0836453Z key_states = self.k(current_states) 2025-08-26T20:35:10.0836457Z 2025-08-26T20:35:10.0836574Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0836787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0836858Z return mod(**inputs) 2025-08-26T20:35:10.0837124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0837203Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0837469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0837545Z layer_outputs = layer_module( 2025-08-26T20:35:10.0837779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0837873Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0838148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0838241Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0838498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0838594Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0838854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0838995Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0838999Z 2025-08-26T20:35:10.0839116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0839326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0839406Z return mod(**inputs) 2025-08-26T20:35:10.0839747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0839833Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0840108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0840191Z layer_outputs = layer_module( 2025-08-26T20:35:10.0840449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0840535Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0840818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0840907Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0841164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0841263Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0841517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0841695Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0841699Z 2025-08-26T20:35:10.0841826Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0842045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0842121Z return mod(**inputs) 2025-08-26T20:35:10.0842368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0842450Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0842693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0842786Z layer_outputs = layer_module( 2025-08-26T20:35:10.0843018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0843098Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0843346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0843429Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0843678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0843763Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0844006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0844093Z value_states = self.v(current_states) 2025-08-26T20:35:10.0844098Z 2025-08-26T20:35:10.0844202Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0844430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0844497Z return mod(**inputs) 2025-08-26T20:35:10.0844739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0844821Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0845065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0845145Z layer_outputs = layer_module( 2025-08-26T20:35:10.0845370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0845456Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0845698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0845783Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0846041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0846122Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0846365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0846477Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0846481Z 2025-08-26T20:35:10.0846583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0846802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0846868Z return mod(**inputs) 2025-08-26T20:35:10.0847119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0847193Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0847430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0847510Z layer_outputs = layer_module( 2025-08-26T20:35:10.0847727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0847825Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0848060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0848148Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0848390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0848470Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0848730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0848840Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0848843Z 2025-08-26T20:35:10.0848952Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0849148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0849213Z return mod(**inputs) 2025-08-26T20:35:10.0849460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0849533Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0849780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0849851Z layer_outputs = layer_module( 2025-08-26T20:35:10.0850078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0850162Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0850424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0850513Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0850751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0850841Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0851083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0851191Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0851196Z 2025-08-26T20:35:10.0851308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0851508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0851584Z return mod(**inputs) 2025-08-26T20:35:10.0851828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0851904Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0852159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0852233Z layer_outputs = layer_module( 2025-08-26T20:35:10.0852469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0852549Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0852813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0852898Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0853138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0853229Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0853478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0853562Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0853566Z 2025-08-26T20:35:10.0853646Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0853762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0853966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0854030Z return mod(**inputs) 2025-08-26T20:35:10.0854277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0854350Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0854589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0854683Z layer_outputs = layer_module( 2025-08-26T20:35:10.0854903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0854988Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0855231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0855320Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0855560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-26T20:35:10.0855669Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0855920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0855998Z return self.weight * hidden_states 2025-08-26T20:35:10.0856003Z 2025-08-26T20:35:10.0856114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0856333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0856398Z return mod(**inputs) 2025-08-26T20:35:10.0856651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0856725Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0856989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0857066Z layer_outputs = layer_module( 2025-08-26T20:35:10.0857310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0857395Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0857650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0857745Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0858001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0858099Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0858355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0858440Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0858444Z 2025-08-26T20:35:10.0858560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0858787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0858867Z return mod(**inputs) 2025-08-26T20:35:10.0859127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0859207Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0859472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0859549Z layer_outputs = layer_module( 2025-08-26T20:35:10.0859794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0859878Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0860169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0860258Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0860512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0860619Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0860858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0860963Z key_states = self.k(current_states) 2025-08-26T20:35:10.0860968Z 2025-08-26T20:35:10.0861073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0861274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0861349Z return mod(**inputs) 2025-08-26T20:35:10.0861596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0861678Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0861923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0861998Z layer_outputs = layer_module( 2025-08-26T20:35:10.0862229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0862313Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0862574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0862680Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0862947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0863031Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0863271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0863409Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0863413Z 2025-08-26T20:35:10.0863519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0863728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0863796Z return mod(**inputs) 2025-08-26T20:35:10.0864040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0864122Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0864364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0864444Z layer_outputs = layer_module( 2025-08-26T20:35:10.0864668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0864753Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0865016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0865101Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0865346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0865432Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0865679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0865836Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0865839Z 2025-08-26T20:35:10.0865942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0866166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0866235Z return mod(**inputs) 2025-08-26T20:35:10.0866487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0866564Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0866817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0866909Z layer_outputs = layer_module( 2025-08-26T20:35:10.0867133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0867222Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0867465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0867554Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0867803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0867892Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0868157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0868242Z value_states = self.v(current_states) 2025-08-26T20:35:10.0868246Z 2025-08-26T20:35:10.0868369Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0868584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0868673Z return mod(**inputs) 2025-08-26T20:35:10.0868935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0869012Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0869274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0869353Z layer_outputs = layer_module( 2025-08-26T20:35:10.0869593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0869678Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0869931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0870023Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0870274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0870376Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0870613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0870722Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0870726Z 2025-08-26T20:35:10.0870839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0871037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0871125Z return mod(**inputs) 2025-08-26T20:35:10.0871370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0871446Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0871696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0871770Z layer_outputs = layer_module( 2025-08-26T20:35:10.0872001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0872080Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0872346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0872429Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0872669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0872760Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0873001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0873134Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0873140Z 2025-08-26T20:35:10.0873244Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0873452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0873525Z return mod(**inputs) 2025-08-26T20:35:10.0873775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0873858Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0874119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0874207Z layer_outputs = layer_module( 2025-08-26T20:35:10.0874448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0874532Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0874803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0874907Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0875170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0875258Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0875526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0875650Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0875654Z 2025-08-26T20:35:10.0875766Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0875986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0876058Z return mod(**inputs) 2025-08-26T20:35:10.0876327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0876414Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0876685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0876772Z layer_outputs = layer_module( 2025-08-26T20:35:10.0877009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0877100Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0877363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0877487Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0877750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0877841Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0878112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0878197Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0878201Z 2025-08-26T20:35:10.0878287Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0878406Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0878636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0878716Z return mod(**inputs) 2025-08-26T20:35:10.0878982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0879063Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0879343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0879439Z layer_outputs = layer_module( 2025-08-26T20:35:10.0879767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0879854Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0880119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0880219Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0880486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0880600Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0880868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0880959Z return self.weight * hidden_states 2025-08-26T20:35:10.0880963Z 2025-08-26T20:35:10.0881075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0881286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0881386Z return mod(**inputs) 2025-08-26T20:35:10.0881644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0881733Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0882004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0882091Z layer_outputs = layer_module( 2025-08-26T20:35:10.0882326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0882411Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0882677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0882774Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0883036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0883165Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0883419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0883536Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0883540Z 2025-08-26T20:35:10.0883648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0883869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0883966Z return mod(**inputs) 2025-08-26T20:35:10.0884243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0884325Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0884591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0884676Z layer_outputs = layer_module( 2025-08-26T20:35:10.0884910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0885002Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0885276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0885374Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0885635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0885759Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0886018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0886120Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0886126Z 2025-08-26T20:35:10.0886235Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0886452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0886522Z return mod(**inputs) 2025-08-26T20:35:10.0886788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0886869Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0887131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0887210Z layer_outputs = layer_module( 2025-08-26T20:35:10.0887447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0887539Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0887792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0887913Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0888170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0888294Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0888561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0888655Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0888659Z 2025-08-26T20:35:10.0888778Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0888999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0889081Z return mod(**inputs) 2025-08-26T20:35:10.0889353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0889435Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0889712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0889791Z layer_outputs = layer_module( 2025-08-26T20:35:10.0890046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0890133Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0890425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0890532Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0890789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0890918Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0891175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0891262Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0891273Z 2025-08-26T20:35:10.0891361Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0891489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0891713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0891782Z return mod(**inputs) 2025-08-26T20:35:10.0892049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0892127Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0892381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0892484Z layer_outputs = layer_module( 2025-08-26T20:35:10.0892720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0892810Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0893065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0893154Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0893417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0893532Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0893795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0893879Z return self.weight * hidden_states 2025-08-26T20:35:10.0893884Z 2025-08-26T20:35:10.0894001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0894231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0894301Z return mod(**inputs) 2025-08-26T20:35:10.0894564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0894642Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0894907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0894984Z layer_outputs = layer_module( 2025-08-26T20:35:10.0895217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0895309Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0895560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0895655Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0895910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0896000Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0896429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0896521Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0896526Z 2025-08-26T20:35:10.0896644Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0896904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0896990Z return mod(**inputs) 2025-08-26T20:35:10.0897255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0897337Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0897616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0897698Z layer_outputs = layer_module( 2025-08-26T20:35:10.0897951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0898039Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0898332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0898427Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0898681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0898779Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0899035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0899144Z key_states = self.k(current_states) 2025-08-26T20:35:10.0899158Z 2025-08-26T20:35:10.0899269Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0899489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0899569Z return mod(**inputs) 2025-08-26T20:35:10.0899836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0899923Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0900187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0900267Z layer_outputs = layer_module( 2025-08-26T20:35:10.0900515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0900603Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0900872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0900990Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0901254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0901353Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0901617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0901765Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0901769Z 2025-08-26T20:35:10.0901882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0902109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0902184Z return mod(**inputs) 2025-08-26T20:35:10.0902450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0902539Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0902801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0902885Z layer_outputs = layer_module( 2025-08-26T20:35:10.0903132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0903218Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0903507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0903598Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0903868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0903960Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0904227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0904407Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0904411Z 2025-08-26T20:35:10.0904526Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0904772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0904847Z return mod(**inputs) 2025-08-26T20:35:10.0905120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0905203Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0905469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0905577Z layer_outputs = layer_module( 2025-08-26T20:35:10.0905825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0905921Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0906193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0906282Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0906565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0906657Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0906933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0907013Z value_states = self.v(current_states) 2025-08-26T20:35:10.0907016Z 2025-08-26T20:35:10.0907129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0907335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0907432Z return mod(**inputs) 2025-08-26T20:35:10.0907692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0907766Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0908023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0908096Z layer_outputs = layer_module( 2025-08-26T20:35:10.0908320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0908411Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0908664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0908755Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0908999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0909084Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0909340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0909452Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0909456Z 2025-08-26T20:35:10.0909567Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0909771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0909865Z return mod(**inputs) 2025-08-26T20:35:10.0910111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0910188Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0910439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0910515Z layer_outputs = layer_module( 2025-08-26T20:35:10.0910744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0910823Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0911081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0911178Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0911435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0911530Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0911785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0911922Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0911934Z 2025-08-26T20:35:10.0912044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0912255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0912336Z return mod(**inputs) 2025-08-26T20:35:10.0912599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0912684Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0912942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0913022Z layer_outputs = layer_module( 2025-08-26T20:35:10.0913276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0913366Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0913635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0913745Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0914005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0914103Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0914370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0914494Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0914497Z 2025-08-26T20:35:10.0914614Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0914837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0914911Z return mod(**inputs) 2025-08-26T20:35:10.0915175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0915264Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0915528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0915613Z layer_outputs = layer_module( 2025-08-26T20:35:10.0915853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0915940Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0916211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0916323Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0916592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0916684Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0916966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0917058Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0917062Z 2025-08-26T20:35:10.0917172Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0917390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0917481Z return mod(**inputs) 2025-08-26T20:35:10.0917756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0917834Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0918101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0918184Z layer_outputs = layer_module( 2025-08-26T20:35:10.0919279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0919376Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0919709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0919803Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0920086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-26T20:35:10.0920233Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:35:10.0920238Z 2025-08-26T20:35:10.0920340Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0920456Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0920687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0920757Z return mod(**inputs) 2025-08-26T20:35:10.0921004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0921114Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0921357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0921437Z layer_outputs = layer_module( 2025-08-26T20:35:10.0921662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0921743Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0921994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0922081Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0922329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-26T20:35:10.0922438Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0922678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0922765Z return self.weight * hidden_states 2025-08-26T20:35:10.0922768Z 2025-08-26T20:35:10.0922872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0923082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0923152Z return mod(**inputs) 2025-08-26T20:35:10.0923403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0923476Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0923744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0923830Z layer_outputs = layer_module( 2025-08-26T20:35:10.0924050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0924136Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0924375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0924456Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0924725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0924813Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0925062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0925141Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0925145Z 2025-08-26T20:35:10.0925254Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0925471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0925540Z return mod(**inputs) 2025-08-26T20:35:10.0925808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0925887Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0926165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0926242Z layer_outputs = layer_module( 2025-08-26T20:35:10.0926478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0926569Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0926832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0926926Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0927180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0927288Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0927555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0927631Z key_states = self.k(current_states) 2025-08-26T20:35:10.0927635Z 2025-08-26T20:35:10.0927747Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0927949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0928026Z return mod(**inputs) 2025-08-26T20:35:10.0928286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0928365Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0928635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0928715Z layer_outputs = layer_module( 2025-08-26T20:35:10.0928958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0929042Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0929300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0929392Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0929650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0929767Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0930022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0930163Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0930174Z 2025-08-26T20:35:10.0930286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0930498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0930576Z return mod(**inputs) 2025-08-26T20:35:10.0930835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0930937Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0931195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0931273Z layer_outputs = layer_module( 2025-08-26T20:35:10.0931517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0931600Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0931888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0931976Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0932230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0932326Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0932581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0932754Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0932758Z 2025-08-26T20:35:10.0932868Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0933086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0933157Z return mod(**inputs) 2025-08-26T20:35:10.0933419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0933524Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0933783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0933866Z layer_outputs = layer_module( 2025-08-26T20:35:10.0934107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0934191Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0934454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0934542Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0934805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0934896Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0935152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0935243Z value_states = self.v(current_states) 2025-08-26T20:35:10.0935247Z 2025-08-26T20:35:10.0935357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0935578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0935651Z return mod(**inputs) 2025-08-26T20:35:10.0935919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0935997Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0936273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0936362Z layer_outputs = layer_module( 2025-08-26T20:35:10.0936597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0936690Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0936948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0937034Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0937315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0937404Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0937666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0937787Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0937791Z 2025-08-26T20:35:10.0937910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0938144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0938211Z return mod(**inputs) 2025-08-26T20:35:10.0938462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0938536Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0938786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0938861Z layer_outputs = layer_module( 2025-08-26T20:35:10.0939083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0939169Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0939414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0939506Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0939751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0939864Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0940115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0940224Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0940228Z 2025-08-26T20:35:10.0940341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0940543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0940617Z return mod(**inputs) 2025-08-26T20:35:10.0940865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0940941Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0941191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0941266Z layer_outputs = layer_module( 2025-08-26T20:35:10.0941498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0941578Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0941821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0941911Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0942153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0942245Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0942503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0942620Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0942625Z 2025-08-26T20:35:10.0942730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0942936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0943014Z return mod(**inputs) 2025-08-26T20:35:10.0943263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0943363Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0943611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0943685Z layer_outputs = layer_module( 2025-08-26T20:35:10.0943917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0943998Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0944245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0944346Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0944588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0944678Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0944920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0945009Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0945012Z 2025-08-26T20:35:10.0945096Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0945208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0945409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0945478Z return mod(**inputs) 2025-08-26T20:35:10.0945731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0945822Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0946070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0946142Z layer_outputs = layer_module( 2025-08-26T20:35:10.0946364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0946451Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0946690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0946792Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0947034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.0947136Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0947383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0947464Z return self.weight * hidden_states 2025-08-26T20:35:10.0947468Z 2025-08-26T20:35:10.0947579Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0947780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0947858Z return mod(**inputs) 2025-08-26T20:35:10.0948102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0948175Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0948442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0948517Z layer_outputs = layer_module( 2025-08-26T20:35:10.0948747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0948828Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0949071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0949172Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0949431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0949562Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0949801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.0949911Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.0949914Z 2025-08-26T20:35:10.0950020Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0950240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0950317Z return mod(**inputs) 2025-08-26T20:35:10.0950564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0950645Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0950887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0950963Z layer_outputs = layer_module( 2025-08-26T20:35:10.0951194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0951272Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0951519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0951611Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0951850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0951997Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0952241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.0952329Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.0952333Z 2025-08-26T20:35:10.0952439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0952645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0952712Z return mod(**inputs) 2025-08-26T20:35:10.0952957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0953039Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0953284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0953365Z layer_outputs = layer_module( 2025-08-26T20:35:10.0953594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0953678Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0953945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0954039Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0954300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0954441Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0954703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.0954801Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.0954805Z 2025-08-26T20:35:10.0954915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0955136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0955208Z return mod(**inputs) 2025-08-26T20:35:10.0955481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0955585Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0955855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0955941Z layer_outputs = layer_module( 2025-08-26T20:35:10.0956188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0956281Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0956566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.0956666Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.0956939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.0957065Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.0957340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.0957430Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.0957434Z 2025-08-26T20:35:10.0957531Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0957646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0957867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0957949Z return mod(**inputs) 2025-08-26T20:35:10.0958219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0958327Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0958602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0958681Z layer_outputs = layer_module( 2025-08-26T20:35:10.0958941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0959027Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0959313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0959405Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0959759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.0959885Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0960167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0960266Z return self.weight * hidden_states 2025-08-26T20:35:10.0960271Z 2025-08-26T20:35:10.0960385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0960615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0960688Z return mod(**inputs) 2025-08-26T20:35:10.0960954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0961065Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0961347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0961437Z layer_outputs = layer_module( 2025-08-26T20:35:10.0961681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0961773Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0962057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0962146Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0962435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0962530Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0962802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0962888Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0962892Z 2025-08-26T20:35:10.0963413Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0963657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0963730Z return mod(**inputs) 2025-08-26T20:35:10.0964001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0964080Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0964350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0964435Z layer_outputs = layer_module( 2025-08-26T20:35:10.0964670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0964762Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0965027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0965122Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0965387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0965499Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0965763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0965847Z key_states = self.k(current_states) 2025-08-26T20:35:10.0965851Z 2025-08-26T20:35:10.0965970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0966181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0966253Z return mod(**inputs) 2025-08-26T20:35:10.0966531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0966608Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0966882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0966961Z layer_outputs = layer_module( 2025-08-26T20:35:10.0967196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0967287Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0967556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0967651Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0967916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0968034Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0968303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0968447Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0968451Z 2025-08-26T20:35:10.0968569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0968784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0968862Z return mod(**inputs) 2025-08-26T20:35:10.0969140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0969236Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0969512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0969590Z layer_outputs = layer_module( 2025-08-26T20:35:10.0969832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0969917Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0970198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0970288Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0970543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0970638Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0970895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0971070Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0971074Z 2025-08-26T20:35:10.0971185Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0971399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0971477Z return mod(**inputs) 2025-08-26T20:35:10.0971740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0971850Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0972115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0972200Z layer_outputs = layer_module( 2025-08-26T20:35:10.0972443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0972529Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0972800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0972886Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0973152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0973243Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0973503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0973597Z value_states = self.v(current_states) 2025-08-26T20:35:10.0973601Z 2025-08-26T20:35:10.0973711Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0973930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0974002Z return mod(**inputs) 2025-08-26T20:35:10.0974266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0974352Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0974631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0974716Z layer_outputs = layer_module( 2025-08-26T20:35:10.0974951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0975045Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0975301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0975385Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0975666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0975754Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0976014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0976134Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0976138Z 2025-08-26T20:35:10.0976247Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0976467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0976556Z return mod(**inputs) 2025-08-26T20:35:10.0976827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0976906Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0977175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0977253Z layer_outputs = layer_module( 2025-08-26T20:35:10.0977495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0977587Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0977851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0977944Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0978208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0978337Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0978601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.0978719Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.0978723Z 2025-08-26T20:35:10.0978841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0979053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0979125Z return mod(**inputs) 2025-08-26T20:35:10.0979388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0979466Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0979733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0979812Z layer_outputs = layer_module( 2025-08-26T20:35:10.0980055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0980141Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0980399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0980494Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0980747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0980843Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0981128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.0981249Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.0981254Z 2025-08-26T20:35:10.0981374Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0981589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0981669Z return mod(**inputs) 2025-08-26T20:35:10.0981929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0982014Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0982298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0982374Z layer_outputs = layer_module( 2025-08-26T20:35:10.0982606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0982685Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0982939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.0983040Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.0983282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.0983373Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.0983612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.0983700Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.0983704Z 2025-08-26T20:35:10.0983786Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.0983891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0984099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0984167Z return mod(**inputs) 2025-08-26T20:35:10.0984416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0984492Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0984762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0984834Z layer_outputs = layer_module( 2025-08-26T20:35:10.0985057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0985148Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0985387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0985476Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0985717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-26T20:35:10.0985829Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.0986088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.0986174Z return self.weight * hidden_states 2025-08-26T20:35:10.0986178Z 2025-08-26T20:35:10.0986293Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0986505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0986577Z return mod(**inputs) 2025-08-26T20:35:10.0986840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0986918Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0987198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0987276Z layer_outputs = layer_module( 2025-08-26T20:35:10.0987523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0987608Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0987867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0987960Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0988215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0988330Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0988587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.0988672Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.0988678Z 2025-08-26T20:35:10.0988798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0989011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0989106Z return mod(**inputs) 2025-08-26T20:35:10.0989365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0989445Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0989711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0989786Z layer_outputs = layer_module( 2025-08-26T20:35:10.0990035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0990116Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0990382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0990469Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0990725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0990824Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0991099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.0991189Z key_states = self.k(current_states) 2025-08-26T20:35:10.0991193Z 2025-08-26T20:35:10.0991302Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0991517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0991595Z return mod(**inputs) 2025-08-26T20:35:10.0991853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0991939Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0992197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0992281Z layer_outputs = layer_module( 2025-08-26T20:35:10.0992517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0992603Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0992866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0992953Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0993212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0993301Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0993573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.0993722Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.0993728Z 2025-08-26T20:35:10.0993839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0994059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0994130Z return mod(**inputs) 2025-08-26T20:35:10.0994394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0994479Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0994755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0994843Z layer_outputs = layer_module( 2025-08-26T20:35:10.0995083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0995175Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0995436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0995544Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0995819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0995912Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0996344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.0996527Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.0996532Z 2025-08-26T20:35:10.0996648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0996876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0996950Z return mod(**inputs) 2025-08-26T20:35:10.0997226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0997310Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.0997589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.0997716Z layer_outputs = layer_module( 2025-08-26T20:35:10.0997957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.0998052Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.0998319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.0998417Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.0998679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.0998772Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.0999044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.0999132Z value_states = self.v(current_states) 2025-08-26T20:35:10.0999136Z 2025-08-26T20:35:10.0999257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.0999525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.0999605Z return mod(**inputs) 2025-08-26T20:35:10.0999885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.0999967Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1000280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1000361Z layer_outputs = layer_module( 2025-08-26T20:35:10.1000611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1000707Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1000962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1001060Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1001314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1001443Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1001701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1001818Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1001831Z 2025-08-26T20:35:10.1001943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1002155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1002261Z return mod(**inputs) 2025-08-26T20:35:10.1002525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1002613Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1002877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1002954Z layer_outputs = layer_module( 2025-08-26T20:35:10.1003206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1003290Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1003557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1003643Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1003904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1004002Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1004282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1004405Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1004409Z 2025-08-26T20:35:10.1004519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1004739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1004811Z return mod(**inputs) 2025-08-26T20:35:10.1005066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1005153Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1005409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1005494Z layer_outputs = layer_module( 2025-08-26T20:35:10.1005728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1005811Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1006071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1006155Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1006416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1006504Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1006776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1006899Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1006903Z 2025-08-26T20:35:10.1007015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1007234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1007304Z return mod(**inputs) 2025-08-26T20:35:10.1007568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1007647Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1007922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1008009Z layer_outputs = layer_module( 2025-08-26T20:35:10.1008243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1008334Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1008589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1008695Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1008956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1009055Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1009303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1009384Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1009388Z 2025-08-26T20:35:10.1009496Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1009707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1009776Z return mod(**inputs) 2025-08-26T20:35:10.1010031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1010109Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1010366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1010458Z layer_outputs = layer_module( 2025-08-26T20:35:10.1010680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1010766Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1011020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1011115Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1011372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-08-26T20:35:10.1011516Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:35:10.1011519Z 2025-08-26T20:35:10.1011613Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1011725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1011942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1012014Z return mod(**inputs) 2025-08-26T20:35:10.1012274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1012361Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1012622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1012708Z layer_outputs = layer_module( 2025-08-26T20:35:10.1012969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1013061Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1013313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1013415Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1013678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.1013775Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1014025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1014121Z return self.weight * hidden_states 2025-08-26T20:35:10.1014125Z 2025-08-26T20:35:10.1014230Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1014438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1014505Z return mod(**inputs) 2025-08-26T20:35:10.1014754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1014844Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1015093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1015168Z layer_outputs = layer_module( 2025-08-26T20:35:10.1015388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1015474Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1015714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1015812Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1016052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1016171Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1016418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.1016521Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.1016545Z 2025-08-26T20:35:10.1016663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1016876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1016952Z return mod(**inputs) 2025-08-26T20:35:10.1017213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1017291Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1017556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1017633Z layer_outputs = layer_module( 2025-08-26T20:35:10.1017874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1017960Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1018215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1018322Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1018576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1018702Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1018943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.1019024Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.1019035Z 2025-08-26T20:35:10.1019159Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1019362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1019438Z return mod(**inputs) 2025-08-26T20:35:10.1019680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1019764Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1020007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1020079Z layer_outputs = layer_module( 2025-08-26T20:35:10.1020331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1020417Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1020679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1020777Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1021028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1021179Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1021435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.1021544Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.1021547Z 2025-08-26T20:35:10.1021649Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1021856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1021921Z return mod(**inputs) 2025-08-26T20:35:10.1022165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1022248Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1022491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1022570Z layer_outputs = layer_module( 2025-08-26T20:35:10.1022793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1022890Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1023137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1023225Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1023471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1023586Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1023824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.1023912Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.1023917Z 2025-08-26T20:35:10.1023999Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1024110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1024310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1024382Z return mod(**inputs) 2025-08-26T20:35:10.1024627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1024701Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1024951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1025024Z layer_outputs = layer_module( 2025-08-26T20:35:10.1025266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1025347Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1025591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1025684Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1025928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.1026042Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1026303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1026382Z return self.weight * hidden_states 2025-08-26T20:35:10.1026392Z 2025-08-26T20:35:10.1026503Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1026731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1026810Z return mod(**inputs) 2025-08-26T20:35:10.1027082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1027198Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1027461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1027538Z layer_outputs = layer_module( 2025-08-26T20:35:10.1027783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1027868Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1028131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1028218Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1028484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1028583Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1028838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1028950Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1028954Z 2025-08-26T20:35:10.1029063Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1029293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1029364Z return mod(**inputs) 2025-08-26T20:35:10.1029641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1029725Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1029972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1030051Z layer_outputs = layer_module( 2025-08-26T20:35:10.1030283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1030369Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1030632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1030717Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1030983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1031073Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1031329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1031417Z key_states = self.k(current_states) 2025-08-26T20:35:10.1031421Z 2025-08-26T20:35:10.1031547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1031767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1031838Z return mod(**inputs) 2025-08-26T20:35:10.1032113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1032194Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1032467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1032554Z layer_outputs = layer_module( 2025-08-26T20:35:10.1032821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1032913Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1033185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1033272Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1033540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1033662Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1033926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1034064Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1034068Z 2025-08-26T20:35:10.1034184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1034393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1034463Z return mod(**inputs) 2025-08-26T20:35:10.1034727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1034805Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1035069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1035149Z layer_outputs = layer_module( 2025-08-26T20:35:10.1035381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1035491Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1035746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1035839Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1036104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1036194Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1036469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1036642Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1036647Z 2025-08-26T20:35:10.1036770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1036991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1037069Z return mod(**inputs) 2025-08-26T20:35:10.1037336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1037415Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1037690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1037770Z layer_outputs = layer_module( 2025-08-26T20:35:10.1038036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1038124Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1038382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1038480Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1038738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1038835Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1039108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1039209Z value_states = self.v(current_states) 2025-08-26T20:35:10.1039221Z 2025-08-26T20:35:10.1039333Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1039615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1039703Z return mod(**inputs) 2025-08-26T20:35:10.1039972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1040087Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1040357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1040439Z layer_outputs = layer_module( 2025-08-26T20:35:10.1040695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1040781Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1041057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1041147Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1041420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1041517Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1041772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1041902Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1041926Z 2025-08-26T20:35:10.1042038Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1042259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1042330Z return mod(**inputs) 2025-08-26T20:35:10.1042590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1042679Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1042938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1043024Z layer_outputs = layer_module( 2025-08-26T20:35:10.1043259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1043346Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1043610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1043697Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1043957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1044044Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1044300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1044422Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1044426Z 2025-08-26T20:35:10.1044558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1044779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1044851Z return mod(**inputs) 2025-08-26T20:35:10.1045118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1045198Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1045453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1045537Z layer_outputs = layer_module( 2025-08-26T20:35:10.1045790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1045883Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1046137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1046225Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1046486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1046595Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1046858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1046967Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1046971Z 2025-08-26T20:35:10.1047084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1047284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1047350Z return mod(**inputs) 2025-08-26T20:35:10.1047612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1047692Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1047955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1048034Z layer_outputs = layer_module( 2025-08-26T20:35:10.1048268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1048385Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1048639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1048731Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1048987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1049075Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1049337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1049419Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1049422Z 2025-08-26T20:35:10.1049517Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1049627Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1049846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1049927Z return mod(**inputs) 2025-08-26T20:35:10.1050170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1050252Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1050492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1050571Z layer_outputs = layer_module( 2025-08-26T20:35:10.1050791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1050887Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1051138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1051221Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1051479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-26T20:35:10.1051594Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1051847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1051969Z return self.weight * hidden_states 2025-08-26T20:35:10.1051974Z 2025-08-26T20:35:10.1052080Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1052287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1052356Z return mod(**inputs) 2025-08-26T20:35:10.1052608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1052701Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1052945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1053026Z layer_outputs = layer_module( 2025-08-26T20:35:10.1053250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1053337Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1053579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1053662Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1053910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1053995Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1054241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1054323Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1054344Z 2025-08-26T20:35:10.1054454Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1054653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1054719Z return mod(**inputs) 2025-08-26T20:35:10.1054973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1055048Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1055299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1055371Z layer_outputs = layer_module( 2025-08-26T20:35:10.1055591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1055679Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1055918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1056009Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1056249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1056333Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1056584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1056662Z key_states = self.k(current_states) 2025-08-26T20:35:10.1056666Z 2025-08-26T20:35:10.1056776Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1056990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1057066Z return mod(**inputs) 2025-08-26T20:35:10.1057314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1057392Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1057647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1057719Z layer_outputs = layer_module( 2025-08-26T20:35:10.1057965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1058045Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1058285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1058375Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1058622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1058736Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1058991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1059138Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1059142Z 2025-08-26T20:35:10.1059251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1059465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1059544Z return mod(**inputs) 2025-08-26T20:35:10.1059803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1059886Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1060144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1060221Z layer_outputs = layer_module( 2025-08-26T20:35:10.1060465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1060572Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1060833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1060916Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1061170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1061266Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1061519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1061692Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1061697Z 2025-08-26T20:35:10.1061805Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1062025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1062097Z return mod(**inputs) 2025-08-26T20:35:10.1062351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1062436Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1062696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1062779Z layer_outputs = layer_module( 2025-08-26T20:35:10.1063015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1063116Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1063380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1063468Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1063731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1063821Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1064081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1064165Z value_states = self.v(current_states) 2025-08-26T20:35:10.1064168Z 2025-08-26T20:35:10.1064295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1064520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1064589Z return mod(**inputs) 2025-08-26T20:35:10.1064855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1064935Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1065213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1065299Z layer_outputs = layer_module( 2025-08-26T20:35:10.1065535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1065625Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1065883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1065968Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1066229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1066317Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1066581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1066696Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1066700Z 2025-08-26T20:35:10.1066816Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1067052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1067121Z return mod(**inputs) 2025-08-26T20:35:10.1067390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1067469Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1067736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1067812Z layer_outputs = layer_module( 2025-08-26T20:35:10.1068049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1068140Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1068397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1068490Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1068745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1068833Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1069096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1069211Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1069214Z 2025-08-26T20:35:10.1069330Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1069568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1069648Z return mod(**inputs) 2025-08-26T20:35:10.1069910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1069990Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1070259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1070335Z layer_outputs = layer_module( 2025-08-26T20:35:10.1070578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1070679Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1070934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1071027Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1071280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1071377Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1071656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1071781Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1071785Z 2025-08-26T20:35:10.1071893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1072105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1072183Z return mod(**inputs) 2025-08-26T20:35:10.1072441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1072529Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1072790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1072867Z layer_outputs = layer_module( 2025-08-26T20:35:10.1073111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1073217Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1073478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1073563Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1073816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1073911Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1074171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1074267Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1074271Z 2025-08-26T20:35:10.1074360Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1074479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1074695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1074768Z return mod(**inputs) 2025-08-26T20:35:10.1075040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1075131Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1075395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1075472Z layer_outputs = layer_module( 2025-08-26T20:35:10.1075707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1075821Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1076078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1076187Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1076444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.1076559Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1076817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1076899Z return self.weight * hidden_states 2025-08-26T20:35:10.1076902Z 2025-08-26T20:35:10.1077036Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1077256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1077337Z return mod(**inputs) 2025-08-26T20:35:10.1077608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1077692Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1078002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1078084Z layer_outputs = layer_module( 2025-08-26T20:35:10.1078336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1078422Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1078690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1078799Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1079062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1079200Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1079531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.1079663Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.1079668Z 2025-08-26T20:35:10.1079806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1080025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1080107Z return mod(**inputs) 2025-08-26T20:35:10.1080377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1080466Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1080743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1080822Z layer_outputs = layer_module( 2025-08-26T20:35:10.1081066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1081152Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1081423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1081525Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1081791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1081919Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1082183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.1082278Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.1082282Z 2025-08-26T20:35:10.1082395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1082640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1082714Z return mod(**inputs) 2025-08-26T20:35:10.1082983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1083072Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1083335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1083420Z layer_outputs = layer_module( 2025-08-26T20:35:10.1083681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1083769Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1084040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1084140Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1084407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1084552Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1084823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.1084922Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.1084926Z 2025-08-26T20:35:10.1085040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1085266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1085339Z return mod(**inputs) 2025-08-26T20:35:10.1085615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1085695Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1085970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1086058Z layer_outputs = layer_module( 2025-08-26T20:35:10.1086299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1086413Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1086687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1086793Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1087113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1087239Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1087510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.1087599Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.1087603Z 2025-08-26T20:35:10.1087724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1087947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1088022Z return mod(**inputs) 2025-08-26T20:35:10.1088351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1088433Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1088711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1088792Z layer_outputs = layer_module( 2025-08-26T20:35:10.1089032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1089146Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1089424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1089531Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1089805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-08-26T20:35:10.1089960Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-08-26T20:35:10.1089964Z 2025-08-26T20:35:10.1090054Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1090166Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1090426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1090501Z return mod(**inputs) 2025-08-26T20:35:10.1090794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1090877Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1091153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1091259Z layer_outputs = layer_module( 2025-08-26T20:35:10.1091502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1091599Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1091874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1091970Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1092249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.1092369Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1092648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1092734Z return self.weight * hidden_states 2025-08-26T20:35:10.1092738Z 2025-08-26T20:35:10.1092860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1093080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1093176Z return mod(**inputs) 2025-08-26T20:35:10.1093454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1093536Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1093822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1093902Z layer_outputs = layer_module( 2025-08-26T20:35:10.1094148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1094245Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1094530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1094629Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1094904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1095006Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1095315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1095402Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1095408Z 2025-08-26T20:35:10.1095532Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1095763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1095844Z return mod(**inputs) 2025-08-26T20:35:10.1096138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1096352Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1096640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1096723Z layer_outputs = layer_module( 2025-08-26T20:35:10.1096980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1097068Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1097377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1097478Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1097741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1097853Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1098108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1098225Z key_states = self.k(current_states) 2025-08-26T20:35:10.1098229Z 2025-08-26T20:35:10.1098339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1098554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1098632Z return mod(**inputs) 2025-08-26T20:35:10.1098892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1098978Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1099236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1099312Z layer_outputs = layer_module( 2025-08-26T20:35:10.1099561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1099646Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1099911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1100033Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1100295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1100384Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1100640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1100788Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1100792Z 2025-08-26T20:35:10.1100903Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1101128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1101200Z return mod(**inputs) 2025-08-26T20:35:10.1101468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1101558Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1101823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1101910Z layer_outputs = layer_module( 2025-08-26T20:35:10.1102153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1102242Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1102518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1102607Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1102905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1103012Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1103275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1103445Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1103449Z 2025-08-26T20:35:10.1103559Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1103774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1103861Z return mod(**inputs) 2025-08-26T20:35:10.1104130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1104210Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1104467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1104551Z layer_outputs = layer_module( 2025-08-26T20:35:10.1104805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1104899Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1105154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1105250Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1105505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1105594Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1105858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1105944Z value_states = self.v(current_states) 2025-08-26T20:35:10.1105947Z 2025-08-26T20:35:10.1106065Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1106281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1106351Z return mod(**inputs) 2025-08-26T20:35:10.1106635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1106714Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1106978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1107056Z layer_outputs = layer_module( 2025-08-26T20:35:10.1107291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1107381Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1107636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1107728Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1107982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1108079Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1108335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1108453Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1108457Z 2025-08-26T20:35:10.1108577Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1108788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1108866Z return mod(**inputs) 2025-08-26T20:35:10.1109144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1109223Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1109491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1109569Z layer_outputs = layer_module( 2025-08-26T20:35:10.1109815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1109899Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1110162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1110268Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1110529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1110629Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1110887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1111012Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1111033Z 2025-08-26T20:35:10.1111146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1111362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1111441Z return mod(**inputs) 2025-08-26T20:35:10.1111702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1111790Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1112048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1112125Z layer_outputs = layer_module( 2025-08-26T20:35:10.1112371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1112459Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1112731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1112820Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1113111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1113202Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1113463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1113591Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1113595Z 2025-08-26T20:35:10.1113708Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1113932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1114004Z return mod(**inputs) 2025-08-26T20:35:10.1114268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1114358Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1114623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1114707Z layer_outputs = layer_module( 2025-08-26T20:35:10.1114943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1115038Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1115299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1115388Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1115682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1115773Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1116046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1116132Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1116137Z 2025-08-26T20:35:10.1116225Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1116347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1116564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1116664Z return mod(**inputs) 2025-08-26T20:35:10.1116933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1117014Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1117288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1117367Z layer_outputs = layer_module( 2025-08-26T20:35:10.1117634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1117722Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1117990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1118079Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1118342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-26T20:35:10.1118470Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1118734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1118829Z return self.weight * hidden_states 2025-08-26T20:35:10.1118833Z 2025-08-26T20:35:10.1118946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1119167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1119247Z return mod(**inputs) 2025-08-26T20:35:10.1119615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1119710Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1119976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1120057Z layer_outputs = layer_module( 2025-08-26T20:35:10.1120308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1120395Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1120670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1120762Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1121035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1121131Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1121396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1121490Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1121495Z 2025-08-26T20:35:10.1121613Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1121838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1121912Z return mod(**inputs) 2025-08-26T20:35:10.1122206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1122301Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1122564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1122649Z layer_outputs = layer_module( 2025-08-26T20:35:10.1122892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1122987Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1123251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1123373Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1123642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1123734Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1124002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1124087Z key_states = self.k(current_states) 2025-08-26T20:35:10.1124121Z 2025-08-26T20:35:10.1124236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1124463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1124538Z return mod(**inputs) 2025-08-26T20:35:10.1124812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1124892Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1125158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1125243Z layer_outputs = layer_module( 2025-08-26T20:35:10.1125490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1125584Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1125847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1125945Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1126221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1126312Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1126584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1126728Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1126732Z 2025-08-26T20:35:10.1126855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1127073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1127146Z return mod(**inputs) 2025-08-26T20:35:10.1127429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1127510Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1127785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1127865Z layer_outputs = layer_module( 2025-08-26T20:35:10.1128110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1128202Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1128467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1128560Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1128840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1128938Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1129199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1129367Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1129371Z 2025-08-26T20:35:10.1129487Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1129711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1129787Z return mod(**inputs) 2025-08-26T20:35:10.1130072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1130150Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1130416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1130493Z layer_outputs = layer_module( 2025-08-26T20:35:10.1130732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1130833Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1131101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1131190Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1131457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1131560Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1131819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1131936Z value_states = self.v(current_states) 2025-08-26T20:35:10.1131941Z 2025-08-26T20:35:10.1132052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1132268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1132353Z return mod(**inputs) 2025-08-26T20:35:10.1132628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1132734Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1133007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1133084Z layer_outputs = layer_module( 2025-08-26T20:35:10.1133332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1133415Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1133682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1133768Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1134044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1134134Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1134400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1134525Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1134529Z 2025-08-26T20:35:10.1134640Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1134862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1134932Z return mod(**inputs) 2025-08-26T20:35:10.1135226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1135316Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1135590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1135680Z layer_outputs = layer_module( 2025-08-26T20:35:10.1135918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1136013Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1136282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1136372Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1136655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1136744Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1137017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1137133Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1137155Z 2025-08-26T20:35:10.1137267Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1137507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1137579Z return mod(**inputs) 2025-08-26T20:35:10.1137843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1137922Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1138178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1138263Z layer_outputs = layer_module( 2025-08-26T20:35:10.1138500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1138593Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1138856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1138952Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1139233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1139322Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1139594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1139715Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1139719Z 2025-08-26T20:35:10.1139850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1140064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1140136Z return mod(**inputs) 2025-08-26T20:35:10.1140402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1140482Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1140746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1140823Z layer_outputs = layer_module( 2025-08-26T20:35:10.1141065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1141147Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1141405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1141498Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1141772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1141871Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1142124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1142208Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1142213Z 2025-08-26T20:35:10.1142310Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1142420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1142638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1142707Z return mod(**inputs) 2025-08-26T20:35:10.1142982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1143072Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1143334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1143421Z layer_outputs = layer_module( 2025-08-26T20:35:10.1143660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1143768Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1144024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1144120Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1144383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.1144487Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1144747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1144830Z return self.weight * hidden_states 2025-08-26T20:35:10.1144837Z 2025-08-26T20:35:10.1144945Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1145164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1145234Z return mod(**inputs) 2025-08-26T20:35:10.1145501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1145596Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1145851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1145934Z layer_outputs = layer_module( 2025-08-26T20:35:10.1146169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1146261Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1146516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1146618Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1146869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1146999Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1147264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.1147370Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.1147375Z 2025-08-26T20:35:10.1147490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1147702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1147773Z return mod(**inputs) 2025-08-26T20:35:10.1148058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1148140Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1148407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1148488Z layer_outputs = layer_module( 2025-08-26T20:35:10.1148736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1148822Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1149078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1149208Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1149465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1149598Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1149856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.1149943Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.1149969Z 2025-08-26T20:35:10.1150090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1150308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1150389Z return mod(**inputs) 2025-08-26T20:35:10.1150654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1150743Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1151010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1151090Z layer_outputs = layer_module( 2025-08-26T20:35:10.1151345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1151432Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1151701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1151803Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1152084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1152219Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1152480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.1152594Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.1152598Z 2025-08-26T20:35:10.1152708Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1152922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1153007Z return mod(**inputs) 2025-08-26T20:35:10.1153262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1153349Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1153606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1153690Z layer_outputs = layer_module( 2025-08-26T20:35:10.1153924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1154011Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1154274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1154369Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1154649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1154774Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1155030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.1155126Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.1155130Z 2025-08-26T20:35:10.1155217Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1155335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1155545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1155641Z return mod(**inputs) 2025-08-26T20:35:10.1155900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1155978Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1156245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1156321Z layer_outputs = layer_module( 2025-08-26T20:35:10.1156581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1156668Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1156922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1157017Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1157273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.1157395Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1157647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1157731Z return self.weight * hidden_states 2025-08-26T20:35:10.1157743Z 2025-08-26T20:35:10.1157853Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1158064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1158143Z return mod(**inputs) 2025-08-26T20:35:10.1158424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1158508Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1158765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1158841Z layer_outputs = layer_module( 2025-08-26T20:35:10.1159082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1159167Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1159432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1159590Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1159861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1159965Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1160229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1160323Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1160327Z 2025-08-26T20:35:10.1160442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1160671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1160746Z return mod(**inputs) 2025-08-26T20:35:10.1161038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1161132Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1161402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1161490Z layer_outputs = layer_module( 2025-08-26T20:35:10.1161735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1161823Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1162102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1162210Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1162475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1162566Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1162822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1162913Z key_states = self.k(current_states) 2025-08-26T20:35:10.1162939Z 2025-08-26T20:35:10.1163049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1163269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1163339Z return mod(**inputs) 2025-08-26T20:35:10.1163602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1163681Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1163936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1164022Z layer_outputs = layer_module( 2025-08-26T20:35:10.1164258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1164352Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1164603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1164691Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1164972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1165060Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1165324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1165464Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1165468Z 2025-08-26T20:35:10.1165578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1165802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1165874Z return mod(**inputs) 2025-08-26T20:35:10.1166139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1166219Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1166484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1166563Z layer_outputs = layer_module( 2025-08-26T20:35:10.1166800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1166893Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1167151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1167244Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1167514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1167604Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1167867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1168036Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1168041Z 2025-08-26T20:35:10.1168158Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1168371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1168448Z return mod(**inputs) 2025-08-26T20:35:10.1168970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1169053Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1169329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1169407Z layer_outputs = layer_module( 2025-08-26T20:35:10.1169652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1169754Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1170011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1170107Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1170368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1170467Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1170719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1170802Z value_states = self.v(current_states) 2025-08-26T20:35:10.1170812Z 2025-08-26T20:35:10.1170925Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1171135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1171216Z return mod(**inputs) 2025-08-26T20:35:10.1171485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1171593Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1171862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1171938Z layer_outputs = layer_module( 2025-08-26T20:35:10.1172185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1172270Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1172544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1172629Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1172896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1172994Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1173253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1173376Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1173380Z 2025-08-26T20:35:10.1173489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1173723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1173794Z return mod(**inputs) 2025-08-26T20:35:10.1174063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1174176Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1174442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1174525Z layer_outputs = layer_module( 2025-08-26T20:35:10.1174760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1174844Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1175117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1175202Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1175484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1175572Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1175838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1175961Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1175982Z 2025-08-26T20:35:10.1176094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1176319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1176391Z return mod(**inputs) 2025-08-26T20:35:10.1176665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1176743Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1177014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1177099Z layer_outputs = layer_module( 2025-08-26T20:35:10.1177335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1177427Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1177682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1177769Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1178039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1178145Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1178412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1178529Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1178533Z 2025-08-26T20:35:10.1178649Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1178861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1178934Z return mod(**inputs) 2025-08-26T20:35:10.1179202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1179283Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1179558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1179639Z layer_outputs = layer_module( 2025-08-26T20:35:10.1179892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1179984Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1180241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1180336Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1180613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1180705Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1180977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1181063Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1181068Z 2025-08-26T20:35:10.1181189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1181410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1181491Z return mod(**inputs) 2025-08-26T20:35:10.1181802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1181885Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1182160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1182240Z layer_outputs = layer_module( 2025-08-26T20:35:10.1182492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1182598Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1182860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1182960Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1183222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-26T20:35:10.1183375Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:35:10.1183381Z 2025-08-26T20:35:10.1183470Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1183590Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1183809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1183882Z return mod(**inputs) 2025-08-26T20:35:10.1184153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1184233Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1184503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1184601Z layer_outputs = layer_module( 2025-08-26T20:35:10.1184845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1184942Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1185211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1185307Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1185574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-26T20:35:10.1185692Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1185964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1186051Z return self.weight * hidden_states 2025-08-26T20:35:10.1186058Z 2025-08-26T20:35:10.1186179Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1186399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1186478Z return mod(**inputs) 2025-08-26T20:35:10.1186744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1186823Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1187095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1187190Z layer_outputs = layer_module( 2025-08-26T20:35:10.1187440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1187526Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1187788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1187886Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1188144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1188244Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1188519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1188607Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1188619Z 2025-08-26T20:35:10.1188734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1188954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1189052Z return mod(**inputs) 2025-08-26T20:35:10.1189323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1189412Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1189682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1189759Z layer_outputs = layer_module( 2025-08-26T20:35:10.1190016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1190102Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1190372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1190464Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1190729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1190831Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1191097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1191207Z key_states = self.k(current_states) 2025-08-26T20:35:10.1191211Z 2025-08-26T20:35:10.1191323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1191548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1191619Z return mod(**inputs) 2025-08-26T20:35:10.1191884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1191971Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1192238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1192323Z layer_outputs = layer_module( 2025-08-26T20:35:10.1192565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1192651Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1192918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1193004Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1193269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1193358Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1193618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1193782Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1193786Z 2025-08-26T20:35:10.1193901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1194127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1194200Z return mod(**inputs) 2025-08-26T20:35:10.1194476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1194557Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1194841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1194930Z layer_outputs = layer_module( 2025-08-26T20:35:10.1195173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1195267Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1195528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1195635Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1195907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1195999Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1196406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1196585Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1196593Z 2025-08-26T20:35:10.1196717Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1196934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1197007Z return mod(**inputs) 2025-08-26T20:35:10.1197282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1197363Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1197639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1197775Z layer_outputs = layer_module( 2025-08-26T20:35:10.1198022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1198114Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1198380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1198475Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1198738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1198831Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1199106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1199196Z value_states = self.v(current_states) 2025-08-26T20:35:10.1199202Z 2025-08-26T20:35:10.1199325Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1199595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1199680Z return mod(**inputs) 2025-08-26T20:35:10.1199952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1200031Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1200306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1200384Z layer_outputs = layer_module( 2025-08-26T20:35:10.1200668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1200757Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1201022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1201121Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1201383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1201483Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1201770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1201894Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1201906Z 2025-08-26T20:35:10.1202019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1202239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1202319Z return mod(**inputs) 2025-08-26T20:35:10.1202616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1202706Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1202972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1203048Z layer_outputs = layer_module( 2025-08-26T20:35:10.1203310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1203393Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1203656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1203743Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1203995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1204096Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1204350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1204491Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1204495Z 2025-08-26T20:35:10.1204606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1204823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1204896Z return mod(**inputs) 2025-08-26T20:35:10.1205153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1205237Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1205496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1205579Z layer_outputs = layer_module( 2025-08-26T20:35:10.1205816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1205899Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1206158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1206243Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1206504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1206592Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1206845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1206983Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1206988Z 2025-08-26T20:35:10.1207100Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1207321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1207392Z return mod(**inputs) 2025-08-26T20:35:10.1207653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1207731Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1207988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1208090Z layer_outputs = layer_module( 2025-08-26T20:35:10.1208324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1208415Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1208671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1208756Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1209035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1209125Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1209387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1209470Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1209474Z 2025-08-26T20:35:10.1209570Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1209680Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1209893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1209969Z return mod(**inputs) 2025-08-26T20:35:10.1210225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1210310Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1210571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1210668Z layer_outputs = layer_module( 2025-08-26T20:35:10.1210910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1210992Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1211256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1211355Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1211609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.1211722Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1211978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1212071Z return self.weight * hidden_states 2025-08-26T20:35:10.1212075Z 2025-08-26T20:35:10.1212185Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1212403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1212473Z return mod(**inputs) 2025-08-26T20:35:10.1212734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1212819Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1213077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1213161Z layer_outputs = layer_module( 2025-08-26T20:35:10.1213413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1213498Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1213760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1213858Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1214121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1214248Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1214519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.1214631Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.1214635Z 2025-08-26T20:35:10.1214745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1214974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1215045Z return mod(**inputs) 2025-08-26T20:35:10.1215328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1215408Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1215663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1215747Z layer_outputs = layer_module( 2025-08-26T20:35:10.1215982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1216074Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1216323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1216422Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1216684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1216812Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1217074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.1217177Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.1217180Z 2025-08-26T20:35:10.1217295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1217507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1217577Z return mod(**inputs) 2025-08-26T20:35:10.1217843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1217923Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1218191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1218267Z layer_outputs = layer_module( 2025-08-26T20:35:10.1218503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1218598Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1218852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1218956Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1219214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1219344Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1219639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.1219734Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.1219738Z 2025-08-26T20:35:10.1219855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1220067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1220150Z return mod(**inputs) 2025-08-26T20:35:10.1220408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1220490Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1220774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1220853Z layer_outputs = layer_module( 2025-08-26T20:35:10.1221099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1221185Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1221442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1221574Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1221824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1221957Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1222211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.1222303Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.1222309Z 2025-08-26T20:35:10.1222395Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1222505Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1222730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1222804Z return mod(**inputs) 2025-08-26T20:35:10.1223073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1223154Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1223420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1223525Z layer_outputs = layer_module( 2025-08-26T20:35:10.1223761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1223852Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1224110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1224198Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1224465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-26T20:35:10.1224578Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1224840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1224925Z return self.weight * hidden_states 2025-08-26T20:35:10.1224930Z 2025-08-26T20:35:10.1225045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1225262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1225334Z return mod(**inputs) 2025-08-26T20:35:10.1225613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1225695Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1225968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1226065Z layer_outputs = layer_module( 2025-08-26T20:35:10.1226309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1226406Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1226667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1226765Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1227030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1227130Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1227415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1227502Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1227507Z 2025-08-26T20:35:10.1227628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1227847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1227946Z return mod(**inputs) 2025-08-26T20:35:10.1228212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1228294Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1228571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1228650Z layer_outputs = layer_module( 2025-08-26T20:35:10.1228901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1228987Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1229250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1229349Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1229607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1229709Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1229968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1230078Z key_states = self.k(current_states) 2025-08-26T20:35:10.1230082Z 2025-08-26T20:35:10.1230196Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1230413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1230494Z return mod(**inputs) 2025-08-26T20:35:10.1230758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1230846Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1231109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1231188Z layer_outputs = layer_module( 2025-08-26T20:35:10.1231438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1231525Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1231795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1231883Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1232145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1232242Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1232504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1232672Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1232677Z 2025-08-26T20:35:10.1232791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1233017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1233092Z return mod(**inputs) 2025-08-26T20:35:10.1233355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1233443Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1233725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1233813Z layer_outputs = layer_module( 2025-08-26T20:35:10.1234056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1234144Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1234415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1234521Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1234791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1234883Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1235152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1235322Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1235328Z 2025-08-26T20:35:10.1235442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1235669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1235740Z return mod(**inputs) 2025-08-26T20:35:10.1236018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1236099Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1236365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1236470Z layer_outputs = layer_module( 2025-08-26T20:35:10.1236719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1236810Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1237066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1237159Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1237413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1237500Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1237762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1237845Z value_states = self.v(current_states) 2025-08-26T20:35:10.1237849Z 2025-08-26T20:35:10.1237967Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1238179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1238249Z return mod(**inputs) 2025-08-26T20:35:10.1238514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1238596Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1238865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1238942Z layer_outputs = layer_module( 2025-08-26T20:35:10.1239202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1239298Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1239645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1239749Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1240022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1240122Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1240420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1240542Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1240547Z 2025-08-26T20:35:10.1240671Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1240890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1240973Z return mod(**inputs) 2025-08-26T20:35:10.1241271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1241355Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1241629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1241717Z layer_outputs = layer_module( 2025-08-26T20:35:10.1241962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1242049Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1242301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1242394Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1242648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1242745Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1242996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1243139Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1243143Z 2025-08-26T20:35:10.1243254Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1243468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1243548Z return mod(**inputs) 2025-08-26T20:35:10.1243805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1243891Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1244148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1244225Z layer_outputs = layer_module( 2025-08-26T20:35:10.1244472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1244557Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1244818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1244903Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1245165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1245253Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1245508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1245649Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1245653Z 2025-08-26T20:35:10.1245766Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1245990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1246061Z return mod(**inputs) 2025-08-26T20:35:10.1246326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1246414Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1246674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1246781Z layer_outputs = layer_module( 2025-08-26T20:35:10.1247018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1247102Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1247368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-26T20:35:10.1247453Z self_attention_outputs = self.layer[0]( 2025-08-26T20:35:10.1247731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-26T20:35:10.1247820Z attention_output = self.SelfAttention( 2025-08-26T20:35:10.1248081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1248163Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1248167Z 2025-08-26T20:35:10.1248253Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1248374Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1248584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1248665Z return mod(**inputs) 2025-08-26T20:35:10.1248921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1249002Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1249271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1249368Z layer_outputs = layer_module( 2025-08-26T20:35:10.1249607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1249690Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1249942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1250036Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1250289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-26T20:35:10.1250411Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1250666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1250758Z return self.weight * hidden_states 2025-08-26T20:35:10.1250762Z 2025-08-26T20:35:10.1250872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1251085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1251163Z return mod(**inputs) 2025-08-26T20:35:10.1251417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1251500Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1251756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1251833Z layer_outputs = layer_module( 2025-08-26T20:35:10.1252095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1252179Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1252441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1252527Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1252789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1252875Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1253125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-26T20:35:10.1253214Z query_states = self.q(hidden_states) 2025-08-26T20:35:10.1253218Z 2025-08-26T20:35:10.1253323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1253529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1253595Z return mod(**inputs) 2025-08-26T20:35:10.1253841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1253942Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1254189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1254270Z layer_outputs = layer_module( 2025-08-26T20:35:10.1254491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1254572Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1254820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1254901Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1255152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1255236Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1255488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-26T20:35:10.1255587Z key_states = self.k(current_states) 2025-08-26T20:35:10.1255591Z 2025-08-26T20:35:10.1255694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1255900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1255968Z return mod(**inputs) 2025-08-26T20:35:10.1256242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1256320Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1256624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1256704Z layer_outputs = layer_module( 2025-08-26T20:35:10.1256923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1257011Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1257251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1257338Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1257578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1257662Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1257912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-26T20:35:10.1258070Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:35:10.1258074Z 2025-08-26T20:35:10.1258195Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1258411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1258487Z return mod(**inputs) 2025-08-26T20:35:10.1258774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1258852Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1259123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1259217Z layer_outputs = layer_module( 2025-08-26T20:35:10.1259452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1259544Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1259809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1259902Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1260193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1260292Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1260557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-26T20:35:10.1260722Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:35:10.1260726Z 2025-08-26T20:35:10.1260847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1261059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1261137Z return mod(**inputs) 2025-08-26T20:35:10.1261404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1261481Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1261756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1261834Z layer_outputs = layer_module( 2025-08-26T20:35:10.1262096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1262179Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1262450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1262537Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1262787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1262884Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1263148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-26T20:35:10.1263239Z value_states = self.v(current_states) 2025-08-26T20:35:10.1263244Z 2025-08-26T20:35:10.1263352Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1263564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1263641Z return mod(**inputs) 2025-08-26T20:35:10.1263895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1263978Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1264235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1264311Z layer_outputs = layer_module( 2025-08-26T20:35:10.1264570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1264654Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1264912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1265000Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1265260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1265347Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1265597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1265739Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1265743Z 2025-08-26T20:35:10.1265854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1266075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1266146Z return mod(**inputs) 2025-08-26T20:35:10.1266404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1266507Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1266766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1266848Z layer_outputs = layer_module( 2025-08-26T20:35:10.1267086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1267176Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1267429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1267515Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1267774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1267861Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1268121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-26T20:35:10.1268257Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:35:10.1268261Z 2025-08-26T20:35:10.1268369Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1268595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1268664Z return mod(**inputs) 2025-08-26T20:35:10.1268933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1269011Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1269277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1269359Z layer_outputs = layer_module( 2025-08-26T20:35:10.1269597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1269687Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1269945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1270037Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1270295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1270492Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1270765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-26T20:35:10.1270880Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:35:10.1270914Z 2025-08-26T20:35:10.1271034Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1271248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1271323Z return mod(**inputs) 2025-08-26T20:35:10.1271589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1271670Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1271936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1272013Z layer_outputs = layer_module( 2025-08-26T20:35:10.1272274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1272360Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1272614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1272709Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1272963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-26T20:35:10.1273076Z attention_output = self.EncDecAttention( 2025-08-26T20:35:10.1273337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-26T20:35:10.1273420Z attn_output = self.o(attn_output) 2025-08-26T20:35:10.1273424Z 2025-08-26T20:35:10.1273544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1273759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1273838Z return mod(**inputs) 2025-08-26T20:35:10.1274100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1274180Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1274452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1274531Z layer_outputs = layer_module( 2025-08-26T20:35:10.1274774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1274875Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1275136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-26T20:35:10.1275220Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:35:10.1275476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-08-26T20:35:10.1275625Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:35:10.1275629Z 2025-08-26T20:35:10.1275717Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1275835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1276053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1276125Z return mod(**inputs) 2025-08-26T20:35:10.1276392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1276470Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1276736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1276813Z layer_outputs = layer_module( 2025-08-26T20:35:10.1277048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1277137Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1277416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1277522Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1277780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-26T20:35:10.1277892Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:35:10.1278143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1278227Z return self.weight * hidden_states 2025-08-26T20:35:10.1278231Z 2025-08-26T20:35:10.1278349Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1278575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1278654Z return mod(**inputs) 2025-08-26T20:35:10.1278917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1278997Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1279270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1279368Z layer_outputs = layer_module( 2025-08-26T20:35:10.1279687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1279777Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1280050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1280153Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1280415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1280555Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1280819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-26T20:35:10.1280935Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-26T20:35:10.1280952Z 2025-08-26T20:35:10.1281062Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1281293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1281371Z return mod(**inputs) 2025-08-26T20:35:10.1281627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1281713Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1281975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1282052Z layer_outputs = layer_module( 2025-08-26T20:35:10.1282299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1282383Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1282643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1282742Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1283009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1283135Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1283394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-26T20:35:10.1283489Z hidden_linear = self.wi_1(hidden_states) 2025-08-26T20:35:10.1283493Z 2025-08-26T20:35:10.1283604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1283874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1283948Z return mod(**inputs) 2025-08-26T20:35:10.1284215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1284303Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1284570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1284653Z layer_outputs = layer_module( 2025-08-26T20:35:10.1284895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1285004Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1285268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1285365Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1285640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1285764Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1286041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-26T20:35:10.1286138Z hidden_states = hidden_gelu * hidden_linear 2025-08-26T20:35:10.1286142Z 2025-08-26T20:35:10.1286253Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1286472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1286542Z return mod(**inputs) 2025-08-26T20:35:10.1286818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1286897Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1287163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-26T20:35:10.1287249Z layer_outputs = layer_module( 2025-08-26T20:35:10.1287490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:10.1287588Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:10.1287874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-26T20:35:10.1287981Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:35:10.1288246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-26T20:35:10.1288375Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:35:10.1288647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-26T20:35:10.1288736Z hidden_states = self.wo(hidden_states) 2025-08-26T20:35:10.1288742Z 2025-08-26T20:35:10.1288839Z cudagraph partition due to non gpu ops 2025-08-26T20:35:10.1288953Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1289172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1289254Z return mod(**inputs) 2025-08-26T20:35:10.1289520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-26T20:35:10.1289607Z decoder_outputs = self.decoder( 2025-08-26T20:35:10.1289874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-08-26T20:35:10.1290000Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-26T20:35:10.1290265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-26T20:35:10.1290371Z return self.weight * hidden_states 2025-08-26T20:35:10.1290376Z 2025-08-26T20:35:10.1290496Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1290715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1290795Z return mod(**inputs) 2025-08-26T20:35:10.1291061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1816, in forward 2025-08-26T20:35:10.1291156Z lm_logits = self.lm_head(sequence_output) 2025-08-26T20:35:10.1291161Z 2025-08-26T20:35:10.1291283Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1291517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1291601Z return mod(**inputs) 2025-08-26T20:35:10.1291867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-26T20:35:10.1292028Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-26T20:35:10.1292039Z 2025-08-26T20:35:10.1292151Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1292378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1292458Z return mod(**inputs) 2025-08-26T20:35:10.1292726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-26T20:35:10.1292884Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-26T20:35:10.1292888Z 2025-08-26T20:35:10.1292998Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:10.1293214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:10.1293293Z return mod(**inputs) 2025-08-26T20:35:10.1293563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-26T20:35:10.1293712Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-26T20:35:10.1293716Z 2025-08-26T20:35:22.0627344Z Compilation time (from dynamo_timed): 24.03865129 2025-08-26T20:35:22.0815836Z pass 2025-08-26T20:35:22.0817827Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:35:22.0819121Z TIMING: _recursive_pre_grad_passes:0.0179 _recursive_joint_graph_passes:0.79853 _recursive_post_grad_passes:0.27331 async_compile.wait:0.87263 code_gen:11.35736 inductor_compile:14.32065 backend_compile:19.64702 gc:0.00059 entire_frame_compile:24.03865 total_wall_time:24.03865 2025-08-26T20:35:22.0820146Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:29413 | FakeTensor.__torch_dispatch__:8057 | ProxyTorchDispatchMode.__torch_dispatch__:10618 2025-08-26T20:35:22.0820697Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-08-26T20:35:27.9234289Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:35:27.9235297Z from pkg_resources import resource_filename 2025-08-26T20:35:28.5240795Z 2025-08-26T20:35:28.5370805Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-26T20:35:28.5371547Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-26T20:35:31.9660929Z 2025-08-26T20:35:31.9661627Z loading model: 0it [00:03, ?it/s] 2025-08-26T20:35:31.9680963Z cpu eval MegatronBertForCausalLM 2025-08-26T20:35:33.6531586Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:35:34.2736074Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:35:34.8969010Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:35:49.9155284Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9155909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9156294Z return mod(**inputs) 2025-08-26T20:35:49.9156802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9157682Z outputs = self.bert( 2025-08-26T20:35:49.9158309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9160002Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9160666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9161509Z layer_outputs = layer_module( 2025-08-26T20:35:49.9161929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9162357Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9162865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9163363Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9163861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9164349Z self_outputs = self.self( 2025-08-26T20:35:49.9164773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9165257Z return func(*args, **kwargs) 2025-08-26T20:35:49.9165726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9166216Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9166455Z 2025-08-26T20:35:49.9166588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9167003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9167382Z return mod(**inputs) 2025-08-26T20:35:49.9167835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9168309Z outputs = self.bert( 2025-08-26T20:35:49.9168745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9169258Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9169729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9170210Z layer_outputs = layer_module( 2025-08-26T20:35:49.9170603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9171010Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9171503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9171999Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9172482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9172954Z self_outputs = self.self( 2025-08-26T20:35:49.9173424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9173859Z return func(*args, **kwargs) 2025-08-26T20:35:49.9174338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9174814Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9174968Z 2025-08-26T20:35:49.9175097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9175497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9175891Z return mod(**inputs) 2025-08-26T20:35:49.9176348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9176816Z outputs = self.bert( 2025-08-26T20:35:49.9177252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9177719Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9178199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9178659Z layer_outputs = layer_module( 2025-08-26T20:35:49.9179038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9179436Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9179906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9180379Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9180885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9181344Z self_outputs = self.self( 2025-08-26T20:35:49.9181740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9182150Z return func(*args, **kwargs) 2025-08-26T20:35:49.9182608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9183097Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9183243Z 2025-08-26T20:35:49.9183335Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9183574Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9183837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9184239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9184613Z return mod(**inputs) 2025-08-26T20:35:49.9185056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9185507Z outputs = self.bert( 2025-08-26T20:35:49.9185939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9186398Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9186840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9187316Z layer_outputs = layer_module( 2025-08-26T20:35:49.9187697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9188089Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9188546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9189033Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9189519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9190036Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9190562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9191033Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9191183Z 2025-08-26T20:35:49.9191296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9191709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9192069Z return mod(**inputs) 2025-08-26T20:35:49.9192504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9192963Z outputs = self.bert( 2025-08-26T20:35:49.9193387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9193884Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9194339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9194789Z layer_outputs = layer_module( 2025-08-26T20:35:49.9195904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9196869Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9197667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9198385Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9199041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9199906Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9200525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9201416Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9202203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9202902Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9203140Z 2025-08-26T20:35:49.9203317Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9203845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9204230Z return mod(**inputs) 2025-08-26T20:35:49.9204693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9205274Z outputs = self.bert( 2025-08-26T20:35:49.9205729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9206215Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9206727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9207203Z layer_outputs = layer_module( 2025-08-26T20:35:49.9207592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9208003Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9208534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9209003Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9209448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9211139Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9211660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9212188Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9212779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9213284Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9213701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9214093Z return self.act(input) 2025-08-26T20:35:49.9214228Z 2025-08-26T20:35:49.9214347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9230284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9230806Z return mod(**inputs) 2025-08-26T20:35:49.9231316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9232874Z outputs = self.bert( 2025-08-26T20:35:49.9233352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9233857Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9234340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9234831Z layer_outputs = layer_module( 2025-08-26T20:35:49.9235374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9235915Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9236395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9237072Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9237535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9238200Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9238715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9239283Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9240986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9241507Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9241676Z 2025-08-26T20:35:49.9241799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9242207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9242557Z return mod(**inputs) 2025-08-26T20:35:49.9242994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9243442Z outputs = self.bert( 2025-08-26T20:35:49.9243850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9244272Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9244751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9245237Z layer_outputs = layer_module( 2025-08-26T20:35:49.9245627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9246020Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9246486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9246917Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9247424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9247861Z self_outputs = self.self( 2025-08-26T20:35:49.9248241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9248627Z return func(*args, **kwargs) 2025-08-26T20:35:49.9249061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9249550Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9249691Z 2025-08-26T20:35:49.9249811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9250185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9250514Z return mod(**inputs) 2025-08-26T20:35:49.9250923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9251346Z outputs = self.bert( 2025-08-26T20:35:49.9251747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9252177Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9252598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9253025Z layer_outputs = layer_module( 2025-08-26T20:35:49.9253384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9253791Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9254254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9254714Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9255174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9255630Z self_outputs = self.self( 2025-08-26T20:35:49.9256004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9256381Z return func(*args, **kwargs) 2025-08-26T20:35:49.9256801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9257239Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9257377Z 2025-08-26T20:35:49.9257494Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9257867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9258192Z return mod(**inputs) 2025-08-26T20:35:49.9258602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9259048Z outputs = self.bert( 2025-08-26T20:35:49.9259488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9259922Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9260339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9260773Z layer_outputs = layer_module( 2025-08-26T20:35:49.9261124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9261485Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9261913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9262341Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9262762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9263178Z self_outputs = self.self( 2025-08-26T20:35:49.9263550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9263968Z return func(*args, **kwargs) 2025-08-26T20:35:49.9264405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9264863Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9265009Z 2025-08-26T20:35:49.9265107Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9265352Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9265591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9265965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9266300Z return mod(**inputs) 2025-08-26T20:35:49.9266706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9267131Z outputs = self.bert( 2025-08-26T20:35:49.9267518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9267946Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9268399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9268828Z layer_outputs = layer_module( 2025-08-26T20:35:49.9269197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9269592Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9270056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9270522Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9270966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9271464Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9271942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9272376Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9272515Z 2025-08-26T20:35:49.9272629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9272986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9273336Z return mod(**inputs) 2025-08-26T20:35:49.9273768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9274239Z outputs = self.bert( 2025-08-26T20:35:49.9274665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9275114Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9275563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9276021Z layer_outputs = layer_module( 2025-08-26T20:35:49.9276392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9276781Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9277255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9277722Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9278161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9278591Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9279103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9279804Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9280318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9280799Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9280963Z 2025-08-26T20:35:49.9281090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9281475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9281826Z return mod(**inputs) 2025-08-26T20:35:49.9282260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9282713Z outputs = self.bert( 2025-08-26T20:35:49.9283143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9283614Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9284065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9284516Z layer_outputs = layer_module( 2025-08-26T20:35:49.9284895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9285284Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9285741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9286209Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9286646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9287075Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9287552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9288077Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9288578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9289094Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9289485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9289835Z return self.act(input) 2025-08-26T20:35:49.9289984Z 2025-08-26T20:35:49.9290100Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9290512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9290873Z return mod(**inputs) 2025-08-26T20:35:49.9291311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9291775Z outputs = self.bert( 2025-08-26T20:35:49.9292182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9292655Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9293084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9293566Z layer_outputs = layer_module( 2025-08-26T20:35:49.9293914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9294301Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9294792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9295258Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9295684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9296112Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9296778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9297337Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9297854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9298320Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9298480Z 2025-08-26T20:35:49.9298598Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9298989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9299427Z return mod(**inputs) 2025-08-26T20:35:49.9299860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9300308Z outputs = self.bert( 2025-08-26T20:35:49.9300753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9301212Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9301645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9302082Z layer_outputs = layer_module( 2025-08-26T20:35:49.9302430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9302802Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9303238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9303678Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9304084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9304488Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9304946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9305496Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9305983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9306422Z return input_tensor + hidden_states 2025-08-26T20:35:49.9306569Z 2025-08-26T20:35:49.9306683Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9307054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9307395Z return mod(**inputs) 2025-08-26T20:35:49.9307827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9308254Z outputs = self.bert( 2025-08-26T20:35:49.9308678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9309132Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9309581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9310061Z layer_outputs = layer_module( 2025-08-26T20:35:49.9310440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9310814Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9311246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9311688Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9312148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9312601Z self_outputs = self.self( 2025-08-26T20:35:49.9313000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9313402Z return func(*args, **kwargs) 2025-08-26T20:35:49.9313844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9314346Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9314503Z 2025-08-26T20:35:49.9314617Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9315005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9315356Z return mod(**inputs) 2025-08-26T20:35:49.9315788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9316233Z outputs = self.bert( 2025-08-26T20:35:49.9316668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9317119Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9317571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9318028Z layer_outputs = layer_module( 2025-08-26T20:35:49.9318399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9318790Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9319263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9319804Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9320308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9320804Z self_outputs = self.self( 2025-08-26T20:35:49.9321221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9321610Z return func(*args, **kwargs) 2025-08-26T20:35:49.9322033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9322479Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9322635Z 2025-08-26T20:35:49.9322752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9323153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9323538Z return mod(**inputs) 2025-08-26T20:35:49.9324000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9324461Z outputs = self.bert( 2025-08-26T20:35:49.9324967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9325438Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9325923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9326392Z layer_outputs = layer_module( 2025-08-26T20:35:49.9326772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9327175Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9327656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9328131Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9328626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9329090Z self_outputs = self.self( 2025-08-26T20:35:49.9329463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9329848Z return func(*args, **kwargs) 2025-08-26T20:35:49.9330283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9330714Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9330860Z 2025-08-26T20:35:49.9330944Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9331166Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9331411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9331792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9332139Z return mod(**inputs) 2025-08-26T20:35:49.9332565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9332984Z outputs = self.bert( 2025-08-26T20:35:49.9333387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9333816Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9334243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9334665Z layer_outputs = layer_module( 2025-08-26T20:35:49.9335018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9335381Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9335828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9336268Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9336706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9337200Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9337686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9338120Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9338271Z 2025-08-26T20:35:49.9338379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9338768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9339107Z return mod(**inputs) 2025-08-26T20:35:49.9339509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9339950Z outputs = self.bert( 2025-08-26T20:35:49.9340372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9340853Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9341312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9341758Z layer_outputs = layer_module( 2025-08-26T20:35:49.9342138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9342535Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9342991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9343457Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9343881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9344291Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9344751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9345302Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9345785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9346248Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9346397Z 2025-08-26T20:35:49.9346506Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9346904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9347264Z return mod(**inputs) 2025-08-26T20:35:49.9347697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9348152Z outputs = self.bert( 2025-08-26T20:35:49.9348581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9349041Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9349488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9349937Z layer_outputs = layer_module( 2025-08-26T20:35:49.9350321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9350713Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9351212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9351680Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9352119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9352559Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9353055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9353581Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9354098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9354594Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9355012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9355384Z return self.act(input) 2025-08-26T20:35:49.9355504Z 2025-08-26T20:35:49.9356159Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9356571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9356917Z return mod(**inputs) 2025-08-26T20:35:49.9357356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9357801Z outputs = self.bert( 2025-08-26T20:35:49.9358233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9358691Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9359161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9359730Z layer_outputs = layer_module( 2025-08-26T20:35:49.9360138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9360546Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9361017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9361516Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9361957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9362396Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9362887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9363428Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9363946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9364424Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9364574Z 2025-08-26T20:35:49.9364694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9365094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9365422Z return mod(**inputs) 2025-08-26T20:35:49.9365829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9366265Z outputs = self.bert( 2025-08-26T20:35:49.9366669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9367091Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9367542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9367972Z layer_outputs = layer_module( 2025-08-26T20:35:49.9368333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9368702Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9369131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9369581Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9370075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9370531Z self_outputs = self.self( 2025-08-26T20:35:49.9370926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9371334Z return func(*args, **kwargs) 2025-08-26T20:35:49.9371755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9372234Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9372384Z 2025-08-26T20:35:49.9372502Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9372885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9373238Z return mod(**inputs) 2025-08-26T20:35:49.9373677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9374129Z outputs = self.bert( 2025-08-26T20:35:49.9374564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9375010Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9375456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9375905Z layer_outputs = layer_module( 2025-08-26T20:35:49.9376280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9376694Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9377155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9377623Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9378081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9378530Z self_outputs = self.self( 2025-08-26T20:35:49.9378918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9379320Z return func(*args, **kwargs) 2025-08-26T20:35:49.9379771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9380245Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9380390Z 2025-08-26T20:35:49.9380511Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9380895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9381249Z return mod(**inputs) 2025-08-26T20:35:49.9381688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9382132Z outputs = self.bert( 2025-08-26T20:35:49.9382594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9383049Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9383513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9383972Z layer_outputs = layer_module( 2025-08-26T20:35:49.9384345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9384732Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9385211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9385672Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9386137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9386598Z self_outputs = self.self( 2025-08-26T20:35:49.9386987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9387409Z return func(*args, **kwargs) 2025-08-26T20:35:49.9387856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9388322Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9388467Z 2025-08-26T20:35:49.9388564Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9388793Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9389055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9389449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9389801Z return mod(**inputs) 2025-08-26T20:35:49.9390225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9390670Z outputs = self.bert( 2025-08-26T20:35:49.9391093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9391575Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9392028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9392471Z layer_outputs = layer_module( 2025-08-26T20:35:49.9392851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9393238Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9393700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9394156Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9394615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9395128Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9395638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9396101Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9396413Z 2025-08-26T20:35:49.9396534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9396934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9397302Z return mod(**inputs) 2025-08-26T20:35:49.9397748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9398272Z outputs = self.bert( 2025-08-26T20:35:49.9398715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9399187Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9399706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9400189Z layer_outputs = layer_module( 2025-08-26T20:35:49.9400580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9401014Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9401500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9401995Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9402453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9402894Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9403436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9403973Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9404484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9404970Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9405127Z 2025-08-26T20:35:49.9405250Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9405663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9406027Z return mod(**inputs) 2025-08-26T20:35:49.9406488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9406951Z outputs = self.bert( 2025-08-26T20:35:49.9407391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9407895Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9408321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9408746Z layer_outputs = layer_module( 2025-08-26T20:35:49.9409097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9409461Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9409892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9410326Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9410739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9411139Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9411594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9412087Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9412543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9413010Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9413397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9413816Z return self.act(input) 2025-08-26T20:35:49.9413942Z 2025-08-26T20:35:49.9414051Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9414421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9414753Z return mod(**inputs) 2025-08-26T20:35:49.9415151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9415579Z outputs = self.bert( 2025-08-26T20:35:49.9415983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9416436Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9416867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9417290Z layer_outputs = layer_module( 2025-08-26T20:35:49.9417667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9418061Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9418549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9419011Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9419448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9419855Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9420316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9420845Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9421324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9421767Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9421921Z 2025-08-26T20:35:49.9422029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9422400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9422752Z return mod(**inputs) 2025-08-26T20:35:49.9423152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9423578Z outputs = self.bert( 2025-08-26T20:35:49.9423988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9424403Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9424821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9425241Z layer_outputs = layer_module( 2025-08-26T20:35:49.9425596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9425969Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9426406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9426836Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9427244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9427645Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9428125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9428686Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9429190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9429658Z return input_tensor + hidden_states 2025-08-26T20:35:49.9429819Z 2025-08-26T20:35:49.9429927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9430300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9430634Z return mod(**inputs) 2025-08-26T20:35:49.9431061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9431493Z outputs = self.bert( 2025-08-26T20:35:49.9431895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9432324Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9432747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9433211Z layer_outputs = layer_module( 2025-08-26T20:35:49.9433589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9433983Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9434438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9434892Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9435355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9435805Z self_outputs = self.self( 2025-08-26T20:35:49.9436203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9436605Z return func(*args, **kwargs) 2025-08-26T20:35:49.9437052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9437530Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9437685Z 2025-08-26T20:35:49.9437798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9438197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9438555Z return mod(**inputs) 2025-08-26T20:35:49.9439001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9439535Z outputs = self.bert( 2025-08-26T20:35:49.9439996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9440465Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9440928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9441387Z layer_outputs = layer_module( 2025-08-26T20:35:49.9441771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9442168Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9442624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9443079Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9443536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9444014Z self_outputs = self.self( 2025-08-26T20:35:49.9444408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9444812Z return func(*args, **kwargs) 2025-08-26T20:35:49.9445257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9445720Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9445874Z 2025-08-26T20:35:49.9445989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9446398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9446748Z return mod(**inputs) 2025-08-26T20:35:49.9447172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9447628Z outputs = self.bert( 2025-08-26T20:35:49.9448049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9448522Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9448963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9449419Z layer_outputs = layer_module( 2025-08-26T20:35:49.9449794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9450179Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9450638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9451098Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9451570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9452036Z self_outputs = self.self( 2025-08-26T20:35:49.9452429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9452829Z return func(*args, **kwargs) 2025-08-26T20:35:49.9453280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9453721Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9453867Z 2025-08-26T20:35:49.9453952Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9454181Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9454421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9454795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9455133Z return mod(**inputs) 2025-08-26T20:35:49.9455546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9455976Z outputs = self.bert( 2025-08-26T20:35:49.9456377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9456817Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9457248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9457679Z layer_outputs = layer_module( 2025-08-26T20:35:49.9458032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9458399Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9458858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9459293Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9459736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9460236Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9460747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9461196Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9461339Z 2025-08-26T20:35:49.9461477Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9461845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9462168Z return mod(**inputs) 2025-08-26T20:35:49.9462577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9463000Z outputs = self.bert( 2025-08-26T20:35:49.9463401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9463867Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9464288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9464715Z layer_outputs = layer_module( 2025-08-26T20:35:49.9465073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9465445Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9465872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9466317Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9466726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9467131Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9467588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9468086Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9468542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9468972Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9469112Z 2025-08-26T20:35:49.9469229Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9469600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9469921Z return mod(**inputs) 2025-08-26T20:35:49.9470326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9470763Z outputs = self.bert( 2025-08-26T20:35:49.9471156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9471583Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9472002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9472426Z layer_outputs = layer_module( 2025-08-26T20:35:49.9472780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9473154Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9473624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9474091Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9474527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9474963Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9475448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9475963Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9476467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9476976Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9477390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9477757Z return self.act(input) 2025-08-26T20:35:49.9477879Z 2025-08-26T20:35:49.9477993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9478404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9478754Z return mod(**inputs) 2025-08-26T20:35:49.9479182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9479714Z outputs = self.bert( 2025-08-26T20:35:49.9480161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9480641Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9481094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9481532Z layer_outputs = layer_module( 2025-08-26T20:35:49.9481870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9482234Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9482659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9483127Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9483538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9483938Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9484402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9484905Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9485374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9485797Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9485932Z 2025-08-26T20:35:49.9486036Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9486395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9486716Z return mod(**inputs) 2025-08-26T20:35:49.9487112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9487518Z outputs = self.bert( 2025-08-26T20:35:49.9487901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9488345Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9488769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9489195Z layer_outputs = layer_module( 2025-08-26T20:35:49.9489543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9489915Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9490369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9490816Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9491263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9491684Z self_outputs = self.self( 2025-08-26T20:35:49.9492056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9492435Z return func(*args, **kwargs) 2025-08-26T20:35:49.9492851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9493308Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9493448Z 2025-08-26T20:35:49.9493555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9493928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9494271Z return mod(**inputs) 2025-08-26T20:35:49.9494687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9495114Z outputs = self.bert( 2025-08-26T20:35:49.9495544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9496017Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9496566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9497003Z layer_outputs = layer_module( 2025-08-26T20:35:49.9497414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9497811Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9498264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9498706Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9499136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9499574Z self_outputs = self.self( 2025-08-26T20:35:49.9499969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9500372Z return func(*args, **kwargs) 2025-08-26T20:35:49.9500816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9501274Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9501420Z 2025-08-26T20:35:49.9501527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9501899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9502232Z return mod(**inputs) 2025-08-26T20:35:49.9502637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9503052Z outputs = self.bert( 2025-08-26T20:35:49.9503487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9503916Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9504341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9504765Z layer_outputs = layer_module( 2025-08-26T20:35:49.9505112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9505477Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9505931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9506367Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9506795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9507212Z self_outputs = self.self( 2025-08-26T20:35:49.9507578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9507981Z return func(*args, **kwargs) 2025-08-26T20:35:49.9508400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9508859Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9509011Z 2025-08-26T20:35:49.9509099Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9509332Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9509594Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9509987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9510330Z return mod(**inputs) 2025-08-26T20:35:49.9510762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9511207Z outputs = self.bert( 2025-08-26T20:35:49.9511635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9512106Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9512555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9513010Z layer_outputs = layer_module( 2025-08-26T20:35:49.9513387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9513778Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9514228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9514689Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9515145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9515660Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9516170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9516622Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9516777Z 2025-08-26T20:35:49.9516890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9517283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9517633Z return mod(**inputs) 2025-08-26T20:35:49.9518077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9518531Z outputs = self.bert( 2025-08-26T20:35:49.9518971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9519490Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9519968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9520432Z layer_outputs = layer_module( 2025-08-26T20:35:49.9520815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9521251Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9521712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9522185Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9522616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9523064Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9523553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9524069Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9524549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9524999Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9525157Z 2025-08-26T20:35:49.9525270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9525659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9526011Z return mod(**inputs) 2025-08-26T20:35:49.9526421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9526858Z outputs = self.bert( 2025-08-26T20:35:49.9527287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9527765Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9528221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9528647Z layer_outputs = layer_module( 2025-08-26T20:35:49.9529009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9529400Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9529864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9530334Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9530766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9531211Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9531678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9532175Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9532639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9533119Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9533578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9533948Z return self.act(input) 2025-08-26T20:35:49.9534071Z 2025-08-26T20:35:49.9534193Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9534586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9534950Z return mod(**inputs) 2025-08-26T20:35:49.9535359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9535797Z outputs = self.bert( 2025-08-26T20:35:49.9536214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9536635Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9537064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9537492Z layer_outputs = layer_module( 2025-08-26T20:35:49.9537850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9538259Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9538698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9539171Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9539603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9540028Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9540506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9541051Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9541564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9542022Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9542173Z 2025-08-26T20:35:49.9542294Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9542699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9543049Z return mod(**inputs) 2025-08-26T20:35:49.9543484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9543942Z outputs = self.bert( 2025-08-26T20:35:49.9544374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9544828Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9545290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9545749Z layer_outputs = layer_module( 2025-08-26T20:35:49.9546132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9546526Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9546995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9547459Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9547896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9548326Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9548831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9549370Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9549874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9550405Z return input_tensor + hidden_states 2025-08-26T20:35:49.9550550Z 2025-08-26T20:35:49.9550672Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9551050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9551395Z return mod(**inputs) 2025-08-26T20:35:49.9551844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9552297Z outputs = self.bert( 2025-08-26T20:35:49.9552732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9553188Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9553658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9554129Z layer_outputs = layer_module( 2025-08-26T20:35:49.9554506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9554898Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9555348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9555815Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9556275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9556726Z self_outputs = self.self( 2025-08-26T20:35:49.9557112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9557514Z return func(*args, **kwargs) 2025-08-26T20:35:49.9557958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9558446Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9558598Z 2025-08-26T20:35:49.9558723Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9559116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9559556Z return mod(**inputs) 2025-08-26T20:35:49.9560013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9560483Z outputs = self.bert( 2025-08-26T20:35:49.9560923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9561376Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9561837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9562308Z layer_outputs = layer_module( 2025-08-26T20:35:49.9562698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9563101Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9563569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9564048Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9564561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9565026Z self_outputs = self.self( 2025-08-26T20:35:49.9565422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9565838Z return func(*args, **kwargs) 2025-08-26T20:35:49.9566295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9566768Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9566917Z 2025-08-26T20:35:49.9567041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9567454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9567822Z return mod(**inputs) 2025-08-26T20:35:49.9568271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9568734Z outputs = self.bert( 2025-08-26T20:35:49.9569166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9569648Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9570090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9570517Z layer_outputs = layer_module( 2025-08-26T20:35:49.9570871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9571247Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9571708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9572168Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9572625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9573048Z self_outputs = self.self( 2025-08-26T20:35:49.9573414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9573816Z return func(*args, **kwargs) 2025-08-26T20:35:49.9574237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9574671Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9574810Z 2025-08-26T20:35:49.9574909Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9575135Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9575392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9575777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9576133Z return mod(**inputs) 2025-08-26T20:35:49.9576530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9576954Z outputs = self.bert( 2025-08-26T20:35:49.9577358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9577811Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9578245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9578670Z layer_outputs = layer_module( 2025-08-26T20:35:49.9579044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9579431Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9579893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9580328Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9580761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9581266Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9581789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9582254Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9582403Z 2025-08-26T20:35:49.9582546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9582927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9583278Z return mod(**inputs) 2025-08-26T20:35:49.9583709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9584154Z outputs = self.bert( 2025-08-26T20:35:49.9584605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9585059Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9585510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9585932Z layer_outputs = layer_module( 2025-08-26T20:35:49.9586287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9586645Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9587074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9587511Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9587920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9588326Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9588823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9589343Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9589829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9590295Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9590446Z 2025-08-26T20:35:49.9590560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9590961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9591291Z return mod(**inputs) 2025-08-26T20:35:49.9591724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9592173Z outputs = self.bert( 2025-08-26T20:35:49.9592587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9593039Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9593485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9593937Z layer_outputs = layer_module( 2025-08-26T20:35:49.9594307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9594686Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9595163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9595629Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9596058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9596622Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9597106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9597638Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9598190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9598702Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9599123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9599555Z return self.act(input) 2025-08-26T20:35:49.9599727Z 2025-08-26T20:35:49.9599847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9600272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9600639Z return mod(**inputs) 2025-08-26T20:35:49.9601144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9601601Z outputs = self.bert( 2025-08-26T20:35:49.9602032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9602493Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9602954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9603400Z layer_outputs = layer_module( 2025-08-26T20:35:49.9603780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9604169Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9604673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9605138Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9605583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9606020Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9606512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9607072Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9607586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9608063Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9608225Z 2025-08-26T20:35:49.9608342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9608740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9609098Z return mod(**inputs) 2025-08-26T20:35:49.9609537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9609993Z outputs = self.bert( 2025-08-26T20:35:49.9610428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9610891Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9611319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9611735Z layer_outputs = layer_module( 2025-08-26T20:35:49.9612092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9612459Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9612889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9613338Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9613773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9614213Z self_outputs = self.self( 2025-08-26T20:35:49.9614575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9614943Z return func(*args, **kwargs) 2025-08-26T20:35:49.9615360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9615783Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9615925Z 2025-08-26T20:35:49.9616028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9616384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9616706Z return mod(**inputs) 2025-08-26T20:35:49.9617098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9617174Z outputs = self.bert( 2025-08-26T20:35:49.9617466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9617550Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9617836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9617931Z layer_outputs = layer_module( 2025-08-26T20:35:49.9618166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9618251Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9618572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9618659Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9618976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9619058Z self_outputs = self.self( 2025-08-26T20:35:49.9619313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9619394Z return func(*args, **kwargs) 2025-08-26T20:35:49.9619697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9619791Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9619795Z 2025-08-26T20:35:49.9619908Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9620126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9620219Z return mod(**inputs) 2025-08-26T20:35:49.9620516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9620589Z outputs = self.bert( 2025-08-26T20:35:49.9620906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9620984Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9621274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9621349Z layer_outputs = layer_module( 2025-08-26T20:35:49.9621583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9621660Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9621976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9622059Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9622349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9622427Z self_outputs = self.self( 2025-08-26T20:35:49.9622671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9622768Z return func(*args, **kwargs) 2025-08-26T20:35:49.9623059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9623139Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9623143Z 2025-08-26T20:35:49.9623234Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9623316Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9623440Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9623637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9623704Z return mod(**inputs) 2025-08-26T20:35:49.9624001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9624068Z outputs = self.bert( 2025-08-26T20:35:49.9624357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9624460Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9624749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9624820Z layer_outputs = layer_module( 2025-08-26T20:35:49.9625044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9625129Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9625414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9625501Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9625786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9625919Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9626217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9626302Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9626306Z 2025-08-26T20:35:49.9626421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9626622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9626697Z return mod(**inputs) 2025-08-26T20:35:49.9627008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9627077Z outputs = self.bert( 2025-08-26T20:35:49.9627376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9627452Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9627750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9627823Z layer_outputs = layer_module( 2025-08-26T20:35:49.9628850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9628960Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9629251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9629346Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9629609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9629720Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9630045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9630155Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9630457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9630545Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9630549Z 2025-08-26T20:35:49.9630664Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9630867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9630936Z return mod(**inputs) 2025-08-26T20:35:49.9631241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9631311Z outputs = self.bert( 2025-08-26T20:35:49.9631611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9631733Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9632026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9632102Z layer_outputs = layer_module( 2025-08-26T20:35:49.9632339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9632429Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9632743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9632841Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9633119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9633203Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9633547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9633661Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9633975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9634098Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9634352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9634431Z return self.act(input) 2025-08-26T20:35:49.9634435Z 2025-08-26T20:35:49.9634547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9634770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9634842Z return mod(**inputs) 2025-08-26T20:35:49.9635162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9635233Z outputs = self.bert( 2025-08-26T20:35:49.9635559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9635647Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9635952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9636038Z layer_outputs = layer_module( 2025-08-26T20:35:49.9636275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9636385Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9636690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9636783Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9637071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9637153Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9637499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9637644Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9637952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9638049Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9638053Z 2025-08-26T20:35:49.9638163Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9638404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9638475Z return mod(**inputs) 2025-08-26T20:35:49.9638799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9638875Z outputs = self.bert( 2025-08-26T20:35:49.9639188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9639276Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9639676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9639773Z layer_outputs = layer_module( 2025-08-26T20:35:49.9640019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9640109Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9640435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9640527Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9640825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9640909Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9641293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9641441Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9641757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9641849Z return input_tensor + hidden_states 2025-08-26T20:35:49.9641853Z 2025-08-26T20:35:49.9641958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9642165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9642232Z return mod(**inputs) 2025-08-26T20:35:49.9642543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9642621Z outputs = self.bert( 2025-08-26T20:35:49.9642917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9642999Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9643308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9643412Z layer_outputs = layer_module( 2025-08-26T20:35:49.9643648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9643732Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9644050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9644139Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9644462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9644538Z self_outputs = self.self( 2025-08-26T20:35:49.9644786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9644867Z return func(*args, **kwargs) 2025-08-26T20:35:49.9645157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9645268Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9645271Z 2025-08-26T20:35:49.9645377Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9645588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9645655Z return mod(**inputs) 2025-08-26T20:35:49.9645948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9646022Z outputs = self.bert( 2025-08-26T20:35:49.9646312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9646395Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9646683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9646757Z layer_outputs = layer_module( 2025-08-26T20:35:49.9646989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9647068Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9647361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9647445Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9647757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9647830Z self_outputs = self.self( 2025-08-26T20:35:49.9648087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9648174Z return func(*args, **kwargs) 2025-08-26T20:35:49.9648483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9648572Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9648576Z 2025-08-26T20:35:49.9648686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9648916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9648998Z return mod(**inputs) 2025-08-26T20:35:49.9649311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9649389Z outputs = self.bert( 2025-08-26T20:35:49.9649700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9649811Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9650130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9650204Z layer_outputs = layer_module( 2025-08-26T20:35:49.9650437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9650518Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9650817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9650900Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9651197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9651283Z self_outputs = self.self( 2025-08-26T20:35:49.9651545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9651648Z return func(*args, **kwargs) 2025-08-26T20:35:49.9651957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9652043Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9652054Z 2025-08-26T20:35:49.9652144Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9652230Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9652347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9652559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9652636Z return mod(**inputs) 2025-08-26T20:35:49.9652942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9653016Z outputs = self.bert( 2025-08-26T20:35:49.9653330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9653411Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9653722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9653799Z layer_outputs = layer_module( 2025-08-26T20:35:49.9654037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9654126Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9654458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9654555Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9654862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9655009Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9655318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9655404Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9655426Z 2025-08-26T20:35:49.9655547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9655761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9655839Z return mod(**inputs) 2025-08-26T20:35:49.9656199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9656291Z outputs = self.bert( 2025-08-26T20:35:49.9656617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9656697Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9657013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9657090Z layer_outputs = layer_module( 2025-08-26T20:35:49.9657336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9657419Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9657735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9657832Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9658110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9658202Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9658565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9658679Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9659003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9659092Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9659096Z 2025-08-26T20:35:49.9659216Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9659431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9659509Z return mod(**inputs) 2025-08-26T20:35:49.9659820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9659894Z outputs = self.bert( 2025-08-26T20:35:49.9660208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9660287Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9660601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9660678Z layer_outputs = layer_module( 2025-08-26T20:35:49.9660914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9661036Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9661327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9661421Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9661685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9661765Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9662088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9662214Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9662530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9662644Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9662861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9662933Z return self.act(input) 2025-08-26T20:35:49.9662951Z 2025-08-26T20:35:49.9663057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9663266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9663331Z return mod(**inputs) 2025-08-26T20:35:49.9663625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9663693Z outputs = self.bert( 2025-08-26T20:35:49.9663986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9664070Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9664368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9664447Z layer_outputs = layer_module( 2025-08-26T20:35:49.9664669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9664753Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9665052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9665135Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9665402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9665476Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9665797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9665932Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9666220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9666303Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9666308Z 2025-08-26T20:35:49.9666411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9666615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9666679Z return mod(**inputs) 2025-08-26T20:35:49.9666973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9667039Z outputs = self.bert( 2025-08-26T20:35:49.9667338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9667419Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9667700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9667779Z layer_outputs = layer_module( 2025-08-26T20:35:49.9667999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9668076Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9668366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9668463Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9668755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9668827Z self_outputs = self.self( 2025-08-26T20:35:49.9669075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9669145Z return func(*args, **kwargs) 2025-08-26T20:35:49.9669444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9669534Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9669537Z 2025-08-26T20:35:49.9669639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9669842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9669907Z return mod(**inputs) 2025-08-26T20:35:49.9670201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9670274Z outputs = self.bert( 2025-08-26T20:35:49.9670567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9670650Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9670941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9671037Z layer_outputs = layer_module( 2025-08-26T20:35:49.9671258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9671339Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9671637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9671719Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9672020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9672092Z self_outputs = self.self( 2025-08-26T20:35:49.9672341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9672428Z return func(*args, **kwargs) 2025-08-26T20:35:49.9672710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9672796Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9672799Z 2025-08-26T20:35:49.9672901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9673104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9673169Z return mod(**inputs) 2025-08-26T20:35:49.9673457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9673546Z outputs = self.bert( 2025-08-26T20:35:49.9673841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9673923Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9674209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9674282Z layer_outputs = layer_module( 2025-08-26T20:35:49.9674514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9674591Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9674930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9675014Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9675310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9675381Z self_outputs = self.self( 2025-08-26T20:35:49.9675642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9675724Z return func(*args, **kwargs) 2025-08-26T20:35:49.9676028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9676120Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9676125Z 2025-08-26T20:35:49.9676212Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9676300Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9676417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9676631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9676712Z return mod(**inputs) 2025-08-26T20:35:49.9677023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9677096Z outputs = self.bert( 2025-08-26T20:35:49.9677411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9677509Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9677824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9677902Z layer_outputs = layer_module( 2025-08-26T20:35:49.9678150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9678232Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9678543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9678638Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9678950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9679098Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9679404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9679571Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9679586Z 2025-08-26T20:35:49.9679707Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9679935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9680018Z return mod(**inputs) 2025-08-26T20:35:49.9680363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9680447Z outputs = self.bert( 2025-08-26T20:35:49.9680775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9680851Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9681154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9681229Z layer_outputs = layer_module( 2025-08-26T20:35:49.9681478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9681559Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9681852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9681950Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9682214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9682323Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9682644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9682759Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9683051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9683134Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9683138Z 2025-08-26T20:35:49.9683249Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9683449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9683522Z return mod(**inputs) 2025-08-26T20:35:49.9683813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9683882Z outputs = self.bert( 2025-08-26T20:35:49.9684180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9684274Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9684574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9684649Z layer_outputs = layer_module( 2025-08-26T20:35:49.9684881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9684961Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9685255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9685345Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9685612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9685698Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9686023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9686129Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9686430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9686545Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9686787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9686862Z return self.act(input) 2025-08-26T20:35:49.9686865Z 2025-08-26T20:35:49.9686976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9687177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9687245Z return mod(**inputs) 2025-08-26T20:35:49.9687545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9687612Z outputs = self.bert( 2025-08-26T20:35:49.9687926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9688001Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9688292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9688369Z layer_outputs = layer_module( 2025-08-26T20:35:49.9688592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9688694Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9688982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9689070Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9689333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9689411Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9689748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9689891Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9690213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9690296Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9690300Z 2025-08-26T20:35:49.9690422Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9690629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9690697Z return mod(**inputs) 2025-08-26T20:35:49.9691006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9691075Z outputs = self.bert( 2025-08-26T20:35:49.9691374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9691454Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9691761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9691849Z layer_outputs = layer_module( 2025-08-26T20:35:49.9692087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9692180Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9692488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9692575Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9692866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9692948Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9693315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9693463Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9693764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9693846Z return input_tensor + hidden_states 2025-08-26T20:35:49.9693850Z 2025-08-26T20:35:49.9693955Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9694168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9694252Z return mod(**inputs) 2025-08-26T20:35:49.9694559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9694632Z outputs = self.bert( 2025-08-26T20:35:49.9694950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9695038Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9695364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9695448Z layer_outputs = layer_module( 2025-08-26T20:35:49.9695686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9695778Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9696095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9696369Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9696702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9696780Z self_outputs = self.self( 2025-08-26T20:35:49.9697054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9697132Z return func(*args, **kwargs) 2025-08-26T20:35:49.9697439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9697598Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9697602Z 2025-08-26T20:35:49.9697716Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9698007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9698108Z return mod(**inputs) 2025-08-26T20:35:49.9698621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9698723Z outputs = self.bert( 2025-08-26T20:35:49.9699033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9699125Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9699434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9699521Z layer_outputs = layer_module( 2025-08-26T20:35:49.9699819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9699904Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9700223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9700310Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9700679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9700756Z self_outputs = self.self( 2025-08-26T20:35:49.9701019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9701102Z return func(*args, **kwargs) 2025-08-26T20:35:49.9701413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9701506Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9701510Z 2025-08-26T20:35:49.9701624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9701878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9701953Z return mod(**inputs) 2025-08-26T20:35:49.9702277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9702356Z outputs = self.bert( 2025-08-26T20:35:49.9702674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9702792Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9703107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9703186Z layer_outputs = layer_module( 2025-08-26T20:35:49.9703437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9703534Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9703847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9703936Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9704244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9704320Z self_outputs = self.self( 2025-08-26T20:35:49.9704578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9704679Z return func(*args, **kwargs) 2025-08-26T20:35:49.9704984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9705076Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9705081Z 2025-08-26T20:35:49.9705168Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9705252Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9705369Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9705581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9705656Z return mod(**inputs) 2025-08-26T20:35:49.9705969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9706040Z outputs = self.bert( 2025-08-26T20:35:49.9706354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9706433Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9706749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9706825Z layer_outputs = layer_module( 2025-08-26T20:35:49.9707069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9707152Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9707476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9707573Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9707883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9708025Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9708314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9708417Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9708430Z 2025-08-26T20:35:49.9708535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9708734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9708813Z return mod(**inputs) 2025-08-26T20:35:49.9709104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9709198Z outputs = self.bert( 2025-08-26T20:35:49.9709491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9709568Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9709865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9709938Z layer_outputs = layer_module( 2025-08-26T20:35:49.9710169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9710247Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9710539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9710632Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9710898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9711006Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9711330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9711445Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9711743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9711830Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9711833Z 2025-08-26T20:35:49.9711946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9712152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9712225Z return mod(**inputs) 2025-08-26T20:35:49.9712546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9712638Z outputs = self.bert( 2025-08-26T20:35:49.9713121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9713237Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9713761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9713875Z layer_outputs = layer_module( 2025-08-26T20:35:49.9714130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9714238Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9714554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9714657Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9714948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9715039Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9715386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9715524Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9715850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9715977Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9716222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9716326Z return self.act(input) 2025-08-26T20:35:49.9716330Z 2025-08-26T20:35:49.9716451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9716674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9716747Z return mod(**inputs) 2025-08-26T20:35:49.9717075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9717149Z outputs = self.bert( 2025-08-26T20:35:49.9717474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9717555Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9717872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9717958Z layer_outputs = layer_module( 2025-08-26T20:35:49.9718202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9718318Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9718646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9718746Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9719045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9719128Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9719580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9719738Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9720073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9720168Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9720173Z 2025-08-26T20:35:49.9720295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9720526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9720599Z return mod(**inputs) 2025-08-26T20:35:49.9720918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9720986Z outputs = self.bert( 2025-08-26T20:35:49.9721310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9721388Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9721679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9721762Z layer_outputs = layer_module( 2025-08-26T20:35:49.9721987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9722076Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9722382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9722467Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9722771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9722848Z self_outputs = self.self( 2025-08-26T20:35:49.9723104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9723195Z return func(*args, **kwargs) 2025-08-26T20:35:49.9723490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9723574Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9723577Z 2025-08-26T20:35:49.9723681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9723893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9723960Z return mod(**inputs) 2025-08-26T20:35:49.9724264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9724329Z outputs = self.bert( 2025-08-26T20:35:49.9724621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9724703Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9724999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9725107Z layer_outputs = layer_module( 2025-08-26T20:35:49.9725332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9725417Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9725709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9725792Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9726090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9726163Z self_outputs = self.self( 2025-08-26T20:35:49.9726419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9726492Z return func(*args, **kwargs) 2025-08-26T20:35:49.9726783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9726870Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9726874Z 2025-08-26T20:35:49.9726978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9727189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9727255Z return mod(**inputs) 2025-08-26T20:35:49.9727557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9727643Z outputs = self.bert( 2025-08-26T20:35:49.9727935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9728021Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9728310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9728396Z layer_outputs = layer_module( 2025-08-26T20:35:49.9728635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9728735Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9729059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9729141Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9729442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9729513Z self_outputs = self.self( 2025-08-26T20:35:49.9729778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9729860Z return func(*args, **kwargs) 2025-08-26T20:35:49.9730150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9730238Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9730242Z 2025-08-26T20:35:49.9730325Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9730413Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9730517Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9730716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9730792Z return mod(**inputs) 2025-08-26T20:35:49.9731088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9731163Z outputs = self.bert( 2025-08-26T20:35:49.9731517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9731616Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9731932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9732010Z layer_outputs = layer_module( 2025-08-26T20:35:49.9732260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9732336Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9732637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9732719Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9733011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9733152Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9733442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9733532Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9733535Z 2025-08-26T20:35:49.9733641Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9733842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9733914Z return mod(**inputs) 2025-08-26T20:35:49.9734227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9734303Z outputs = self.bert( 2025-08-26T20:35:49.9734590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9734673Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9734962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9735034Z layer_outputs = layer_module( 2025-08-26T20:35:49.9735279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9735360Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9735659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9735743Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9736008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9736134Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9736457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9736571Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9736862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9736952Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9736956Z 2025-08-26T20:35:49.9737058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9737260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9737334Z return mod(**inputs) 2025-08-26T20:35:49.9737627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9737703Z outputs = self.bert( 2025-08-26T20:35:49.9738010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9738084Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9738407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9738484Z layer_outputs = layer_module( 2025-08-26T20:35:49.9738726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9738809Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9739123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9739212Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9739503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9739587Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9739907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9740020Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9740312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9740427Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9740672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9740750Z return self.act(input) 2025-08-26T20:35:49.9740755Z 2025-08-26T20:35:49.9740873Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9741088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9741167Z return mod(**inputs) 2025-08-26T20:35:49.9741475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9741546Z outputs = self.bert( 2025-08-26T20:35:49.9741878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9741959Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9742275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9742353Z layer_outputs = layer_module( 2025-08-26T20:35:49.9742589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9742702Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9743012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9743110Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9743390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9743471Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9743819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9743964Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9744279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9744370Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9744390Z 2025-08-26T20:35:49.9744510Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9744722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9744793Z return mod(**inputs) 2025-08-26T20:35:49.9745117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9745188Z outputs = self.bert( 2025-08-26T20:35:49.9745507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9745586Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9745899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9745977Z layer_outputs = layer_module( 2025-08-26T20:35:49.9746215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9746310Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9746616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9746713Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9747000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9747080Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9747446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9747590Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9747907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9747992Z return input_tensor + hidden_states 2025-08-26T20:35:49.9747997Z 2025-08-26T20:35:49.9748116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9748324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9748417Z return mod(**inputs) 2025-08-26T20:35:49.9748738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9748809Z outputs = self.bert( 2025-08-26T20:35:49.9749139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9749217Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9749549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9749636Z layer_outputs = layer_module( 2025-08-26T20:35:49.9749872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9749964Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9750280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9750376Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9750683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9750759Z self_outputs = self.self( 2025-08-26T20:35:49.9751028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9751103Z return func(*args, **kwargs) 2025-08-26T20:35:49.9751437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9751526Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9751530Z 2025-08-26T20:35:49.9751643Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9751866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9751937Z return mod(**inputs) 2025-08-26T20:35:49.9752257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9752326Z outputs = self.bert( 2025-08-26T20:35:49.9752631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9752718Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9753021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9753108Z layer_outputs = layer_module( 2025-08-26T20:35:49.9753344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9753436Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9753746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9753833Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9754164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9754241Z self_outputs = self.self( 2025-08-26T20:35:49.9754516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9754592Z return func(*args, **kwargs) 2025-08-26T20:35:49.9754900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9754988Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9754992Z 2025-08-26T20:35:49.9755122Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9755342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9755412Z return mod(**inputs) 2025-08-26T20:35:49.9755740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9755808Z outputs = self.bert( 2025-08-26T20:35:49.9756115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9756218Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9756525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9756608Z layer_outputs = layer_module( 2025-08-26T20:35:49.9756844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9756927Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9757242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9757332Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9757649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9757726Z self_outputs = self.self( 2025-08-26T20:35:49.9757991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9758084Z return func(*args, **kwargs) 2025-08-26T20:35:49.9758389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9758482Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9758486Z 2025-08-26T20:35:49.9758574Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9758665Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9758777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9758996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9759076Z return mod(**inputs) 2025-08-26T20:35:49.9759407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9759871Z outputs = self.bert( 2025-08-26T20:35:49.9760239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9760321Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9760666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9760745Z layer_outputs = layer_module( 2025-08-26T20:35:49.9761012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9761134Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9761504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9761599Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9761923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9762079Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9762393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9762514Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9762519Z 2025-08-26T20:35:49.9762633Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9762894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9762976Z return mod(**inputs) 2025-08-26T20:35:49.9763302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9763401Z outputs = self.bert( 2025-08-26T20:35:49.9763739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9763834Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9764169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9764251Z layer_outputs = layer_module( 2025-08-26T20:35:49.9764511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9764598Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9764939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9765035Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9765329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9765443Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9765794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9765920Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9766245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9766345Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9766349Z 2025-08-26T20:35:49.9766463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9766683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9766764Z return mod(**inputs) 2025-08-26T20:35:49.9767094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9767176Z outputs = self.bert( 2025-08-26T20:35:49.9767504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9767585Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9767960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9768040Z layer_outputs = layer_module( 2025-08-26T20:35:49.9768311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9768397Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9768728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9768821Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9769110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9769204Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9769573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9769702Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9770031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9770159Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9770401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9770496Z return self.act(input) 2025-08-26T20:35:49.9770501Z 2025-08-26T20:35:49.9770624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9770853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9770929Z return mod(**inputs) 2025-08-26T20:35:49.9771251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9771323Z outputs = self.bert( 2025-08-26T20:35:49.9771638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9771716Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9772041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9772118Z layer_outputs = layer_module( 2025-08-26T20:35:49.9772354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9772467Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9772785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9772877Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9773155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9773244Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9773582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9773726Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9774042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9774132Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9774135Z 2025-08-26T20:35:49.9774251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9774467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9774537Z return mod(**inputs) 2025-08-26T20:35:49.9774854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9774926Z outputs = self.bert( 2025-08-26T20:35:49.9775268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9775349Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9775666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9775745Z layer_outputs = layer_module( 2025-08-26T20:35:49.9775979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9776072Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9776411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9776508Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9776814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9776893Z self_outputs = self.self( 2025-08-26T20:35:49.9777164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9777257Z return func(*args, **kwargs) 2025-08-26T20:35:49.9777579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9777670Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9777674Z 2025-08-26T20:35:49.9777792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9778007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9778076Z return mod(**inputs) 2025-08-26T20:35:49.9778392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9778464Z outputs = self.bert( 2025-08-26T20:35:49.9778795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9778873Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9779184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9779287Z layer_outputs = layer_module( 2025-08-26T20:35:49.9779526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9779617Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9780006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9780105Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9780428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9780506Z self_outputs = self.self( 2025-08-26T20:35:49.9780792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9780868Z return func(*args, **kwargs) 2025-08-26T20:35:49.9781187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9781271Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9781275Z 2025-08-26T20:35:49.9781388Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9781611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9781681Z return mod(**inputs) 2025-08-26T20:35:49.9782021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9782093Z outputs = self.bert( 2025-08-26T20:35:49.9782405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9782491Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9782806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9782891Z layer_outputs = layer_module( 2025-08-26T20:35:49.9783130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9783241Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9783549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9783636Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9783952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9784046Z self_outputs = self.self( 2025-08-26T20:35:49.9784311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9784386Z return func(*args, **kwargs) 2025-08-26T20:35:49.9784695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9784790Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9784793Z 2025-08-26T20:35:49.9784881Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9784974Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9785084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9785302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9785371Z return mod(**inputs) 2025-08-26T20:35:49.9785679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9785758Z outputs = self.bert( 2025-08-26T20:35:49.9786089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9786175Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9786481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9786560Z layer_outputs = layer_module( 2025-08-26T20:35:49.9786807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9786889Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9787205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9787291Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9787596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9787746Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9788052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9788149Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9788153Z 2025-08-26T20:35:49.9788262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9788482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9788570Z return mod(**inputs) 2025-08-26T20:35:49.9788880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9788960Z outputs = self.bert( 2025-08-26T20:35:49.9789266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9789353Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9789659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9789734Z layer_outputs = layer_module( 2025-08-26T20:35:49.9789994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9790079Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9790394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9790483Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9790771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9790872Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9791210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9791332Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9791641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9791736Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9791740Z 2025-08-26T20:35:49.9791847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9792069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9792139Z return mod(**inputs) 2025-08-26T20:35:49.9792448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9792549Z outputs = self.bert( 2025-08-26T20:35:49.9792858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9792942Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9793252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9793329Z layer_outputs = layer_module( 2025-08-26T20:35:49.9793575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9793661Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9793974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9794067Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9794350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9794443Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9794789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9794915Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9795232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9795382Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9795620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9795699Z return self.act(input) 2025-08-26T20:35:49.9795704Z 2025-08-26T20:35:49.9795827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9796051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9796130Z return mod(**inputs) 2025-08-26T20:35:49.9796843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9796930Z outputs = self.bert( 2025-08-26T20:35:49.9797330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9797414Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9797740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9797822Z layer_outputs = layer_module( 2025-08-26T20:35:49.9798106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9798198Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9798516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9798620Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9798909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9798999Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9799354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9799565Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9799902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9799995Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9800033Z 2025-08-26T20:35:49.9800158Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9800377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9800459Z return mod(**inputs) 2025-08-26T20:35:49.9800785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9800859Z outputs = self.bert( 2025-08-26T20:35:49.9801185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9801267Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9801591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9801671Z layer_outputs = layer_module( 2025-08-26T20:35:49.9801916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9802009Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9802326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9802426Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9802715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9802804Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9803196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9803345Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9803673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9803763Z return input_tensor + hidden_states 2025-08-26T20:35:49.9803767Z 2025-08-26T20:35:49.9803888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9804122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9804196Z return mod(**inputs) 2025-08-26T20:35:49.9804523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9804599Z outputs = self.bert( 2025-08-26T20:35:49.9804924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9805024Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9805346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9805427Z layer_outputs = layer_module( 2025-08-26T20:35:49.9805667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9805763Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9806077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9806174Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9806490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9806567Z self_outputs = self.self( 2025-08-26T20:35:49.9806844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9806923Z return func(*args, **kwargs) 2025-08-26T20:35:49.9807268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9807360Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9807364Z 2025-08-26T20:35:49.9807483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9807705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9807776Z return mod(**inputs) 2025-08-26T20:35:49.9808106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9808178Z outputs = self.bert( 2025-08-26T20:35:49.9808498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9808580Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9808895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9808978Z layer_outputs = layer_module( 2025-08-26T20:35:49.9809216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9809309Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9809616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9809710Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9810039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9810118Z self_outputs = self.self( 2025-08-26T20:35:49.9810387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9810464Z return func(*args, **kwargs) 2025-08-26T20:35:49.9810777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9810860Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9810864Z 2025-08-26T20:35:49.9810993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9811213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9811282Z return mod(**inputs) 2025-08-26T20:35:49.9811606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9811677Z outputs = self.bert( 2025-08-26T20:35:49.9812009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9812090Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9812398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9812482Z layer_outputs = layer_module( 2025-08-26T20:35:49.9812721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9812812Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9813135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9813222Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9813530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9813602Z self_outputs = self.self( 2025-08-26T20:35:49.9813874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9813947Z return func(*args, **kwargs) 2025-08-26T20:35:49.9814238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9814328Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9814332Z 2025-08-26T20:35:49.9814412Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9814502Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9814606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9814813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9814878Z return mod(**inputs) 2025-08-26T20:35:49.9815172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9815249Z outputs = self.bert( 2025-08-26T20:35:49.9815540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9815631Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9815923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9815995Z layer_outputs = layer_module( 2025-08-26T20:35:49.9816223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9816322Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9816625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9816709Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9817011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9817143Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9817453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9817550Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9817553Z 2025-08-26T20:35:49.9817665Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9817883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9817953Z return mod(**inputs) 2025-08-26T20:35:49.9818261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9818356Z outputs = self.bert( 2025-08-26T20:35:49.9818665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9818750Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9819056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9819138Z layer_outputs = layer_module( 2025-08-26T20:35:49.9819361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9819439Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9819737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9819824Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9820092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9820190Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9820506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9820624Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9820942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9821039Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9821043Z 2025-08-26T20:35:49.9821155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9821380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9821448Z return mod(**inputs) 2025-08-26T20:35:49.9821739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9821813Z outputs = self.bert( 2025-08-26T20:35:49.9822100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9822178Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9822467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9822538Z layer_outputs = layer_module( 2025-08-26T20:35:49.9822788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9822868Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9823164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9823247Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9823519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9823597Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9823931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9824045Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9824338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9824460Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9824677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9824764Z return self.act(input) 2025-08-26T20:35:49.9824768Z 2025-08-26T20:35:49.9824880Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9825080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9825152Z return mod(**inputs) 2025-08-26T20:35:49.9825446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9825521Z outputs = self.bert( 2025-08-26T20:35:49.9825811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9825886Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9826183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9826259Z layer_outputs = layer_module( 2025-08-26T20:35:49.9826487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9826596Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9826891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9826988Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9827259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9827347Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9827675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9827823Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9828133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9828227Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9828231Z 2025-08-26T20:35:49.9828354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9828574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9828657Z return mod(**inputs) 2025-08-26T20:35:49.9828973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9829049Z outputs = self.bert( 2025-08-26T20:35:49.9829395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9829472Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9829770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9829844Z layer_outputs = layer_module( 2025-08-26T20:35:49.9830074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9830154Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9830464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9830562Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9830868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9830949Z self_outputs = self.self( 2025-08-26T20:35:49.9831208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9831301Z return func(*args, **kwargs) 2025-08-26T20:35:49.9831616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9831704Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9831708Z 2025-08-26T20:35:49.9831824Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9832039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9832117Z return mod(**inputs) 2025-08-26T20:35:49.9832424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9832496Z outputs = self.bert( 2025-08-26T20:35:49.9832809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9832886Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9833211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9833305Z layer_outputs = layer_module( 2025-08-26T20:35:49.9833542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9833635Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9833955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9834051Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9834357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9834432Z self_outputs = self.self( 2025-08-26T20:35:49.9834705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9834782Z return func(*args, **kwargs) 2025-08-26T20:35:49.9835097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9835179Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9835183Z 2025-08-26T20:35:49.9835304Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9835515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9835585Z return mod(**inputs) 2025-08-26T20:35:49.9835927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9836000Z outputs = self.bert( 2025-08-26T20:35:49.9836311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9836391Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9836700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9836784Z layer_outputs = layer_module( 2025-08-26T20:35:49.9837039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9837130Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9837435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9837530Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9837839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9837933Z self_outputs = self.self( 2025-08-26T20:35:49.9838200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9838275Z return func(*args, **kwargs) 2025-08-26T20:35:49.9838586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9838672Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9838677Z 2025-08-26T20:35:49.9838764Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9838856Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9838965Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9839184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9839256Z return mod(**inputs) 2025-08-26T20:35:49.9839675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9839763Z outputs = self.bert( 2025-08-26T20:35:49.9840107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9840197Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9840527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9840613Z layer_outputs = layer_module( 2025-08-26T20:35:49.9840864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9840949Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9841264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9841352Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9841676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9841817Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9842125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9842223Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9842227Z 2025-08-26T20:35:49.9842341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9842561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9842655Z return mod(**inputs) 2025-08-26T20:35:49.9842973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9843046Z outputs = self.bert( 2025-08-26T20:35:49.9843352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9843445Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9843751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9843882Z layer_outputs = layer_module( 2025-08-26T20:35:49.9844129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9844212Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9844535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9844620Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9844910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9844991Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9845323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9845430Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9845719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9845812Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9845816Z 2025-08-26T20:35:49.9845918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9846128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9846194Z return mod(**inputs) 2025-08-26T20:35:49.9846488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9846583Z outputs = self.bert( 2025-08-26T20:35:49.9846869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9846950Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9847242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9847322Z layer_outputs = layer_module( 2025-08-26T20:35:49.9847545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9847625Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9847922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9848008Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9848280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9848356Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9848678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9848792Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9849082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9849227Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9849457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9849542Z return self.act(input) 2025-08-26T20:35:49.9849546Z 2025-08-26T20:35:49.9849655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9849872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9849949Z return mod(**inputs) 2025-08-26T20:35:49.9850241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9850337Z outputs = self.bert( 2025-08-26T20:35:49.9850630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9850704Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9851002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9851075Z layer_outputs = layer_module( 2025-08-26T20:35:49.9851336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9851417Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9851721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9851806Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9852074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9852157Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9852487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9852631Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9852927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9853027Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9853039Z 2025-08-26T20:35:49.9853146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9853350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9853426Z return mod(**inputs) 2025-08-26T20:35:49.9853719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9853794Z outputs = self.bert( 2025-08-26T20:35:49.9854089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9854163Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9854459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9854534Z layer_outputs = layer_module( 2025-08-26T20:35:49.9854766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9854844Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9855136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9855226Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9855487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9855585Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9855904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9856044Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9856335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9856414Z return input_tensor + hidden_states 2025-08-26T20:35:49.9856418Z 2025-08-26T20:35:49.9856530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9856753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9856827Z return mod(**inputs) 2025-08-26T20:35:49.9857122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9857192Z outputs = self.bert( 2025-08-26T20:35:49.9857489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9857580Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9857876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9857949Z layer_outputs = layer_module( 2025-08-26T20:35:49.9858180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9858260Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9858552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9858643Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9858934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9859013Z self_outputs = self.self( 2025-08-26T20:35:49.9859259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9859348Z return func(*args, **kwargs) 2025-08-26T20:35:49.9859647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9859730Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9859734Z 2025-08-26T20:35:49.9859849Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9860051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9860125Z return mod(**inputs) 2025-08-26T20:35:49.9860420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9860488Z outputs = self.bert( 2025-08-26T20:35:49.9860786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9860865Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9861180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9861257Z layer_outputs = layer_module( 2025-08-26T20:35:49.9861506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9861594Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9861887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9861994Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9862287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9862368Z self_outputs = self.self( 2025-08-26T20:35:49.9862620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9862697Z return func(*args, **kwargs) 2025-08-26T20:35:49.9863011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9863096Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9863119Z 2025-08-26T20:35:49.9863240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9863460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9863527Z return mod(**inputs) 2025-08-26T20:35:49.9863830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9863914Z outputs = self.bert( 2025-08-26T20:35:49.9864214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9864289Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9864583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9864664Z layer_outputs = layer_module( 2025-08-26T20:35:49.9864892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9864978Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9865279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9865367Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9865661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9865734Z self_outputs = self.self( 2025-08-26T20:35:49.9866010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9866081Z return func(*args, **kwargs) 2025-08-26T20:35:49.9866389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9866474Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9866478Z 2025-08-26T20:35:49.9866564Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9866654Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9866768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9866994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9867061Z return mod(**inputs) 2025-08-26T20:35:49.9867364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9867432Z outputs = self.bert( 2025-08-26T20:35:49.9867722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9867804Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9868093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9868173Z layer_outputs = layer_module( 2025-08-26T20:35:49.9868424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9868508Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9868833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9868916Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9869219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9869351Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9869667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9869754Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9869758Z 2025-08-26T20:35:49.9869863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9870074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9870141Z return mod(**inputs) 2025-08-26T20:35:49.9870443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9870530Z outputs = self.bert( 2025-08-26T20:35:49.9870821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9870904Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9871191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9871269Z layer_outputs = layer_module( 2025-08-26T20:35:49.9871492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9871569Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9871864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9871949Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9872219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9872314Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9872637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9872744Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9873034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9873123Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9873127Z 2025-08-26T20:35:49.9873231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9873438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9873505Z return mod(**inputs) 2025-08-26T20:35:49.9873805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9873873Z outputs = self.bert( 2025-08-26T20:35:49.9874160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9874242Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9874532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9874614Z layer_outputs = layer_module( 2025-08-26T20:35:49.9874866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9874952Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9875268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9875358Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9875641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9875722Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9876083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9876204Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9876509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9876640Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9876865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9876967Z return self.act(input) 2025-08-26T20:35:49.9876973Z 2025-08-26T20:35:49.9877084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9877295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9877373Z return mod(**inputs) 2025-08-26T20:35:49.9877682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9877762Z outputs = self.bert( 2025-08-26T20:35:49.9878068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9878147Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9878460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9878539Z layer_outputs = layer_module( 2025-08-26T20:35:49.9878780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9878883Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9879195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9879284Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9879636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9879736Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9880100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9880253Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9880561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9880660Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9880665Z 2025-08-26T20:35:49.9880783Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9881002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9881099Z return mod(**inputs) 2025-08-26T20:35:49.9881413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9881498Z outputs = self.bert( 2025-08-26T20:35:49.9881833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9881921Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9882244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9882327Z layer_outputs = layer_module( 2025-08-26T20:35:49.9882580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9882667Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9882997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9883095Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9883400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9883484Z self_outputs = self.self( 2025-08-26T20:35:49.9883747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9883849Z return func(*args, **kwargs) 2025-08-26T20:35:49.9884160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9884249Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9884253Z 2025-08-26T20:35:49.9884374Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9884590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9884670Z return mod(**inputs) 2025-08-26T20:35:49.9884986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9885059Z outputs = self.bert( 2025-08-26T20:35:49.9885396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9885477Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9885824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9885901Z layer_outputs = layer_module( 2025-08-26T20:35:49.9886143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9886229Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9886610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9886717Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9887050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9887134Z self_outputs = self.self( 2025-08-26T20:35:49.9887397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9887475Z return func(*args, **kwargs) 2025-08-26T20:35:49.9887813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9887898Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9887902Z 2025-08-26T20:35:49.9888022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9888239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9888315Z return mod(**inputs) 2025-08-26T20:35:49.9889574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9889670Z outputs = self.bert( 2025-08-26T20:35:49.9890004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9890086Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9890428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9890504Z layer_outputs = layer_module( 2025-08-26T20:35:49.9890762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9890854Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9891186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9891278Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9891607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9891709Z self_outputs = self.self( 2025-08-26T20:35:49.9891988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9892066Z return func(*args, **kwargs) 2025-08-26T20:35:49.9892400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9892486Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9892490Z 2025-08-26T20:35:49.9892601Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9892686Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9892796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9893019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9893089Z return mod(**inputs) 2025-08-26T20:35:49.9893429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9893520Z outputs = self.bert( 2025-08-26T20:35:49.9893856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9893942Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9894274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9894358Z layer_outputs = layer_module( 2025-08-26T20:35:49.9894595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9894685Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9894993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9895082Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9895397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9895539Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9895852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9895944Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9895948Z 2025-08-26T20:35:49.9896059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9896599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9896716Z return mod(**inputs) 2025-08-26T20:35:49.9897090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9897166Z outputs = self.bert( 2025-08-26T20:35:49.9897484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9897567Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9897877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9897992Z layer_outputs = layer_module( 2025-08-26T20:35:49.9898229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9898324Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9898634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9898726Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9899044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9899130Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9899476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9899592Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9899906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9899994Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9900001Z 2025-08-26T20:35:49.9900114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9900336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9900411Z return mod(**inputs) 2025-08-26T20:35:49.9900747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9900850Z outputs = self.bert( 2025-08-26T20:35:49.9901170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9901260Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9901575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9901661Z layer_outputs = layer_module( 2025-08-26T20:35:49.9901906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9902000Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9902314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9902405Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9902703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9902787Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9903143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9903257Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9903571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9903743Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9903961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9904043Z return self.act(input) 2025-08-26T20:35:49.9904047Z 2025-08-26T20:35:49.9904153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9904363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9904431Z return mod(**inputs) 2025-08-26T20:35:49.9904738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9904816Z outputs = self.bert( 2025-08-26T20:35:49.9905104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9905189Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9905481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9905571Z layer_outputs = layer_module( 2025-08-26T20:35:49.9905803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9905883Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9906182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9906265Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9906537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9906615Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9906940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9907085Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9907375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9907481Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9907485Z 2025-08-26T20:35:49.9907597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9907792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9907864Z return mod(**inputs) 2025-08-26T20:35:49.9908154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9908226Z outputs = self.bert( 2025-08-26T20:35:49.9908505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9908584Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9908874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9908946Z layer_outputs = layer_module( 2025-08-26T20:35:49.9909177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9909255Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9909555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9909638Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9909902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9910004Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9910343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9910495Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9910803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9910895Z return input_tensor + hidden_states 2025-08-26T20:35:49.9910899Z 2025-08-26T20:35:49.9911009Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9911235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9911324Z return mod(**inputs) 2025-08-26T20:35:49.9911611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9911684Z outputs = self.bert( 2025-08-26T20:35:49.9911967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9912057Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9912347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9912418Z layer_outputs = layer_module( 2025-08-26T20:35:49.9912646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9912728Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9913023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9913108Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9913412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9913499Z self_outputs = self.self( 2025-08-26T20:35:49.9913763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9913882Z return func(*args, **kwargs) 2025-08-26T20:35:49.9914188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9914276Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9914280Z 2025-08-26T20:35:49.9914399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9914613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9914690Z return mod(**inputs) 2025-08-26T20:35:49.9915004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9915082Z outputs = self.bert( 2025-08-26T20:35:49.9915391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9915472Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9915789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9915866Z layer_outputs = layer_module( 2025-08-26T20:35:49.9916111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9916194Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9916505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9916620Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9916928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9917016Z self_outputs = self.self( 2025-08-26T20:35:49.9917274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9917353Z return func(*args, **kwargs) 2025-08-26T20:35:49.9917669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9917769Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9917774Z 2025-08-26T20:35:49.9917893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9918106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9918186Z return mod(**inputs) 2025-08-26T20:35:49.9918496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9918585Z outputs = self.bert( 2025-08-26T20:35:49.9918898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9918977Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9919291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9919368Z layer_outputs = layer_module( 2025-08-26T20:35:49.9919673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9919771Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9920090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9920188Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9920511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9920618Z self_outputs = self.self( 2025-08-26T20:35:49.9920888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9920967Z return func(*args, **kwargs) 2025-08-26T20:35:49.9921300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9921386Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9921390Z 2025-08-26T20:35:49.9921485Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9921571Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9921685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9921906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9921978Z return mod(**inputs) 2025-08-26T20:35:49.9922294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9922367Z outputs = self.bert( 2025-08-26T20:35:49.9922675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9922763Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9923067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9923154Z layer_outputs = layer_module( 2025-08-26T20:35:49.9923410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9923502Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9923810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9923899Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9924212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9924352Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9924680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9924772Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9924776Z 2025-08-26T20:35:49.9924890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9925103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9925172Z return mod(**inputs) 2025-08-26T20:35:49.9925507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9925580Z outputs = self.bert( 2025-08-26T20:35:49.9925895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9925972Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9926282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9926367Z layer_outputs = layer_module( 2025-08-26T20:35:49.9926605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9926696Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9927003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9927094Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9927381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9927494Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9927839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9927954Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9928271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9928362Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9928365Z 2025-08-26T20:35:49.9928474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9928698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9928770Z return mod(**inputs) 2025-08-26T20:35:49.9929088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9929159Z outputs = self.bert( 2025-08-26T20:35:49.9929467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9929555Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9929864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9929949Z layer_outputs = layer_module( 2025-08-26T20:35:49.9930205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9930298Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9930605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9930696Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9930989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9931071Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9931432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9931548Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9931856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9931988Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9932237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9932323Z return self.act(input) 2025-08-26T20:35:49.9932327Z 2025-08-26T20:35:49.9932436Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9932655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9932726Z return mod(**inputs) 2025-08-26T20:35:49.9933042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9933117Z outputs = self.bert( 2025-08-26T20:35:49.9933410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9933490Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9933783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9933857Z layer_outputs = layer_module( 2025-08-26T20:35:49.9934107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9934191Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9934487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9934572Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9934843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9934920Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9935242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9935386Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9935676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9935767Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9935771Z 2025-08-26T20:35:49.9935875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9936078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9936150Z return mod(**inputs) 2025-08-26T20:35:49.9936442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9936534Z outputs = self.bert( 2025-08-26T20:35:49.9936826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9936909Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9937200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9937274Z layer_outputs = layer_module( 2025-08-26T20:35:49.9937505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9937583Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9937893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9937978Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9938271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9938350Z self_outputs = self.self( 2025-08-26T20:35:49.9938614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9938695Z return func(*args, **kwargs) 2025-08-26T20:35:49.9938983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9939072Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9939076Z 2025-08-26T20:35:49.9939181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9939383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9939456Z return mod(**inputs) 2025-08-26T20:35:49.9939748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9939821Z outputs = self.bert( 2025-08-26T20:35:49.9940112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9940187Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9940515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9940591Z layer_outputs = layer_module( 2025-08-26T20:35:49.9940831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9940926Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9941221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9941304Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9941590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9941671Z self_outputs = self.self( 2025-08-26T20:35:49.9941917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9941994Z return func(*args, **kwargs) 2025-08-26T20:35:49.9942283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9942363Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9942367Z 2025-08-26T20:35:49.9942481Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9942681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9942753Z return mod(**inputs) 2025-08-26T20:35:49.9943059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9943134Z outputs = self.bert( 2025-08-26T20:35:49.9943426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9943501Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9943800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9943871Z layer_outputs = layer_module( 2025-08-26T20:35:49.9944117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9944197Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9944489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9944579Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9944874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9944966Z self_outputs = self.self( 2025-08-26T20:35:49.9945210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9945280Z return func(*args, **kwargs) 2025-08-26T20:35:49.9945573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9945655Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9945659Z 2025-08-26T20:35:49.9945747Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9945826Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9945938Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9946132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9946197Z return mod(**inputs) 2025-08-26T20:35:49.9946494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9946575Z outputs = self.bert( 2025-08-26T20:35:49.9946868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9946941Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9947232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9947311Z layer_outputs = layer_module( 2025-08-26T20:35:49.9947536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9947623Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9947914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9947998Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9948296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9948438Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9948737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9948821Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9948824Z 2025-08-26T20:35:49.9948943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9949200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9949273Z return mod(**inputs) 2025-08-26T20:35:49.9949601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9949672Z outputs = self.bert( 2025-08-26T20:35:49.9950001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9950080Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9950425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9950504Z layer_outputs = layer_module( 2025-08-26T20:35:49.9950745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9950845Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9951137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9951255Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9951522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9951602Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9951934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9952042Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9952345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9952428Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9952431Z 2025-08-26T20:35:49.9952545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9952754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9952827Z return mod(**inputs) 2025-08-26T20:35:49.9953147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9953242Z outputs = self.bert( 2025-08-26T20:35:49.9953571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9953651Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9953962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9954048Z layer_outputs = layer_module( 2025-08-26T20:35:49.9954287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9954378Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9954698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9954797Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9955077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9955159Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9955513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9955624Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9955961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9956085Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9956315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9956404Z return self.act(input) 2025-08-26T20:35:49.9956409Z 2025-08-26T20:35:49.9956520Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9956746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9956820Z return mod(**inputs) 2025-08-26T20:35:49.9957157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9957229Z outputs = self.bert( 2025-08-26T20:35:49.9957552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9957638Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9957959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9958059Z layer_outputs = layer_module( 2025-08-26T20:35:49.9958301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9958386Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9958707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9958801Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9959102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9959185Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9959631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9959790Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9960124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9960246Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9960250Z 2025-08-26T20:35:49.9960365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9960596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9960671Z return mod(**inputs) 2025-08-26T20:35:49.9960978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9961055Z outputs = self.bert( 2025-08-26T20:35:49.9961357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9961438Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9961721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9961803Z layer_outputs = layer_module( 2025-08-26T20:35:49.9962046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9962135Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9962465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9962557Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9962870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9962958Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9963306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9963460Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9963777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:49.9963870Z return input_tensor + hidden_states 2025-08-26T20:35:49.9963875Z 2025-08-26T20:35:49.9963988Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9964237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9964313Z return mod(**inputs) 2025-08-26T20:35:49.9964638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9964721Z outputs = self.bert( 2025-08-26T20:35:49.9965037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9965145Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9965459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9965537Z layer_outputs = layer_module( 2025-08-26T20:35:49.9965789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9965878Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9966200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9966292Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9966615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9966695Z self_outputs = self.self( 2025-08-26T20:35:49.9966963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9967066Z return func(*args, **kwargs) 2025-08-26T20:35:49.9967384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9967483Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9967490Z 2025-08-26T20:35:49.9967604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9967823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9967903Z return mod(**inputs) 2025-08-26T20:35:49.9968232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9968312Z outputs = self.bert( 2025-08-26T20:35:49.9968634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9968716Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9969043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9969121Z layer_outputs = layer_module( 2025-08-26T20:35:49.9969372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9969458Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9969800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9969891Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9970208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9970289Z self_outputs = self.self( 2025-08-26T20:35:49.9970536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9970616Z return func(*args, **kwargs) 2025-08-26T20:35:49.9970917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9971012Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9971023Z 2025-08-26T20:35:49.9971126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9971325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9971396Z return mod(**inputs) 2025-08-26T20:35:49.9971682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9971771Z outputs = self.bert( 2025-08-26T20:35:49.9972052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9972125Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9972413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9972486Z layer_outputs = layer_module( 2025-08-26T20:35:49.9972712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9972790Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9973079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9973169Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9973459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9973556Z self_outputs = self.self( 2025-08-26T20:35:49.9973801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9973879Z return func(*args, **kwargs) 2025-08-26T20:35:49.9974182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9974269Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9974272Z 2025-08-26T20:35:49.9974366Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9974452Z cudagraph partition due to non gpu ops 2025-08-26T20:35:49.9974569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9974781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9974851Z return mod(**inputs) 2025-08-26T20:35:49.9975168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9975238Z outputs = self.bert( 2025-08-26T20:35:49.9975551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9975632Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9975941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9976026Z layer_outputs = layer_module( 2025-08-26T20:35:49.9976276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9976380Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9976670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9976759Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9977050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:49.9977188Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:49.9977536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:49.9977629Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9977632Z 2025-08-26T20:35:49.9977752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9977965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9978035Z return mod(**inputs) 2025-08-26T20:35:49.9978373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9978445Z outputs = self.bert( 2025-08-26T20:35:49.9978761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9978840Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9979158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9979233Z layer_outputs = layer_module( 2025-08-26T20:35:49.9979471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9979564Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9979874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9979974Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9980272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9980353Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9980703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9980816Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9981128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:49.9981213Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9981217Z 2025-08-26T20:35:49.9981329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9981534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9981603Z return mod(**inputs) 2025-08-26T20:35:49.9981924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9981995Z outputs = self.bert( 2025-08-26T20:35:49.9982307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9982385Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9982690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9982790Z layer_outputs = layer_module( 2025-08-26T20:35:49.9983031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9983127Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9983447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9983548Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9983837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9983923Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9984305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:49.9984418Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:49.9984740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:49.9984862Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:49.9985107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:49.9985192Z return self.act(input) 2025-08-26T20:35:49.9985195Z 2025-08-26T20:35:49.9985305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9985526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9985594Z return mod(**inputs) 2025-08-26T20:35:49.9985914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9985984Z outputs = self.bert( 2025-08-26T20:35:49.9986292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9986378Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9986691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9986778Z layer_outputs = layer_module( 2025-08-26T20:35:49.9987045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9987134Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9987467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:49.9987559Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:49.9987848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:49.9987936Z return forward_fn(*input_tensors) 2025-08-26T20:35:49.9988292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:49.9988439Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:49.9988754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:49.9988852Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:49.9988856Z 2025-08-26T20:35:49.9988970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9989194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9989269Z return mod(**inputs) 2025-08-26T20:35:49.9989588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9989686Z outputs = self.bert( 2025-08-26T20:35:49.9989995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9990081Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9990391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9990474Z layer_outputs = layer_module( 2025-08-26T20:35:49.9990712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9990811Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9991128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9991215Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9991530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9991608Z self_outputs = self.self( 2025-08-26T20:35:49.9991887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9991972Z return func(*args, **kwargs) 2025-08-26T20:35:49.9992278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:49.9992372Z query_layer = self.query(hidden_states) 2025-08-26T20:35:49.9992376Z 2025-08-26T20:35:49.9992488Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9992706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9992778Z return mod(**inputs) 2025-08-26T20:35:49.9993089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9993168Z outputs = self.bert( 2025-08-26T20:35:49.9993470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9993575Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9993887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9993963Z layer_outputs = layer_module( 2025-08-26T20:35:49.9994216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9994298Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9994620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9994708Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9995022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9995105Z self_outputs = self.self( 2025-08-26T20:35:49.9995374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9995459Z return func(*args, **kwargs) 2025-08-26T20:35:49.9995774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:49.9995869Z key_layer = self.key(current_states) 2025-08-26T20:35:49.9995873Z 2025-08-26T20:35:49.9995984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:49.9996418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:49.9996539Z return mod(**inputs) 2025-08-26T20:35:49.9997020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:49.9997105Z outputs = self.bert( 2025-08-26T20:35:49.9997416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:49.9997503Z encoder_outputs = self.encoder( 2025-08-26T20:35:49.9997830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:49.9997909Z layer_outputs = layer_module( 2025-08-26T20:35:49.9998187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:49.9998278Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:49.9998610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:49.9998704Z self_attention_outputs = self.attention( 2025-08-26T20:35:49.9999026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:49.9999145Z self_outputs = self.self( 2025-08-26T20:35:49.9999420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:49.9999553Z return func(*args, **kwargs) 2025-08-26T20:35:49.9999877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:49.9999966Z value_layer = self.value(current_states) 2025-08-26T20:35:49.9999979Z 2025-08-26T20:35:50.0000070Z cudagraph partition due to non gpu ops 2025-08-26T20:35:50.0000158Z cudagraph partition due to non gpu ops 2025-08-26T20:35:50.0000283Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0000503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0000576Z return mod(**inputs) 2025-08-26T20:35:50.0000913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0001021Z outputs = self.bert( 2025-08-26T20:35:50.0001337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0001417Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0001743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0001821Z layer_outputs = layer_module( 2025-08-26T20:35:50.0002060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0002152Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0002456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0002550Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0002858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:50.0002997Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:50.0003330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:50.0003418Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0003422Z 2025-08-26T20:35:50.0003539Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0003780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0003861Z return mod(**inputs) 2025-08-26T20:35:50.0004170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0004252Z outputs = self.bert( 2025-08-26T20:35:50.0004544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0004616Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0004924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0004996Z layer_outputs = layer_module( 2025-08-26T20:35:50.0005212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0005296Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0005578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0005691Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0005948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0006032Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0006342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:50.0006447Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:50.0006734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:50.0006814Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0006819Z 2025-08-26T20:35:50.0006925Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0007119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0007185Z return mod(**inputs) 2025-08-26T20:35:50.0007482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0007570Z outputs = self.bert( 2025-08-26T20:35:50.0007868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0007942Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0008238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0008310Z layer_outputs = layer_module( 2025-08-26T20:35:50.0008537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0008626Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0008917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0009007Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0009272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0009349Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0009678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:50.0009783Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:50.0010100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:50.0010217Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:50.0010439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:50.0010513Z return self.act(input) 2025-08-26T20:35:50.0010518Z 2025-08-26T20:35:50.0010622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0010829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0010895Z return mod(**inputs) 2025-08-26T20:35:50.0011209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0011279Z outputs = self.bert( 2025-08-26T20:35:50.0011573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0011654Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0011945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0012054Z layer_outputs = layer_module( 2025-08-26T20:35:50.0012281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0012367Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0012657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0012743Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0013017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0013096Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0013423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:50.0013562Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:50.0013855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:50.0013962Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0013966Z 2025-08-26T20:35:50.0014070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0014283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0014349Z return mod(**inputs) 2025-08-26T20:35:50.0014653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0014722Z outputs = self.bert( 2025-08-26T20:35:50.0015017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0015104Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0015399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0015481Z layer_outputs = layer_module( 2025-08-26T20:35:50.0015720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0015799Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0016099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0016183Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0016474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0016556Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0016908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:50.0017054Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:50.0017363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:50.0017454Z return input_tensor + hidden_states 2025-08-26T20:35:50.0017459Z 2025-08-26T20:35:50.0017583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0017813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0017879Z return mod(**inputs) 2025-08-26T20:35:50.0018174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0018248Z outputs = self.bert( 2025-08-26T20:35:50.0018539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0018640Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0018930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0019009Z layer_outputs = layer_module( 2025-08-26T20:35:50.0019236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0019315Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0019611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0019694Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0019992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:50.0020066Z self_outputs = self.self( 2025-08-26T20:35:50.0020311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:50.0020408Z return func(*args, **kwargs) 2025-08-26T20:35:50.0020699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:50.0020790Z query_layer = self.query(hidden_states) 2025-08-26T20:35:50.0020794Z 2025-08-26T20:35:50.0020899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0021112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0021181Z return mod(**inputs) 2025-08-26T20:35:50.0021473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0021550Z outputs = self.bert( 2025-08-26T20:35:50.0021840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0021923Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0022212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0022286Z layer_outputs = layer_module( 2025-08-26T20:35:50.0022519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0022598Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0022911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0022995Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0023296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:50.0023372Z self_outputs = self.self( 2025-08-26T20:35:50.0023622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:50.0023707Z return func(*args, **kwargs) 2025-08-26T20:35:50.0024024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:50.0024114Z key_layer = self.key(current_states) 2025-08-26T20:35:50.0024117Z 2025-08-26T20:35:50.0024220Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0024420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0024493Z return mod(**inputs) 2025-08-26T20:35:50.0024787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0024878Z outputs = self.bert( 2025-08-26T20:35:50.0025172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0025253Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0025540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0025613Z layer_outputs = layer_module( 2025-08-26T20:35:50.0025845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0025922Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0026221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0026303Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0026591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:50.0026697Z self_outputs = self.self( 2025-08-26T20:35:50.0026939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:50.0027015Z return func(*args, **kwargs) 2025-08-26T20:35:50.0027307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:50.0027388Z value_layer = self.value(current_states) 2025-08-26T20:35:50.0027398Z 2025-08-26T20:35:50.0027482Z cudagraph partition due to non gpu ops 2025-08-26T20:35:50.0027565Z cudagraph partition due to non gpu ops 2025-08-26T20:35:50.0027678Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0027879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0027954Z return mod(**inputs) 2025-08-26T20:35:50.0028248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0028316Z outputs = self.bert( 2025-08-26T20:35:50.0028614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0028690Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0028987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0029059Z layer_outputs = layer_module( 2025-08-26T20:35:50.0029299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0029389Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0029685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0029777Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0030070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:50.0030209Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:50.0030520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:50.0030605Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0030609Z 2025-08-26T20:35:50.0030721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0030927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0031019Z return mod(**inputs) 2025-08-26T20:35:50.0031314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0031383Z outputs = self.bert( 2025-08-26T20:35:50.0031680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0031753Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0032066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0032143Z layer_outputs = layer_module( 2025-08-26T20:35:50.0032390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0032473Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0032787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0033309Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0033782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0034219Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0034711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:50.0035236Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:50.0035724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:50.0036194Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0036350Z 2025-08-26T20:35:50.0036480Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0036890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0037251Z return mod(**inputs) 2025-08-26T20:35:50.0037708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0038187Z outputs = self.bert( 2025-08-26T20:35:50.0038638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0039134Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0039678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0040177Z layer_outputs = layer_module( 2025-08-26T20:35:50.0040573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0040987Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0041457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0041944Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0042396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0042865Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0043369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:50.0043911Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:50.0044419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:50.0044959Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:50.0045393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:50.0045777Z return self.act(input) 2025-08-26T20:35:50.0045903Z 2025-08-26T20:35:50.0046022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0046433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0046797Z return mod(**inputs) 2025-08-26T20:35:50.0047251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0047710Z outputs = self.bert( 2025-08-26T20:35:50.0048152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0048609Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0049062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0049545Z layer_outputs = layer_module( 2025-08-26T20:35:50.0049928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0050350Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0050825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0051295Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0051729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0052151Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0052649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:50.0053213Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:50.0053729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:50.0054200Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0054353Z 2025-08-26T20:35:50.0054467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0054861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0055213Z return mod(**inputs) 2025-08-26T20:35:50.0055666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0056108Z outputs = self.bert( 2025-08-26T20:35:50.0056532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0056984Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0057435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0057885Z layer_outputs = layer_module( 2025-08-26T20:35:50.0058256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0058672Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0059127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0059553Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0060029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:50.0060512Z self_outputs = self.self( 2025-08-26T20:35:50.0060925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:50.0061351Z return func(*args, **kwargs) 2025-08-26T20:35:50.0061823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:35:50.0062285Z query_layer = self.query(hidden_states) 2025-08-26T20:35:50.0062443Z 2025-08-26T20:35:50.0062560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0062957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0063304Z return mod(**inputs) 2025-08-26T20:35:50.0063718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0064144Z outputs = self.bert( 2025-08-26T20:35:50.0064583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0065073Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0065529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0065994Z layer_outputs = layer_module( 2025-08-26T20:35:50.0066367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0066764Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0067226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0067683Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0068120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:50.0068545Z self_outputs = self.self( 2025-08-26T20:35:50.0068923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:50.0069330Z return func(*args, **kwargs) 2025-08-26T20:35:50.0069783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:35:50.0070258Z key_layer = self.key(current_states) 2025-08-26T20:35:50.0070412Z 2025-08-26T20:35:50.0070527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0070915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0071294Z return mod(**inputs) 2025-08-26T20:35:50.0071737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0072162Z outputs = self.bert( 2025-08-26T20:35:50.0072568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0073023Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0073477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0073958Z layer_outputs = layer_module( 2025-08-26T20:35:50.0074334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0074730Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0075205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0075691Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0076200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:35:50.0076681Z self_outputs = self.self( 2025-08-26T20:35:50.0077072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:35:50.0077477Z return func(*args, **kwargs) 2025-08-26T20:35:50.0077924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:35:50.0078386Z value_layer = self.value(current_states) 2025-08-26T20:35:50.0078547Z 2025-08-26T20:35:50.0078640Z cudagraph partition due to non gpu ops 2025-08-26T20:35:50.0078880Z cudagraph partition due to non gpu ops 2025-08-26T20:35:50.0079150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0079643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0080040Z return mod(**inputs) 2025-08-26T20:35:50.0080500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0081047Z outputs = self.bert( 2025-08-26T20:35:50.0081493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0081945Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0082402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0082870Z layer_outputs = layer_module( 2025-08-26T20:35:50.0083256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0083655Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0084123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:35:50.0084591Z self_attention_outputs = self.attention( 2025-08-26T20:35:50.0085048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:35:50.0085568Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:35:50.0086095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:35:50.0086564Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0086715Z 2025-08-26T20:35:50.0086835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0087245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0087603Z return mod(**inputs) 2025-08-26T20:35:50.0088055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0088513Z outputs = self.bert( 2025-08-26T20:35:50.0088942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0089405Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0089884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0090340Z layer_outputs = layer_module( 2025-08-26T20:35:50.0090695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0091057Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0091490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0091954Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0092375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0092778Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0093231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:50.0093734Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:50.0094198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:35:50.0094641Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0094781Z 2025-08-26T20:35:50.0094894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0095270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0095633Z return mod(**inputs) 2025-08-26T20:35:50.0096096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0097011Z outputs = self.bert( 2025-08-26T20:35:50.0097467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0097972Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0098434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0098864Z layer_outputs = layer_module( 2025-08-26T20:35:50.0099225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0099585Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0100029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0100470Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0100879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0101283Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0101735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:35:50.0102229Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:35:50.0102754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:35:50.0103231Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:35:50.0103620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:35:50.0103970Z return self.act(input) 2025-08-26T20:35:50.0104096Z 2025-08-26T20:35:50.0104204Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0104576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0104919Z return mod(**inputs) 2025-08-26T20:35:50.0105356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0105791Z outputs = self.bert( 2025-08-26T20:35:50.0106197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0106631Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0107051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0107512Z layer_outputs = layer_module( 2025-08-26T20:35:50.0107875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0108247Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0108683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0109124Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0109539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0109951Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0110411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:50.0110968Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:50.0111484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:35:50.0111953Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0112099Z 2025-08-26T20:35:50.0112206Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0112573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0112904Z return mod(**inputs) 2025-08-26T20:35:50.0113303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-26T20:35:50.0113730Z outputs = self.bert( 2025-08-26T20:35:50.0114130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:35:50.0114564Z encoder_outputs = self.encoder( 2025-08-26T20:35:50.0115021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:35:50.0115469Z layer_outputs = layer_module( 2025-08-26T20:35:50.0115841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:35:50.0116231Z return super().__call__(*args, **kwargs) 2025-08-26T20:35:50.0116691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:35:50.0117156Z layer_output = apply_chunking_to_forward( 2025-08-26T20:35:50.0117614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:35:50.0118050Z return forward_fn(*input_tensors) 2025-08-26T20:35:50.0118537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:35:50.0119082Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:35:50.0119664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:35:50.0120166Z return input_tensor + hidden_states 2025-08-26T20:35:50.0120325Z 2025-08-26T20:35:50.0120475Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0120909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0121279Z return mod(**inputs) 2025-08-26T20:35:50.0121720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-08-26T20:35:50.0122215Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:35:50.0122726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-08-26T20:35:50.0123233Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:35:50.0123739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 640, in forward 2025-08-26T20:35:50.0124211Z hidden_states = self.transform(hidden_states) 2025-08-26T20:35:50.0124704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 615, in forward 2025-08-26T20:35:50.0125170Z hidden_states = self.dense(hidden_states) 2025-08-26T20:35:50.0125321Z 2025-08-26T20:35:50.0125442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0125835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0126177Z return mod(**inputs) 2025-08-26T20:35:50.0126616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-08-26T20:35:50.0127119Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:35:50.0127594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-08-26T20:35:50.0128095Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:35:50.0128577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 641, in forward 2025-08-26T20:35:50.0129048Z hidden_states = self.decoder(hidden_states) 2025-08-26T20:35:50.0129214Z 2025-08-26T20:35:50.0129328Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:35:50.0129725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:35:50.0130057Z return mod(**inputs) 2025-08-26T20:35:50.0130462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1086, in forward 2025-08-26T20:35:50.0130890Z lm_loss = self.loss_function( 2025-08-26T20:35:50.0131268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-26T20:35:50.0131755Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-26T20:35:50.0132247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-26T20:35:50.0132817Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-26T20:35:50.0133092Z 2025-08-26T20:36:01.4638991Z Compilation time (from dynamo_timed): 24.792775681 2025-08-26T20:36:01.4695311Z pass 2025-08-26T20:36:01.4697504Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:36:01.4698545Z TIMING: _recursive_pre_grad_passes:0.01174 _recursive_joint_graph_passes:1.17093 _recursive_post_grad_passes:0.13151 async_compile.wait:0.83526 code_gen:10.08182 inductor_compile:12.49442 backend_compile:19.08945 gc:0.00051 entire_frame_compile:24.79278 total_wall_time:24.79278 2025-08-26T20:36:01.4699873Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:28467 | FakeTensor.__torch_dispatch__:8250 | ProxyTorchDispatchMode.__torch_dispatch__:10946 2025-08-26T20:36:01.4700480Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-08-26T20:36:07.3767533Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:36:07.3768945Z from pkg_resources import resource_filename 2025-08-26T20:36:07.9852076Z 2025-08-26T20:36:11.1577195Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:36:11.1577681Z loading model: 0it [00:03, ?it/s] 2025-08-26T20:36:11.1605377Z cpu eval MegatronBertForQuestionAnswering 2025-08-26T20:36:12.6748818Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:36:13.3354698Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:36:13.9603653Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:36:29.1388552Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1392454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1395830Z return mod(**inputs) 2025-08-26T20:36:29.1396635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1397701Z outputs = self.bert( 2025-08-26T20:36:29.1398166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1398645Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1399143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1399943Z layer_outputs = layer_module( 2025-08-26T20:36:29.1400460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1400892Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1401375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1401866Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1402355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1402828Z self_outputs = self.self( 2025-08-26T20:36:29.1403283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1403698Z return func(*args, **kwargs) 2025-08-26T20:36:29.1404161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1404646Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1404804Z 2025-08-26T20:36:29.1405004Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1405429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1405808Z return mod(**inputs) 2025-08-26T20:36:29.1406261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1406741Z outputs = self.bert( 2025-08-26T20:36:29.1417898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1418446Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1419089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1419598Z layer_outputs = layer_module( 2025-08-26T20:36:29.1420009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1420437Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1420923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1421486Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1421966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1422433Z self_outputs = self.self( 2025-08-26T20:36:29.1422853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1423283Z return func(*args, **kwargs) 2025-08-26T20:36:29.1423744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1424222Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1424373Z 2025-08-26T20:36:29.1424500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1424918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1425292Z return mod(**inputs) 2025-08-26T20:36:29.1425794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1426270Z outputs = self.bert( 2025-08-26T20:36:29.1426715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1427194Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1427663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1428137Z layer_outputs = layer_module( 2025-08-26T20:36:29.1428524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1428924Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1429408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1429888Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1430358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1430814Z self_outputs = self.self( 2025-08-26T20:36:29.1431223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1431655Z return func(*args, **kwargs) 2025-08-26T20:36:29.1432140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1432616Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1432767Z 2025-08-26T20:36:29.1432864Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1433115Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1433385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1433797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1434152Z return mod(**inputs) 2025-08-26T20:36:29.1434603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1435092Z outputs = self.bert( 2025-08-26T20:36:29.1435534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1436008Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1436463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1436961Z layer_outputs = layer_module( 2025-08-26T20:36:29.1437346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1437754Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1438232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1438700Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1439177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1439829Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1440371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1440859Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1441022Z 2025-08-26T20:36:29.1441142Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1441550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1441943Z return mod(**inputs) 2025-08-26T20:36:29.1442394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1442856Z outputs = self.bert( 2025-08-26T20:36:29.1443280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1443741Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1444202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1444654Z layer_outputs = layer_module( 2025-08-26T20:36:29.1445020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1445411Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1445869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1446337Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1446779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1447201Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1447694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1448247Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1448751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1449216Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1449371Z 2025-08-26T20:36:29.1449486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1449885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1450245Z return mod(**inputs) 2025-08-26T20:36:29.1450711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1451152Z outputs = self.bert( 2025-08-26T20:36:29.1451585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1452049Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1452504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1452976Z layer_outputs = layer_module( 2025-08-26T20:36:29.1453343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1453750Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1454221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1454686Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1455122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1455565Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1456073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1456591Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1457097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1457630Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1458048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1458430Z return self.act(input) 2025-08-26T20:36:29.1458566Z 2025-08-26T20:36:29.1458688Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1459106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1459465Z return mod(**inputs) 2025-08-26T20:36:29.1459897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1460350Z outputs = self.bert( 2025-08-26T20:36:29.1460800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1461269Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1461718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1462177Z layer_outputs = layer_module( 2025-08-26T20:36:29.1462560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1462960Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1463437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1463927Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1464373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1464829Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1465322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1465887Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1466448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1466934Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1467092Z 2025-08-26T20:36:29.1467209Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1467612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1467978Z return mod(**inputs) 2025-08-26T20:36:29.1468418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1468909Z outputs = self.bert( 2025-08-26T20:36:29.1469382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1469848Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1470306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1470772Z layer_outputs = layer_module( 2025-08-26T20:36:29.1471158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1471559Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1472038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1472506Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1472987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1473485Z self_outputs = self.self( 2025-08-26T20:36:29.1473894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1474306Z return func(*args, **kwargs) 2025-08-26T20:36:29.1474768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1475248Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1475407Z 2025-08-26T20:36:29.1475530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1475935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1476288Z return mod(**inputs) 2025-08-26T20:36:29.1476732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1477194Z outputs = self.bert( 2025-08-26T20:36:29.1477630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1478102Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1478558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1479023Z layer_outputs = layer_module( 2025-08-26T20:36:29.1479494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1479946Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1480426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1480903Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1481387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1481858Z self_outputs = self.self( 2025-08-26T20:36:29.1482268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1482718Z return func(*args, **kwargs) 2025-08-26T20:36:29.1483184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1483658Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1483809Z 2025-08-26T20:36:29.1483934Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1484343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1484709Z return mod(**inputs) 2025-08-26T20:36:29.1485134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1485578Z outputs = self.bert( 2025-08-26T20:36:29.1485997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1486448Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1486886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1487337Z layer_outputs = layer_module( 2025-08-26T20:36:29.1487711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1488099Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1488555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1489038Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1489518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1489978Z self_outputs = self.self( 2025-08-26T20:36:29.1490385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1490798Z return func(*args, **kwargs) 2025-08-26T20:36:29.1491236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1491700Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1491847Z 2025-08-26T20:36:29.1491943Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1492178Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1492430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1492821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1493199Z return mod(**inputs) 2025-08-26T20:36:29.1493632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1494074Z outputs = self.bert( 2025-08-26T20:36:29.1494510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1494960Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1495433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1495886Z layer_outputs = layer_module( 2025-08-26T20:36:29.1496510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1496939Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1497399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1497865Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1498375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1498883Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1499402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1499863Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1500013Z 2025-08-26T20:36:29.1500176Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1500572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1500931Z return mod(**inputs) 2025-08-26T20:36:29.1501375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1501833Z outputs = self.bert( 2025-08-26T20:36:29.1502270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1502740Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1503195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1503657Z layer_outputs = layer_module( 2025-08-26T20:36:29.1504045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1504444Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1504936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1505401Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1505839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1506270Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1506753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1507270Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1507758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1508222Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1508371Z 2025-08-26T20:36:29.1508493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1508883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1509227Z return mod(**inputs) 2025-08-26T20:36:29.1509654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1510108Z outputs = self.bert( 2025-08-26T20:36:29.1510529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1510979Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1511471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1511926Z layer_outputs = layer_module( 2025-08-26T20:36:29.1512299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1512688Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1513133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1513598Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1514047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1514475Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1514960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1515467Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1515970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1516464Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1516872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1517243Z return self.act(input) 2025-08-26T20:36:29.1517364Z 2025-08-26T20:36:29.1517477Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1517864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1518217Z return mod(**inputs) 2025-08-26T20:36:29.1518647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1519105Z outputs = self.bert( 2025-08-26T20:36:29.1519604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1520084Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1520584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1521052Z layer_outputs = layer_module( 2025-08-26T20:36:29.1521430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1521834Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1522306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1522788Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1523236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1523682Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1524177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1524746Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1525273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1525745Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1525897Z 2025-08-26T20:36:29.1526014Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1526415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1526798Z return mod(**inputs) 2025-08-26T20:36:29.1527244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1527712Z outputs = self.bert( 2025-08-26T20:36:29.1528148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1528618Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1529087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1529570Z layer_outputs = layer_module( 2025-08-26T20:36:29.1529957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1530347Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1530815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1531291Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1531763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1532183Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1532665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1533212Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1533731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.1534189Z return input_tensor + hidden_states 2025-08-26T20:36:29.1534333Z 2025-08-26T20:36:29.1534449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1534838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1535189Z return mod(**inputs) 2025-08-26T20:36:29.1535629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1536101Z outputs = self.bert( 2025-08-26T20:36:29.1536523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1536978Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1537429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1537877Z layer_outputs = layer_module( 2025-08-26T20:36:29.1538245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1538634Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1539088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1539552Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1540015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1540462Z self_outputs = self.self( 2025-08-26T20:36:29.1540859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1541263Z return func(*args, **kwargs) 2025-08-26T20:36:29.1541707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1542171Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1542339Z 2025-08-26T20:36:29.1542457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1542852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1543201Z return mod(**inputs) 2025-08-26T20:36:29.1543633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1544075Z outputs = self.bert( 2025-08-26T20:36:29.1544508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1544980Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1545446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1545892Z layer_outputs = layer_module( 2025-08-26T20:36:29.1546259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1546648Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1547140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1547619Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1548087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1548527Z self_outputs = self.self( 2025-08-26T20:36:29.1548927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1549332Z return func(*args, **kwargs) 2025-08-26T20:36:29.1549777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1550227Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1550378Z 2025-08-26T20:36:29.1550490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1550880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1551257Z return mod(**inputs) 2025-08-26T20:36:29.1551687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1552130Z outputs = self.bert( 2025-08-26T20:36:29.1552557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1553012Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1553459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1553913Z layer_outputs = layer_module( 2025-08-26T20:36:29.1554278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1554672Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1555126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1555593Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1556049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1556502Z self_outputs = self.self( 2025-08-26T20:36:29.1556905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1557315Z return func(*args, **kwargs) 2025-08-26T20:36:29.1557794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1558265Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1558426Z 2025-08-26T20:36:29.1558518Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1558758Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1559024Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1559500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1559876Z return mod(**inputs) 2025-08-26T20:36:29.1560361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1560831Z outputs = self.bert( 2025-08-26T20:36:29.1561252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1561702Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1562169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1562664Z layer_outputs = layer_module( 2025-08-26T20:36:29.1563052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1563456Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1563912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1564382Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1564862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1565385Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1565921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1566392Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1566556Z 2025-08-26T20:36:29.1566674Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1567099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1567457Z return mod(**inputs) 2025-08-26T20:36:29.1567899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1568368Z outputs = self.bert( 2025-08-26T20:36:29.1568802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1569267Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1569730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1570189Z layer_outputs = layer_module( 2025-08-26T20:36:29.1570583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1570972Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1571426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1571887Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1572311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1572734Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1573235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1573756Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1574246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1574705Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1574863Z 2025-08-26T20:36:29.1574976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1575365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1575717Z return mod(**inputs) 2025-08-26T20:36:29.1576170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1576625Z outputs = self.bert( 2025-08-26T20:36:29.1577058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1577508Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1577962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1578428Z layer_outputs = layer_module( 2025-08-26T20:36:29.1578805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1579188Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1579651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1580110Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1580536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1580966Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1581450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1581971Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1582459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1582966Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1583376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1583746Z return self.act(input) 2025-08-26T20:36:29.1583868Z 2025-08-26T20:36:29.1583989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1584376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1584730Z return mod(**inputs) 2025-08-26T20:36:29.1585160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1585611Z outputs = self.bert( 2025-08-26T20:36:29.1586035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1586488Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1586946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1587398Z layer_outputs = layer_module( 2025-08-26T20:36:29.1587774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1588164Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1588654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1589124Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1589555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1589983Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1590464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1591010Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1591571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1592032Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1592183Z 2025-08-26T20:36:29.1592305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1592700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1593058Z return mod(**inputs) 2025-08-26T20:36:29.1593509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1594000Z outputs = self.bert( 2025-08-26T20:36:29.1594448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1594906Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1595367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1595827Z layer_outputs = layer_module( 2025-08-26T20:36:29.1596361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1596777Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1597243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1597726Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1598206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1598729Z self_outputs = self.self( 2025-08-26T20:36:29.1599126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1599600Z return func(*args, **kwargs) 2025-08-26T20:36:29.1600069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1600549Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1600703Z 2025-08-26T20:36:29.1600828Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1601222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1601589Z return mod(**inputs) 2025-08-26T20:36:29.1602033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1602500Z outputs = self.bert( 2025-08-26T20:36:29.1602945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1603404Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1603869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1604333Z layer_outputs = layer_module( 2025-08-26T20:36:29.1604761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1605155Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1605631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1606107Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1606600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1607067Z self_outputs = self.self( 2025-08-26T20:36:29.1607497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1607920Z return func(*args, **kwargs) 2025-08-26T20:36:29.1608389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1608867Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1609017Z 2025-08-26T20:36:29.1609139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1609531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1609908Z return mod(**inputs) 2025-08-26T20:36:29.1610327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1610778Z outputs = self.bert( 2025-08-26T20:36:29.1611205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1611666Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1612117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1612570Z layer_outputs = layer_module( 2025-08-26T20:36:29.1612949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1613337Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1613801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1614293Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1614778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1615233Z self_outputs = self.self( 2025-08-26T20:36:29.1615627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1616037Z return func(*args, **kwargs) 2025-08-26T20:36:29.1616489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1616958Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1617106Z 2025-08-26T20:36:29.1617203Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1617436Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1617698Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1618102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1618460Z return mod(**inputs) 2025-08-26T20:36:29.1618891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1619358Z outputs = self.bert( 2025-08-26T20:36:29.1619770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1620208Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1620663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1621089Z layer_outputs = layer_module( 2025-08-26T20:36:29.1621453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1621827Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1622262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1622695Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1623158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1623650Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1624159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1624626Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1624816Z 2025-08-26T20:36:29.1624928Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1625327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1625663Z return mod(**inputs) 2025-08-26T20:36:29.1626082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1626535Z outputs = self.bert( 2025-08-26T20:36:29.1626953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1627402Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1627850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1628298Z layer_outputs = layer_module( 2025-08-26T20:36:29.1628670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1629045Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1629530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1629991Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1630430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1630849Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1631332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1631851Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1632347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1632810Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1632958Z 2025-08-26T20:36:29.1633071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1633462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1633815Z return mod(**inputs) 2025-08-26T20:36:29.1634246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1634695Z outputs = self.bert( 2025-08-26T20:36:29.1635120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1635597Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1636048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1636517Z layer_outputs = layer_module( 2025-08-26T20:36:29.1636904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1637287Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1637753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1638229Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1638695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1639136Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1639719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1640268Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1640839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1641340Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1641747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1642127Z return self.act(input) 2025-08-26T20:36:29.1642259Z 2025-08-26T20:36:29.1642376Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1642774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1643128Z return mod(**inputs) 2025-08-26T20:36:29.1643565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1644016Z outputs = self.bert( 2025-08-26T20:36:29.1644452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1644942Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1645406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1645894Z layer_outputs = layer_module( 2025-08-26T20:36:29.1646274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1646668Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1647155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1647612Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1648047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1648477Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1648956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1649503Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1650008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1650471Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1650627Z 2025-08-26T20:36:29.1650739Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1651170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1651521Z return mod(**inputs) 2025-08-26T20:36:29.1651951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1652402Z outputs = self.bert( 2025-08-26T20:36:29.1652837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1653291Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1653753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1654223Z layer_outputs = layer_module( 2025-08-26T20:36:29.1654605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1655009Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1655493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1655961Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1656416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1656841Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1657320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1657864Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1658373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.1658832Z return input_tensor + hidden_states 2025-08-26T20:36:29.1658985Z 2025-08-26T20:36:29.1659102Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1659491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1659845Z return mod(**inputs) 2025-08-26T20:36:29.1660267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1660739Z outputs = self.bert( 2025-08-26T20:36:29.1661161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1661614Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1662068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1662512Z layer_outputs = layer_module( 2025-08-26T20:36:29.1662888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1663283Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1663734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1664188Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1664648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1665101Z self_outputs = self.self( 2025-08-26T20:36:29.1665494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1665900Z return func(*args, **kwargs) 2025-08-26T20:36:29.1666334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1666820Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1666979Z 2025-08-26T20:36:29.1667093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1667487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1667839Z return mod(**inputs) 2025-08-26T20:36:29.1668264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1668715Z outputs = self.bert( 2025-08-26T20:36:29.1669150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1669621Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1670058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1670505Z layer_outputs = layer_module( 2025-08-26T20:36:29.1670879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1671263Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1671753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1672208Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1672678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1673132Z self_outputs = self.self( 2025-08-26T20:36:29.1673525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1673928Z return func(*args, **kwargs) 2025-08-26T20:36:29.1674373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1674833Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1674983Z 2025-08-26T20:36:29.1675101Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1675508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1675885Z return mod(**inputs) 2025-08-26T20:36:29.1676336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1676796Z outputs = self.bert( 2025-08-26T20:36:29.1677246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1677710Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1678173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1678640Z layer_outputs = layer_module( 2025-08-26T20:36:29.1679026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1679498Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1679991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1680462Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1680943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1681410Z self_outputs = self.self( 2025-08-26T20:36:29.1681818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1682204Z return func(*args, **kwargs) 2025-08-26T20:36:29.1682642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1683082Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1683231Z 2025-08-26T20:36:29.1683319Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1683543Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1683796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1684186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1684536Z return mod(**inputs) 2025-08-26T20:36:29.1684984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1685438Z outputs = self.bert( 2025-08-26T20:36:29.1685854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1686308Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1686757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1687229Z layer_outputs = layer_module( 2025-08-26T20:36:29.1687604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1688003Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1688466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1688934Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1689404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1689915Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1690433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1690903Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1691056Z 2025-08-26T20:36:29.1691181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1691593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1691940Z return mod(**inputs) 2025-08-26T20:36:29.1692365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1692817Z outputs = self.bert( 2025-08-26T20:36:29.1693252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1693722Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1694178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1694646Z layer_outputs = layer_module( 2025-08-26T20:36:29.1695035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1695426Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1695886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1696503Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1696956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1697391Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1697984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1698494Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1698990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1699456Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1699604Z 2025-08-26T20:36:29.1699729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1700133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1700491Z return mod(**inputs) 2025-08-26T20:36:29.1700926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1701374Z outputs = self.bert( 2025-08-26T20:36:29.1701799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1702248Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1702690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1703173Z layer_outputs = layer_module( 2025-08-26T20:36:29.1703534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1703901Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1704334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1704800Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1705235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1705674Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1706169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1706683Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1707175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1707721Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1708133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1708504Z return self.act(input) 2025-08-26T20:36:29.1708628Z 2025-08-26T20:36:29.1708741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1709129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1709480Z return mod(**inputs) 2025-08-26T20:36:29.1710425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1710879Z outputs = self.bert( 2025-08-26T20:36:29.1711302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1711760Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1712215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1712661Z layer_outputs = layer_module( 2025-08-26T20:36:29.1713026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1713419Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1713899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1714364Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1714796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1715220Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1715711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1716255Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1716788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1717256Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1717406Z 2025-08-26T20:36:29.1717522Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1717916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1718261Z return mod(**inputs) 2025-08-26T20:36:29.1718718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1719167Z outputs = self.bert( 2025-08-26T20:36:29.1719678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1720167Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1720629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1721094Z layer_outputs = layer_module( 2025-08-26T20:36:29.1721473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1721864Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1722326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1722789Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1723243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1723715Z self_outputs = self.self( 2025-08-26T20:36:29.1724108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1724509Z return func(*args, **kwargs) 2025-08-26T20:36:29.1724949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1725416Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1725562Z 2025-08-26T20:36:29.1725678Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1726069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1726414Z return mod(**inputs) 2025-08-26T20:36:29.1726840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1727289Z outputs = self.bert( 2025-08-26T20:36:29.1727704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1728154Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1728598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1729054Z layer_outputs = layer_module( 2025-08-26T20:36:29.1729437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1729828Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1730287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1730740Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1731170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1731588Z self_outputs = self.self( 2025-08-26T20:36:29.1731978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1732362Z return func(*args, **kwargs) 2025-08-26T20:36:29.1732776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1733215Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1733351Z 2025-08-26T20:36:29.1733459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1733851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1734185Z return mod(**inputs) 2025-08-26T20:36:29.1734595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1735020Z outputs = self.bert( 2025-08-26T20:36:29.1735421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1735851Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1736274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1736700Z layer_outputs = layer_module( 2025-08-26T20:36:29.1737051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1737433Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1737886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1738375Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1738839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1739282Z self_outputs = self.self( 2025-08-26T20:36:29.1739651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1740032Z return func(*args, **kwargs) 2025-08-26T20:36:29.1740454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1740880Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1741034Z 2025-08-26T20:36:29.1741124Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1741360Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1741621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1742010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1742352Z return mod(**inputs) 2025-08-26T20:36:29.1742786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1743231Z outputs = self.bert( 2025-08-26T20:36:29.1743655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1744092Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1744520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1744949Z layer_outputs = layer_module( 2025-08-26T20:36:29.1745320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1745712Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1746155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1746589Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1747037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1747543Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1748048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1748484Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1748651Z 2025-08-26T20:36:29.1748760Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1749133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1749483Z return mod(**inputs) 2025-08-26T20:36:29.1749916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1750338Z outputs = self.bert( 2025-08-26T20:36:29.1750761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1751205Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1751630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1752048Z layer_outputs = layer_module( 2025-08-26T20:36:29.1752405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1752772Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1753227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1753665Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1754078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1754485Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1754942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1755436Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1755892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1756326Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1756484Z 2025-08-26T20:36:29.1756596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1756983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1757334Z return mod(**inputs) 2025-08-26T20:36:29.1757764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1758205Z outputs = self.bert( 2025-08-26T20:36:29.1758639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1759117Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1759652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1760126Z layer_outputs = layer_module( 2025-08-26T20:36:29.1760518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1760934Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1761408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1761911Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1762342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1762777Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1763250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1763741Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1764259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1764755Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1765174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1765554Z return self.act(input) 2025-08-26T20:36:29.1765679Z 2025-08-26T20:36:29.1765805Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1766207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1766558Z return mod(**inputs) 2025-08-26T20:36:29.1766997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1767454Z outputs = self.bert( 2025-08-26T20:36:29.1767882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1768349Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1768795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1769245Z layer_outputs = layer_module( 2025-08-26T20:36:29.1769620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1770008Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1770457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1770922Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1771356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1771785Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1772271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1772821Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1773337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1773798Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1773948Z 2025-08-26T20:36:29.1774068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1774484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1774834Z return mod(**inputs) 2025-08-26T20:36:29.1775273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1775734Z outputs = self.bert( 2025-08-26T20:36:29.1776157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1776612Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1777091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1777569Z layer_outputs = layer_module( 2025-08-26T20:36:29.1777949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1778345Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1778823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1779307Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1779737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1780164Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1780660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1781204Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1781715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.1782178Z return input_tensor + hidden_states 2025-08-26T20:36:29.1782320Z 2025-08-26T20:36:29.1782444Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1782833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1783179Z return mod(**inputs) 2025-08-26T20:36:29.1783618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1784109Z outputs = self.bert( 2025-08-26T20:36:29.1784545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1785014Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1785486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1785941Z layer_outputs = layer_module( 2025-08-26T20:36:29.1786326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1786721Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1787179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1787657Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1788119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1788577Z self_outputs = self.self( 2025-08-26T20:36:29.1788984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1789384Z return func(*args, **kwargs) 2025-08-26T20:36:29.1789832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1790386Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1790574Z 2025-08-26T20:36:29.1790699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1791088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1791440Z return mod(**inputs) 2025-08-26T20:36:29.1791873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1792324Z outputs = self.bert( 2025-08-26T20:36:29.1792768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1793218Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1793683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1794151Z layer_outputs = layer_module( 2025-08-26T20:36:29.1794538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1794963Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1795436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1795923Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1796550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1797032Z self_outputs = self.self( 2025-08-26T20:36:29.1797440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1797858Z return func(*args, **kwargs) 2025-08-26T20:36:29.1798325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1798793Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1798945Z 2025-08-26T20:36:29.1799071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1799517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1799943Z return mod(**inputs) 2025-08-26T20:36:29.1800408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1800885Z outputs = self.bert( 2025-08-26T20:36:29.1801334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1801792Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1802268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1802746Z layer_outputs = layer_module( 2025-08-26T20:36:29.1803136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1803544Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1804007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1804484Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1804958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1805419Z self_outputs = self.self( 2025-08-26T20:36:29.1805814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1806222Z return func(*args, **kwargs) 2025-08-26T20:36:29.1806675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1807133Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1807277Z 2025-08-26T20:36:29.1807372Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1807599Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1807853Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1808249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1808595Z return mod(**inputs) 2025-08-26T20:36:29.1809031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1809461Z outputs = self.bert( 2025-08-26T20:36:29.1809867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1810297Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1810717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1811167Z layer_outputs = layer_module( 2025-08-26T20:36:29.1811526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1811903Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1812362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1812827Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1813278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1813762Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1814253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1814696Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1814836Z 2025-08-26T20:36:29.1814971Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1815336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1815668Z return mod(**inputs) 2025-08-26T20:36:29.1816074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1816499Z outputs = self.bert( 2025-08-26T20:36:29.1816908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1817372Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1817819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1818280Z layer_outputs = layer_module( 2025-08-26T20:36:29.1818651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1819032Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1819500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1819966Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1820381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1820782Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1821254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1821758Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1822250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1822707Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1822849Z 2025-08-26T20:36:29.1822961Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1823321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1823651Z return mod(**inputs) 2025-08-26T20:36:29.1824087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1824511Z outputs = self.bert( 2025-08-26T20:36:29.1824908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1825339Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1825785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1826234Z layer_outputs = layer_module( 2025-08-26T20:36:29.1826608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1826999Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1827466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1827931Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1828361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1828792Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1829301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1829796Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1830307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1830807Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1831215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1831596Z return self.act(input) 2025-08-26T20:36:29.1831724Z 2025-08-26T20:36:29.1831836Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1832225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1832612Z return mod(**inputs) 2025-08-26T20:36:29.1833031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1833507Z outputs = self.bert( 2025-08-26T20:36:29.1833951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1834431Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1834902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1835365Z layer_outputs = layer_module( 2025-08-26T20:36:29.1835736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1836146Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1836639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1837105Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1837531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1837951Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1838436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1838999Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1839638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1840129Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1840290Z 2025-08-26T20:36:29.1840410Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1840811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1841183Z return mod(**inputs) 2025-08-26T20:36:29.1841628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1842082Z outputs = self.bert( 2025-08-26T20:36:29.1842506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1842961Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1843424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1843865Z layer_outputs = layer_module( 2025-08-26T20:36:29.1844237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1844623Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1845076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1845547Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1846036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1846496Z self_outputs = self.self( 2025-08-26T20:36:29.1846889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1847289Z return func(*args, **kwargs) 2025-08-26T20:36:29.1847724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1848184Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1848340Z 2025-08-26T20:36:29.1848452Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1848837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1849185Z return mod(**inputs) 2025-08-26T20:36:29.1849601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1849684Z outputs = self.bert( 2025-08-26T20:36:29.1849992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1850082Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1850389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1850466Z layer_outputs = layer_module( 2025-08-26T20:36:29.1850730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1850816Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1851137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1851227Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1851542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1851619Z self_outputs = self.self( 2025-08-26T20:36:29.1851894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1851983Z return func(*args, **kwargs) 2025-08-26T20:36:29.1852299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1852394Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1852398Z 2025-08-26T20:36:29.1852512Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1852749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1852830Z return mod(**inputs) 2025-08-26T20:36:29.1853149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1853228Z outputs = self.bert( 2025-08-26T20:36:29.1853544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1853632Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1853946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1854026Z layer_outputs = layer_module( 2025-08-26T20:36:29.1854277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1854364Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1854684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1854798Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1855101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1855187Z self_outputs = self.self( 2025-08-26T20:36:29.1855446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1855530Z return func(*args, **kwargs) 2025-08-26T20:36:29.1855833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1855923Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1855930Z 2025-08-26T20:36:29.1856018Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1856105Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1856227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1856437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1856515Z return mod(**inputs) 2025-08-26T20:36:29.1856828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1856898Z outputs = self.bert( 2025-08-26T20:36:29.1857212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1857309Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1857624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1857704Z layer_outputs = layer_module( 2025-08-26T20:36:29.1857944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1858036Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1858345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1858454Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1858761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1858907Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1859217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1859334Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1859338Z 2025-08-26T20:36:29.1859459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1859672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1859752Z return mod(**inputs) 2025-08-26T20:36:29.1860062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1860134Z outputs = self.bert( 2025-08-26T20:36:29.1860446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1860526Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1860841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1860920Z layer_outputs = layer_module( 2025-08-26T20:36:29.1861173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1861288Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1861609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1861711Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1861995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1862090Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1862439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1862559Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1862873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1862967Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1862973Z 2025-08-26T20:36:29.1863094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1863311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1863391Z return mod(**inputs) 2025-08-26T20:36:29.1863704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1863783Z outputs = self.bert( 2025-08-26T20:36:29.1864131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1864212Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1864523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1864602Z layer_outputs = layer_module( 2025-08-26T20:36:29.1864842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1864931Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1865238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1865388Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1865668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1865761Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1866114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1866249Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1866570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1866700Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1866940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1867018Z return self.act(input) 2025-08-26T20:36:29.1867024Z 2025-08-26T20:36:29.1867136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1867361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1867432Z return mod(**inputs) 2025-08-26T20:36:29.1867758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1867833Z outputs = self.bert( 2025-08-26T20:36:29.1868157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1868259Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1868569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1868657Z layer_outputs = layer_module( 2025-08-26T20:36:29.1868907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1869002Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1869321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1869415Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1869713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1869801Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1870163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1870309Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1870641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1870731Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1870735Z 2025-08-26T20:36:29.1870848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1871099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1871174Z return mod(**inputs) 2025-08-26T20:36:29.1871498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1871574Z outputs = self.bert( 2025-08-26T20:36:29.1871890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1871979Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1872326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1872413Z layer_outputs = layer_module( 2025-08-26T20:36:29.1872662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1872756Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1873079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1873190Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1873483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1873570Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1873923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1874070Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1874386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.1874483Z return input_tensor + hidden_states 2025-08-26T20:36:29.1874487Z 2025-08-26T20:36:29.1874600Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1874829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1874905Z return mod(**inputs) 2025-08-26T20:36:29.1875230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1875329Z outputs = self.bert( 2025-08-26T20:36:29.1875645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1875736Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1876049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1876137Z layer_outputs = layer_module( 2025-08-26T20:36:29.1876383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1876470Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1876796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1876887Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1877212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1877292Z self_outputs = self.self( 2025-08-26T20:36:29.1877569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1877647Z return func(*args, **kwargs) 2025-08-26T20:36:29.1877978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1878080Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1878084Z 2025-08-26T20:36:29.1878198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1878426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1878500Z return mod(**inputs) 2025-08-26T20:36:29.1878817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1878898Z outputs = self.bert( 2025-08-26T20:36:29.1879229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1879318Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1879725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1879819Z layer_outputs = layer_module( 2025-08-26T20:36:29.1880061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1880177Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1880501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1880593Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1880914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1880994Z self_outputs = self.self( 2025-08-26T20:36:29.1881260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1881347Z return func(*args, **kwargs) 2025-08-26T20:36:29.1881664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1881759Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1881765Z 2025-08-26T20:36:29.1881887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1882101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1882204Z return mod(**inputs) 2025-08-26T20:36:29.1882515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1882594Z outputs = self.bert( 2025-08-26T20:36:29.1882904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1882991Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1883304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1883380Z layer_outputs = layer_module( 2025-08-26T20:36:29.1883626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1883712Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1884027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1884115Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1884447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1884532Z self_outputs = self.self( 2025-08-26T20:36:29.1884794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1884896Z return func(*args, **kwargs) 2025-08-26T20:36:29.1885209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1885307Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1885311Z 2025-08-26T20:36:29.1885400Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1885489Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1885608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1885821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1885903Z return mod(**inputs) 2025-08-26T20:36:29.1886234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1886307Z outputs = self.bert( 2025-08-26T20:36:29.1886627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1886707Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1887026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1887125Z layer_outputs = layer_module( 2025-08-26T20:36:29.1887362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1887454Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1887761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1887856Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1888163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1888311Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1888622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1888711Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1888744Z 2025-08-26T20:36:29.1888865Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1889078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1889155Z return mod(**inputs) 2025-08-26T20:36:29.1889468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1889539Z outputs = self.bert( 2025-08-26T20:36:29.1889853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1889932Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1890243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1890321Z layer_outputs = layer_module( 2025-08-26T20:36:29.1890568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1890651Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1891052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1891159Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1891440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1891530Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1891894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1892011Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1892325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1892416Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1892420Z 2025-08-26T20:36:29.1892536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1892752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1892850Z return mod(**inputs) 2025-08-26T20:36:29.1893199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1893276Z outputs = self.bert( 2025-08-26T20:36:29.1893595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1893676Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1894013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1894092Z layer_outputs = layer_module( 2025-08-26T20:36:29.1894331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1894421Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1894727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1894825Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1895107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1895197Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1895538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1895654Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1895991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1896112Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1896480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1896563Z return self.act(input) 2025-08-26T20:36:29.1896567Z 2025-08-26T20:36:29.1896686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1896901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1896973Z return mod(**inputs) 2025-08-26T20:36:29.1897295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1897370Z outputs = self.bert( 2025-08-26T20:36:29.1897688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1897767Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1898075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1898161Z layer_outputs = layer_module( 2025-08-26T20:36:29.1898401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1898494Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1898859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1898952Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1899251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1899339Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1899692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1899847Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1900189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1900278Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1900282Z 2025-08-26T20:36:29.1900394Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1900616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1900719Z return mod(**inputs) 2025-08-26T20:36:29.1901040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1901114Z outputs = self.bert( 2025-08-26T20:36:29.1901423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1901512Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1901821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1901906Z layer_outputs = layer_module( 2025-08-26T20:36:29.1902145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1902236Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1902543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1902632Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1902977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1903055Z self_outputs = self.self( 2025-08-26T20:36:29.1903321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1903401Z return func(*args, **kwargs) 2025-08-26T20:36:29.1903715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1903817Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1903821Z 2025-08-26T20:36:29.1903935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1904165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1904237Z return mod(**inputs) 2025-08-26T20:36:29.1904552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1904624Z outputs = self.bert( 2025-08-26T20:36:29.1904936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1905024Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1905327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1905410Z layer_outputs = layer_module( 2025-08-26T20:36:29.1906420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1906514Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1906845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1906937Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1907247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1907323Z self_outputs = self.self( 2025-08-26T20:36:29.1907611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1907688Z return func(*args, **kwargs) 2025-08-26T20:36:29.1908006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1908099Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1908103Z 2025-08-26T20:36:29.1908214Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1908460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1908534Z return mod(**inputs) 2025-08-26T20:36:29.1908845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1908923Z outputs = self.bert( 2025-08-26T20:36:29.1909241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1909326Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1909635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1909719Z layer_outputs = layer_module( 2025-08-26T20:36:29.1909954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1910037Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1910353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1910461Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1910775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1910852Z self_outputs = self.self( 2025-08-26T20:36:29.1911110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1911194Z return func(*args, **kwargs) 2025-08-26T20:36:29.1911504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1911595Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1911601Z 2025-08-26T20:36:29.1911687Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1911776Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1911893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1912107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1912187Z return mod(**inputs) 2025-08-26T20:36:29.1912496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1912575Z outputs = self.bert( 2025-08-26T20:36:29.1912903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1912983Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1913300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1913377Z layer_outputs = layer_module( 2025-08-26T20:36:29.1913622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1913705Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1914010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1914118Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1914427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1914576Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1914884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1915006Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1915010Z 2025-08-26T20:36:29.1915124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1915346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1915427Z return mod(**inputs) 2025-08-26T20:36:29.1915747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1915827Z outputs = self.bert( 2025-08-26T20:36:29.1916144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1916224Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1916551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1916630Z layer_outputs = layer_module( 2025-08-26T20:36:29.1916882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1916984Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1917304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1917393Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1917680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1917771Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1918116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1918238Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1918555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1918647Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1918651Z 2025-08-26T20:36:29.1918771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1918989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1919069Z return mod(**inputs) 2025-08-26T20:36:29.1919384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1919532Z outputs = self.bert( 2025-08-26T20:36:29.1919874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1919960Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1920288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1920371Z layer_outputs = layer_module( 2025-08-26T20:36:29.1920625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1920711Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1921047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1921151Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1921439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1921534Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1921883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1922028Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1922346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1922476Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1922719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1922801Z return self.act(input) 2025-08-26T20:36:29.1922805Z 2025-08-26T20:36:29.1922927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1923148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1923224Z return mod(**inputs) 2025-08-26T20:36:29.1923553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1923627Z outputs = self.bert( 2025-08-26T20:36:29.1923953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1924071Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1924390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1924476Z layer_outputs = layer_module( 2025-08-26T20:36:29.1924723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1924815Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1925137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1925235Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1925523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1925609Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1925967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1926112Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1926436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1926525Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1926529Z 2025-08-26T20:36:29.1926648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1926887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1926961Z return mod(**inputs) 2025-08-26T20:36:29.1927293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1927369Z outputs = self.bert( 2025-08-26T20:36:29.1927690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1927772Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1928106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1928186Z layer_outputs = layer_module( 2025-08-26T20:36:29.1928409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1928496Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1928788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1928888Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1929155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1929233Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1929557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1929691Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1929988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.1930070Z return input_tensor + hidden_states 2025-08-26T20:36:29.1930073Z 2025-08-26T20:36:29.1930176Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1930386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1930452Z return mod(**inputs) 2025-08-26T20:36:29.1930767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1930834Z outputs = self.bert( 2025-08-26T20:36:29.1931133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1931208Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1931501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1931582Z layer_outputs = layer_module( 2025-08-26T20:36:29.1931805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1931890Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1932179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1932262Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1932577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1932653Z self_outputs = self.self( 2025-08-26T20:36:29.1932925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1932999Z return func(*args, **kwargs) 2025-08-26T20:36:29.1933330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1933429Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1933433Z 2025-08-26T20:36:29.1933544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1933764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1933837Z return mod(**inputs) 2025-08-26T20:36:29.1934165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1934236Z outputs = self.bert( 2025-08-26T20:36:29.1934567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1934655Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1934971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1935049Z layer_outputs = layer_module( 2025-08-26T20:36:29.1935277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1935372Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1935673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1935758Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1936060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1936133Z self_outputs = self.self( 2025-08-26T20:36:29.1936392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1936463Z return func(*args, **kwargs) 2025-08-26T20:36:29.1936761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1936849Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1936853Z 2025-08-26T20:36:29.1936960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1937189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1937257Z return mod(**inputs) 2025-08-26T20:36:29.1937553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1937629Z outputs = self.bert( 2025-08-26T20:36:29.1937921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1938004Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1938297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1938376Z layer_outputs = layer_module( 2025-08-26T20:36:29.1938601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1938682Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1938977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1939058Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1939356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1939429Z self_outputs = self.self( 2025-08-26T20:36:29.1939676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1939774Z return func(*args, **kwargs) 2025-08-26T20:36:29.1940947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1942341Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1942368Z 2025-08-26T20:36:29.1942484Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1942578Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1942728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1943029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1943281Z return mod(**inputs) 2025-08-26T20:36:29.1943708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1943797Z outputs = self.bert( 2025-08-26T20:36:29.1944133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1944221Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1944589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1944674Z layer_outputs = layer_module( 2025-08-26T20:36:29.1944940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1945031Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1945357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1945455Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1945799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1945950Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1946271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1946375Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1946408Z 2025-08-26T20:36:29.1946527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1946752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1946836Z return mod(**inputs) 2025-08-26T20:36:29.1947161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1947242Z outputs = self.bert( 2025-08-26T20:36:29.1947560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1947642Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1947965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1948057Z layer_outputs = layer_module( 2025-08-26T20:36:29.1948440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1948538Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1948864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1948958Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1949271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1949385Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1949776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1949906Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1950225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1950315Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1950327Z 2025-08-26T20:36:29.1950442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1950693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1950778Z return mod(**inputs) 2025-08-26T20:36:29.1951114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1951195Z outputs = self.bert( 2025-08-26T20:36:29.1951518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1951620Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1951971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1952053Z layer_outputs = layer_module( 2025-08-26T20:36:29.1952300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1952384Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1952711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1952808Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1953111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1953203Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1953568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1953712Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1954041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1954166Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1954412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1954504Z return self.act(input) 2025-08-26T20:36:29.1954509Z 2025-08-26T20:36:29.1954631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1954854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1954926Z return mod(**inputs) 2025-08-26T20:36:29.1955265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1955339Z outputs = self.bert( 2025-08-26T20:36:29.1955659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1955741Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1956077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1956157Z layer_outputs = layer_module( 2025-08-26T20:36:29.1956406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1956520Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1956851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1956951Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1957240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1957327Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1957698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1957867Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1958195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1958286Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1958292Z 2025-08-26T20:36:29.1958418Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1958641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1958731Z return mod(**inputs) 2025-08-26T20:36:29.1959134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1959220Z outputs = self.bert( 2025-08-26T20:36:29.1959642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1959740Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1960077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1960168Z layer_outputs = layer_module( 2025-08-26T20:36:29.1960422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1960518Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1960856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1960978Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1961301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1961381Z self_outputs = self.self( 2025-08-26T20:36:29.1961671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1961756Z return func(*args, **kwargs) 2025-08-26T20:36:29.1962094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1962187Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1962191Z 2025-08-26T20:36:29.1962302Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1962526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1962598Z return mod(**inputs) 2025-08-26T20:36:29.1962917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1962986Z outputs = self.bert( 2025-08-26T20:36:29.1963311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1963392Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1963713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1963816Z layer_outputs = layer_module( 2025-08-26T20:36:29.1964055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1964147Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1964468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1964561Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1964888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1964967Z self_outputs = self.self( 2025-08-26T20:36:29.1965265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1965346Z return func(*args, **kwargs) 2025-08-26T20:36:29.1965668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1965762Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1965786Z 2025-08-26T20:36:29.1965905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1966131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1966205Z return mod(**inputs) 2025-08-26T20:36:29.1966536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1966610Z outputs = self.bert( 2025-08-26T20:36:29.1966932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1967024Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1967337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1967423Z layer_outputs = layer_module( 2025-08-26T20:36:29.1967662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1967745Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1968086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1968172Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1968494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1968570Z self_outputs = self.self( 2025-08-26T20:36:29.1968837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1968911Z return func(*args, **kwargs) 2025-08-26T20:36:29.1969218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.1969311Z value_layer = self.value(current_states) 2025-08-26T20:36:29.1969315Z 2025-08-26T20:36:29.1969404Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1969496Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.1969608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1969825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1969901Z return mod(**inputs) 2025-08-26T20:36:29.1970223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1970299Z outputs = self.bert( 2025-08-26T20:36:29.1970639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1970733Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1971057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1971143Z layer_outputs = layer_module( 2025-08-26T20:36:29.1971404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1971490Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1971839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1971931Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1972249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.1972407Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.1972724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.1972853Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1972857Z 2025-08-26T20:36:29.1972985Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1973209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1973283Z return mod(**inputs) 2025-08-26T20:36:29.1973604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1973685Z outputs = self.bert( 2025-08-26T20:36:29.1974001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1974092Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1974411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1974491Z layer_outputs = layer_module( 2025-08-26T20:36:29.1974745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1974848Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1975175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1975269Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1975574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1975670Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1976024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1976148Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1976472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.1976574Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1976577Z 2025-08-26T20:36:29.1976693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1976913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1977001Z return mod(**inputs) 2025-08-26T20:36:29.1977326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1977409Z outputs = self.bert( 2025-08-26T20:36:29.1977773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1977862Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1978200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1978286Z layer_outputs = layer_module( 2025-08-26T20:36:29.1978541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1978628Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1978976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1979070Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1979361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1979457Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1979819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.1979960Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.1980284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.1980416Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.1980658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.1980738Z return self.act(input) 2025-08-26T20:36:29.1980743Z 2025-08-26T20:36:29.1980862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1981088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1981170Z return mod(**inputs) 2025-08-26T20:36:29.1981493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1981571Z outputs = self.bert( 2025-08-26T20:36:29.1981901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1982011Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1982352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1982437Z layer_outputs = layer_module( 2025-08-26T20:36:29.1982682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1982776Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1983090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1983191Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1983481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1983577Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1983926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1984082Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1984410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.1984500Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.1984507Z 2025-08-26T20:36:29.1984647Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1984870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1984946Z return mod(**inputs) 2025-08-26T20:36:29.1985276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1985351Z outputs = self.bert( 2025-08-26T20:36:29.1985676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1985757Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1986106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1986187Z layer_outputs = layer_module( 2025-08-26T20:36:29.1986431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1986528Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1986842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.1986962Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.1987250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.1987334Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.1987692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.1987842Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.1988171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.1988257Z return input_tensor + hidden_states 2025-08-26T20:36:29.1988261Z 2025-08-26T20:36:29.1988383Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1988612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1988703Z return mod(**inputs) 2025-08-26T20:36:29.1989025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1989096Z outputs = self.bert( 2025-08-26T20:36:29.1989424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1989504Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1989815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1989904Z layer_outputs = layer_module( 2025-08-26T20:36:29.1990142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1990237Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1990560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1990660Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1991232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1991317Z self_outputs = self.self( 2025-08-26T20:36:29.1991605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1991685Z return func(*args, **kwargs) 2025-08-26T20:36:29.1992039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.1992133Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.1992140Z 2025-08-26T20:36:29.1992257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1992488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1992562Z return mod(**inputs) 2025-08-26T20:36:29.1992893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1992969Z outputs = self.bert( 2025-08-26T20:36:29.1993321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1993407Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1993730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1993818Z layer_outputs = layer_module( 2025-08-26T20:36:29.1994063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1994175Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1994496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1994586Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1994927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1995007Z self_outputs = self.self( 2025-08-26T20:36:29.1995284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1995365Z return func(*args, **kwargs) 2025-08-26T20:36:29.1995687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.1995774Z key_layer = self.key(current_states) 2025-08-26T20:36:29.1995778Z 2025-08-26T20:36:29.1995892Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.1996281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.1996401Z return mod(**inputs) 2025-08-26T20:36:29.1996950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.1997034Z outputs = self.bert( 2025-08-26T20:36:29.1997356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.1997445Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.1997774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.1997860Z layer_outputs = layer_module( 2025-08-26T20:36:29.1998108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.1998204Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.1998522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.1998612Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.1998939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.1999018Z self_outputs = self.self( 2025-08-26T20:36:29.1999606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.1999735Z return func(*args, **kwargs) 2025-08-26T20:36:29.2000158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2000267Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2000275Z 2025-08-26T20:36:29.2000366Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2000462Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2000576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2000807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2000928Z return mod(**inputs) 2025-08-26T20:36:29.2001242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2001324Z outputs = self.bert( 2025-08-26T20:36:29.2001637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2001727Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2002066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2002144Z layer_outputs = layer_module( 2025-08-26T20:36:29.2002389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2002470Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2002792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2002878Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2003187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2003337Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2003646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2003779Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2003783Z 2025-08-26T20:36:29.2003897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2004123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2004191Z return mod(**inputs) 2025-08-26T20:36:29.2004515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2004595Z outputs = self.bert( 2025-08-26T20:36:29.2004908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2004993Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2005308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2005385Z layer_outputs = layer_module( 2025-08-26T20:36:29.2005639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2005723Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2006045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2006139Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2006436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2006543Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2006890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2007012Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2007324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2007424Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2007428Z 2025-08-26T20:36:29.2007538Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2007766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2007845Z return mod(**inputs) 2025-08-26T20:36:29.2008158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2008241Z outputs = self.bert( 2025-08-26T20:36:29.2008551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2008656Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2008967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2009046Z layer_outputs = layer_module( 2025-08-26T20:36:29.2009295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2009378Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2009704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2009796Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2010079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2010178Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2010501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2010633Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2010924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2011047Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2011266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2011342Z return self.act(input) 2025-08-26T20:36:29.2011348Z 2025-08-26T20:36:29.2011458Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2011666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2011742Z return mod(**inputs) 2025-08-26T20:36:29.2012044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2012119Z outputs = self.bert( 2025-08-26T20:36:29.2012434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2012515Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2012838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2012913Z layer_outputs = layer_module( 2025-08-26T20:36:29.2013149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2013248Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2013544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2013639Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2013906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2014015Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2014357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2014520Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2014839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2014929Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2014933Z 2025-08-26T20:36:29.2015051Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2015267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2015365Z return mod(**inputs) 2025-08-26T20:36:29.2015681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2015755Z outputs = self.bert( 2025-08-26T20:36:29.2016070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2016149Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2016467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2016542Z layer_outputs = layer_module( 2025-08-26T20:36:29.2016784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2016883Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2017193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2017308Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2017622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2017713Z self_outputs = self.self( 2025-08-26T20:36:29.2017965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2018036Z return func(*args, **kwargs) 2025-08-26T20:36:29.2018340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2018423Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2018427Z 2025-08-26T20:36:29.2018537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2018743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2018811Z return mod(**inputs) 2025-08-26T20:36:29.2019116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2019184Z outputs = self.bert( 2025-08-26T20:36:29.2019487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2019562Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2019867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2019947Z layer_outputs = layer_module( 2025-08-26T20:36:29.2020177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2020272Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2020584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2020678Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2020985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2021077Z self_outputs = self.self( 2025-08-26T20:36:29.2021346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2021424Z return func(*args, **kwargs) 2025-08-26T20:36:29.2021766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2021877Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2021907Z 2025-08-26T20:36:29.2022024Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2022251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2022321Z return mod(**inputs) 2025-08-26T20:36:29.2022639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2022709Z outputs = self.bert( 2025-08-26T20:36:29.2023019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2023093Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2023387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2023466Z layer_outputs = layer_module( 2025-08-26T20:36:29.2023697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2023787Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2024098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2024183Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2024483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2024555Z self_outputs = self.self( 2025-08-26T20:36:29.2024808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2024879Z return func(*args, **kwargs) 2025-08-26T20:36:29.2025179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2025261Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2025265Z 2025-08-26T20:36:29.2025352Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2025445Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2025559Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2025788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2025859Z return mod(**inputs) 2025-08-26T20:36:29.2026176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2026256Z outputs = self.bert( 2025-08-26T20:36:29.2026585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2026675Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2026986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2027066Z layer_outputs = layer_module( 2025-08-26T20:36:29.2027317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2027396Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2027718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2027812Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2028102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2028234Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2028518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2028683Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2028688Z 2025-08-26T20:36:29.2028790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2028997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2029062Z return mod(**inputs) 2025-08-26T20:36:29.2029358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2029433Z outputs = self.bert( 2025-08-26T20:36:29.2029724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2029818Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2030104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2030182Z layer_outputs = layer_module( 2025-08-26T20:36:29.2030403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2030497Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2030788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2030871Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2031139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2031219Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2031540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2031654Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2031952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2032045Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2032048Z 2025-08-26T20:36:29.2032152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2032359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2032431Z return mod(**inputs) 2025-08-26T20:36:29.2032728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2032803Z outputs = self.bert( 2025-08-26T20:36:29.2033117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2033205Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2033520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2033599Z layer_outputs = layer_module( 2025-08-26T20:36:29.2033848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2033933Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2034267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2034358Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2034644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2034729Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2035066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2035203Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2035512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2035644Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2035881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2035958Z return self.act(input) 2025-08-26T20:36:29.2035970Z 2025-08-26T20:36:29.2036084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2036305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2036386Z return mod(**inputs) 2025-08-26T20:36:29.2036702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2036784Z outputs = self.bert( 2025-08-26T20:36:29.2037117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2037195Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2037518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2037597Z layer_outputs = layer_module( 2025-08-26T20:36:29.2037848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2037934Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2038254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2038353Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2038640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2038733Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2039080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2039236Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2039748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2039857Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2039862Z 2025-08-26T20:36:29.2040030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2040257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2040343Z return mod(**inputs) 2025-08-26T20:36:29.2040665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2040741Z outputs = self.bert( 2025-08-26T20:36:29.2041069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2041154Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2041451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2041522Z layer_outputs = layer_module( 2025-08-26T20:36:29.2041759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2041845Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2042164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2042289Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2042577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2042670Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2043018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2043164Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2043489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.2043574Z return input_tensor + hidden_states 2025-08-26T20:36:29.2043578Z 2025-08-26T20:36:29.2043698Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2043921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2044025Z return mod(**inputs) 2025-08-26T20:36:29.2044348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2044420Z outputs = self.bert( 2025-08-26T20:36:29.2044748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2044830Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2045155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2045235Z layer_outputs = layer_module( 2025-08-26T20:36:29.2045481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2045578Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2045894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2045993Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2046311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2046399Z self_outputs = self.self( 2025-08-26T20:36:29.2046671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2046751Z return func(*args, **kwargs) 2025-08-26T20:36:29.2047092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2047187Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2047193Z 2025-08-26T20:36:29.2047313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2047532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2047609Z return mod(**inputs) 2025-08-26T20:36:29.2047903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2047967Z outputs = self.bert( 2025-08-26T20:36:29.2048278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2048351Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2048640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2048712Z layer_outputs = layer_module( 2025-08-26T20:36:29.2048947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2049034Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2049317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2049403Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2049687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2049760Z self_outputs = self.self( 2025-08-26T20:36:29.2050004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2050076Z return func(*args, **kwargs) 2025-08-26T20:36:29.2050365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2050445Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2050449Z 2025-08-26T20:36:29.2050576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2050771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2050836Z return mod(**inputs) 2025-08-26T20:36:29.2051128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2051193Z outputs = self.bert( 2025-08-26T20:36:29.2051485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2051558Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2051840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2051920Z layer_outputs = layer_module( 2025-08-26T20:36:29.2052135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2052218Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2052496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2052576Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2052869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2052941Z self_outputs = self.self( 2025-08-26T20:36:29.2053206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2053278Z return func(*args, **kwargs) 2025-08-26T20:36:29.2053575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2053657Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2053662Z 2025-08-26T20:36:29.2053744Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2053834Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2053938Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2054167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2054236Z return mod(**inputs) 2025-08-26T20:36:29.2054532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2057201Z outputs = self.bert( 2025-08-26T20:36:29.2057498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2058775Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2059116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2059193Z layer_outputs = layer_module( 2025-08-26T20:36:29.2059438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2059531Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2059858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2059950Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2060247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2060423Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2060735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2060826Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2060830Z 2025-08-26T20:36:29.2060940Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2061154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2061221Z return mod(**inputs) 2025-08-26T20:36:29.2061525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2061600Z outputs = self.bert( 2025-08-26T20:36:29.2061899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2061983Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2062339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2062417Z layer_outputs = layer_module( 2025-08-26T20:36:29.2062668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2062751Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2063077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2063169Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2063466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2063568Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2063913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2064037Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2064341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2064434Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2064438Z 2025-08-26T20:36:29.2064543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2064770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2064851Z return mod(**inputs) 2025-08-26T20:36:29.2065164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2065317Z outputs = self.bert( 2025-08-26T20:36:29.2065645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2065759Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2066075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2066151Z layer_outputs = layer_module( 2025-08-26T20:36:29.2066395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2066481Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2066808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2066896Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2067182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2067271Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2067636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2067753Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2068069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2068197Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2068427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2068502Z return self.act(input) 2025-08-26T20:36:29.2068506Z 2025-08-26T20:36:29.2068625Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2068842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2068919Z return mod(**inputs) 2025-08-26T20:36:29.2069235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2069305Z outputs = self.bert( 2025-08-26T20:36:29.2069633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2069710Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2070031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2070107Z layer_outputs = layer_module( 2025-08-26T20:36:29.2070344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2070457Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2070781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2070884Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2071167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2071257Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2071612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2071772Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2072092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2072215Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2072219Z 2025-08-26T20:36:29.2072338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2072554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2072667Z return mod(**inputs) 2025-08-26T20:36:29.2072982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2073054Z outputs = self.bert( 2025-08-26T20:36:29.2073383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2073463Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2073778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2073859Z layer_outputs = layer_module( 2025-08-26T20:36:29.2074097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2074187Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2074500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2074595Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2074903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2074979Z self_outputs = self.self( 2025-08-26T20:36:29.2075250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2075329Z return func(*args, **kwargs) 2025-08-26T20:36:29.2075652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2075745Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2075749Z 2025-08-26T20:36:29.2075869Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2076087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2076160Z return mod(**inputs) 2025-08-26T20:36:29.2076485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2076558Z outputs = self.bert( 2025-08-26T20:36:29.2076881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2076963Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2077301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2077393Z layer_outputs = layer_module( 2025-08-26T20:36:29.2077640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2077739Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2078060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2078161Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2078503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2078583Z self_outputs = self.self( 2025-08-26T20:36:29.2078861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2078938Z return func(*args, **kwargs) 2025-08-26T20:36:29.2079292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2079379Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2079530Z 2025-08-26T20:36:29.2079718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2079958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2080032Z return mod(**inputs) 2025-08-26T20:36:29.2080368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2080444Z outputs = self.bert( 2025-08-26T20:36:29.2080773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2080856Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2081187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2081273Z layer_outputs = layer_module( 2025-08-26T20:36:29.2081514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2081607Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2081920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2082013Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2082341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2082420Z self_outputs = self.self( 2025-08-26T20:36:29.2082701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2082780Z return func(*args, **kwargs) 2025-08-26T20:36:29.2083108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2083199Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2083203Z 2025-08-26T20:36:29.2083292Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2083388Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2083501Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2083728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2083803Z return mod(**inputs) 2025-08-26T20:36:29.2084125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2084218Z outputs = self.bert( 2025-08-26T20:36:29.2084577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2084663Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2084991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2085069Z layer_outputs = layer_module( 2025-08-26T20:36:29.2085314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2085406Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2085744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2085844Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2086163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2086358Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2086678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2086792Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2086796Z 2025-08-26T20:36:29.2086919Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2087141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2087219Z return mod(**inputs) 2025-08-26T20:36:29.2087545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2087617Z outputs = self.bert( 2025-08-26T20:36:29.2087942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2088023Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2088344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2088426Z layer_outputs = layer_module( 2025-08-26T20:36:29.2088671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2088764Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2089084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2089185Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2089475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2089565Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2089923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2090040Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2090364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2090452Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2090456Z 2025-08-26T20:36:29.2090578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2090850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2090944Z return mod(**inputs) 2025-08-26T20:36:29.2091318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2091414Z outputs = self.bert( 2025-08-26T20:36:29.2091740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2091821Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2092144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2092220Z layer_outputs = layer_module( 2025-08-26T20:36:29.2092464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2092554Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2092903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2093005Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2093297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2093403Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2093762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2093898Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2094218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2094343Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2094585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2094673Z return self.act(input) 2025-08-26T20:36:29.2094677Z 2025-08-26T20:36:29.2094786Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2095010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2095081Z return mod(**inputs) 2025-08-26T20:36:29.2095401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2095473Z outputs = self.bert( 2025-08-26T20:36:29.2095780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2095865Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2096399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2096536Z layer_outputs = layer_module( 2025-08-26T20:36:29.2096798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2096896Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2097209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2097302Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2097592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2097675Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2098022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2098167Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2098477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2098577Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2098637Z 2025-08-26T20:36:29.2098750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2098977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2099050Z return mod(**inputs) 2025-08-26T20:36:29.2099369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2099441Z outputs = self.bert( 2025-08-26T20:36:29.2099821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2099941Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2100258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2100345Z layer_outputs = layer_module( 2025-08-26T20:36:29.2100640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2100730Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2101053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2101175Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2101467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2101549Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2101923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2102064Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2102367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.2102460Z return input_tensor + hidden_states 2025-08-26T20:36:29.2102464Z 2025-08-26T20:36:29.2102575Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2102803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2102873Z return mod(**inputs) 2025-08-26T20:36:29.2103181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2103259Z outputs = self.bert( 2025-08-26T20:36:29.2103568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2103655Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2103962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2104046Z layer_outputs = layer_module( 2025-08-26T20:36:29.2104285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2104371Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2104685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2104771Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2105084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2105163Z self_outputs = self.self( 2025-08-26T20:36:29.2105426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2105510Z return func(*args, **kwargs) 2025-08-26T20:36:29.2105842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2105942Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2105948Z 2025-08-26T20:36:29.2106058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2106283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2106353Z return mod(**inputs) 2025-08-26T20:36:29.2106673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2106768Z outputs = self.bert( 2025-08-26T20:36:29.2107076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2107161Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2107490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2107565Z layer_outputs = layer_module( 2025-08-26T20:36:29.2107828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2107914Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2108225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2108314Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2108629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2108704Z self_outputs = self.self( 2025-08-26T20:36:29.2108965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2109048Z return func(*args, **kwargs) 2025-08-26T20:36:29.2109356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2109446Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2109450Z 2025-08-26T20:36:29.2109561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2109773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2109850Z return mod(**inputs) 2025-08-26T20:36:29.2110161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2110236Z outputs = self.bert( 2025-08-26T20:36:29.2110539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2110619Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2110932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2111010Z layer_outputs = layer_module( 2025-08-26T20:36:29.2111250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2111337Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2111650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2111737Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2112038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2112120Z self_outputs = self.self( 2025-08-26T20:36:29.2112399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2112484Z return func(*args, **kwargs) 2025-08-26T20:36:29.2112796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2112883Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2112895Z 2025-08-26T20:36:29.2112980Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2113065Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2113182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2113415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2113496Z return mod(**inputs) 2025-08-26T20:36:29.2113804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2113898Z outputs = self.bert( 2025-08-26T20:36:29.2114226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2114323Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2114637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2114713Z layer_outputs = layer_module( 2025-08-26T20:36:29.2114951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2115041Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2115355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2115447Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2115758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2115899Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2116216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2116304Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2116308Z 2025-08-26T20:36:29.2116424Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2116638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2116718Z return mod(**inputs) 2025-08-26T20:36:29.2117031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2117100Z outputs = self.bert( 2025-08-26T20:36:29.2117427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2117506Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2117830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2117908Z layer_outputs = layer_module( 2025-08-26T20:36:29.2118156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2118247Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2118564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2118803Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2119125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2119224Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2119642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2119770Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2120097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2120187Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2120192Z 2025-08-26T20:36:29.2120311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2120552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2120642Z return mod(**inputs) 2025-08-26T20:36:29.2120937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2121021Z outputs = self.bert( 2025-08-26T20:36:29.2121321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2121414Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2121712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2121790Z layer_outputs = layer_module( 2025-08-26T20:36:29.2122031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2122127Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2122435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2122534Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2122816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2122898Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2123247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2123358Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2123689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2123806Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2124031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2124103Z return self.act(input) 2025-08-26T20:36:29.2124108Z 2025-08-26T20:36:29.2124216Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2124428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2124495Z return mod(**inputs) 2025-08-26T20:36:29.2124795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2124862Z outputs = self.bert( 2025-08-26T20:36:29.2125150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2125232Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2125529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2125608Z layer_outputs = layer_module( 2025-08-26T20:36:29.2125851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2125942Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2126229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2126313Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2126582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2126658Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2127017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2127155Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2127446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2127551Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2127555Z 2025-08-26T20:36:29.2127657Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2127879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2127945Z return mod(**inputs) 2025-08-26T20:36:29.2128252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2128322Z outputs = self.bert( 2025-08-26T20:36:29.2128632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2128716Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2129019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2129105Z layer_outputs = layer_module( 2025-08-26T20:36:29.2129341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2129427Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2129740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2129828Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2130144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2130222Z self_outputs = self.self( 2025-08-26T20:36:29.2130491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2130568Z return func(*args, **kwargs) 2025-08-26T20:36:29.2130874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2130974Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2130978Z 2025-08-26T20:36:29.2131089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2131310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2131379Z return mod(**inputs) 2025-08-26T20:36:29.2131686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2131765Z outputs = self.bert( 2025-08-26T20:36:29.2132071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2132158Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2132483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2132574Z layer_outputs = layer_module( 2025-08-26T20:36:29.2132811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2132896Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2133210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2133296Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2133625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2133702Z self_outputs = self.self( 2025-08-26T20:36:29.2133960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2134076Z return func(*args, **kwargs) 2025-08-26T20:36:29.2134390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2134502Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2134507Z 2025-08-26T20:36:29.2134618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2134837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2134907Z return mod(**inputs) 2025-08-26T20:36:29.2135211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2135289Z outputs = self.bert( 2025-08-26T20:36:29.2135597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2135680Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2135991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2136066Z layer_outputs = layer_module( 2025-08-26T20:36:29.2136311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2136392Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2136701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2136786Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2137097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2137173Z self_outputs = self.self( 2025-08-26T20:36:29.2137430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2137514Z return func(*args, **kwargs) 2025-08-26T20:36:29.2137815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2137908Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2137911Z 2025-08-26T20:36:29.2137999Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2138085Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2138203Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2138416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2138494Z return mod(**inputs) 2025-08-26T20:36:29.2138801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2138871Z outputs = self.bert( 2025-08-26T20:36:29.2139201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2139282Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2139600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2139678Z layer_outputs = layer_module( 2025-08-26T20:36:29.2139923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2140006Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2140330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2140426Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2140734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2140897Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2141203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2141309Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2141321Z 2025-08-26T20:36:29.2141432Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2141644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2141731Z return mod(**inputs) 2025-08-26T20:36:29.2142024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2142097Z outputs = self.bert( 2025-08-26T20:36:29.2142386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2142462Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2142760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2142836Z layer_outputs = layer_module( 2025-08-26T20:36:29.2143078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2143160Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2143471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2143568Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2143848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2143940Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2144281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2144403Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2144708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2144796Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2144799Z 2025-08-26T20:36:29.2144919Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2145123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2145197Z return mod(**inputs) 2025-08-26T20:36:29.2145490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2145579Z outputs = self.bert( 2025-08-26T20:36:29.2145876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2145952Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2146266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2146342Z layer_outputs = layer_module( 2025-08-26T20:36:29.2146588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2146674Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2146997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2147099Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2147384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2147497Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2147848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2147981Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2148305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2148427Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2148665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2148744Z return self.act(input) 2025-08-26T20:36:29.2148748Z 2025-08-26T20:36:29.2148863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2149078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2149149Z return mod(**inputs) 2025-08-26T20:36:29.2149466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2149538Z outputs = self.bert( 2025-08-26T20:36:29.2149853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2149931Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2150234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2150316Z layer_outputs = layer_module( 2025-08-26T20:36:29.2150538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2150628Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2150914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2151009Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2151271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2151348Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2151676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2151837Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2152180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2152313Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2152317Z 2025-08-26T20:36:29.2152423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2152632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2152702Z return mod(**inputs) 2025-08-26T20:36:29.2153003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2153071Z outputs = self.bert( 2025-08-26T20:36:29.2153370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2153462Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2153767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2153850Z layer_outputs = layer_module( 2025-08-26T20:36:29.2154106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2154195Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2154518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2154607Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2154891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2154972Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2155323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2155469Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2155794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.2155882Z return input_tensor + hidden_states 2025-08-26T20:36:29.2155888Z 2025-08-26T20:36:29.2156003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2156229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2156302Z return mod(**inputs) 2025-08-26T20:36:29.2156625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2156698Z outputs = self.bert( 2025-08-26T20:36:29.2157014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2157102Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2157420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2157508Z layer_outputs = layer_module( 2025-08-26T20:36:29.2157752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2157848Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2158164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2158253Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2158575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2158655Z self_outputs = self.self( 2025-08-26T20:36:29.2158930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2159027Z return func(*args, **kwargs) 2025-08-26T20:36:29.2159347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2159523Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2159530Z 2025-08-26T20:36:29.2159647Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2159875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2159948Z return mod(**inputs) 2025-08-26T20:36:29.2160301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2160377Z outputs = self.bert( 2025-08-26T20:36:29.2160693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2160784Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2161124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2161209Z layer_outputs = layer_module( 2025-08-26T20:36:29.2161476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2161561Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2161888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2161978Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2162301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2162378Z self_outputs = self.self( 2025-08-26T20:36:29.2162645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2162730Z return func(*args, **kwargs) 2025-08-26T20:36:29.2163045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2163140Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2163144Z 2025-08-26T20:36:29.2163256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2163479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2163551Z return mod(**inputs) 2025-08-26T20:36:29.2163871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2163950Z outputs = self.bert( 2025-08-26T20:36:29.2164265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2164355Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2164668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2164749Z layer_outputs = layer_module( 2025-08-26T20:36:29.2165000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2165085Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2165410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2165499Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2165823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2165900Z self_outputs = self.self( 2025-08-26T20:36:29.2166186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2166274Z return func(*args, **kwargs) 2025-08-26T20:36:29.2166593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2166690Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2166694Z 2025-08-26T20:36:29.2166783Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2166870Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2166995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2167235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2167315Z return mod(**inputs) 2025-08-26T20:36:29.2167636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2167729Z outputs = self.bert( 2025-08-26T20:36:29.2168051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2168150Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2168484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2168564Z layer_outputs = layer_module( 2025-08-26T20:36:29.2168825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2168912Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2169228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2169323Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2169643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2169790Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2170106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2170192Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2170203Z 2025-08-26T20:36:29.2170313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2170535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2170613Z return mod(**inputs) 2025-08-26T20:36:29.2170930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2171007Z outputs = self.bert( 2025-08-26T20:36:29.2171326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2171405Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2171734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2171810Z layer_outputs = layer_module( 2025-08-26T20:36:29.2172061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2172144Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2172462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2172559Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2172865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2172961Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2173300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2173420Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2173728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2173815Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2173819Z 2025-08-26T20:36:29.2173957Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2174173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2174250Z return mod(**inputs) 2025-08-26T20:36:29.2174559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2175589Z outputs = self.bert( 2025-08-26T20:36:29.2175908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2176037Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2176360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2176440Z layer_outputs = layer_module( 2025-08-26T20:36:29.2176693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2176778Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2177094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2177198Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2177486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2177590Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2177931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2178043Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2178357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2178479Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2178714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2178790Z return self.act(input) 2025-08-26T20:36:29.2178796Z 2025-08-26T20:36:29.2178917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2179133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2179207Z return mod(**inputs) 2025-08-26T20:36:29.2179529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2179599Z outputs = self.bert( 2025-08-26T20:36:29.2179912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2179991Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2180299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2180383Z layer_outputs = layer_module( 2025-08-26T20:36:29.2180639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2180736Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2181049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2181146Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2181424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2181505Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2181875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2182019Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2182338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2182443Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2182447Z 2025-08-26T20:36:29.2182567Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2182803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2182873Z return mod(**inputs) 2025-08-26T20:36:29.2183188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2183258Z outputs = self.bert( 2025-08-26T20:36:29.2183576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2183687Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2183997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2184085Z layer_outputs = layer_module( 2025-08-26T20:36:29.2184324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2184418Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2184729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2184817Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2185134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2185213Z self_outputs = self.self( 2025-08-26T20:36:29.2185483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2185561Z return func(*args, **kwargs) 2025-08-26T20:36:29.2185876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2185966Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2185971Z 2025-08-26T20:36:29.2186083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2186307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2186378Z return mod(**inputs) 2025-08-26T20:36:29.2186695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2186764Z outputs = self.bert( 2025-08-26T20:36:29.2187075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2187161Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2187489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2187578Z layer_outputs = layer_module( 2025-08-26T20:36:29.2187815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2187907Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2188220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2188310Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2188644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2188721Z self_outputs = self.self( 2025-08-26T20:36:29.2188989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2189086Z return func(*args, **kwargs) 2025-08-26T20:36:29.2189394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2189510Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2189514Z 2025-08-26T20:36:29.2189625Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2189843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2189912Z return mod(**inputs) 2025-08-26T20:36:29.2190228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2190300Z outputs = self.bert( 2025-08-26T20:36:29.2190608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2190698Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2191006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2191091Z layer_outputs = layer_module( 2025-08-26T20:36:29.2191330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2191413Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2191728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2191814Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2192129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2192203Z self_outputs = self.self( 2025-08-26T20:36:29.2192465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2192548Z return func(*args, **kwargs) 2025-08-26T20:36:29.2192853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2192945Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2192949Z 2025-08-26T20:36:29.2193036Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2193128Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2193241Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2193454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2193532Z return mod(**inputs) 2025-08-26T20:36:29.2193836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2193934Z outputs = self.bert( 2025-08-26T20:36:29.2194240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2194320Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2194634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2194710Z layer_outputs = layer_module( 2025-08-26T20:36:29.2194954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2195035Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2195388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2195483Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2195792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2195962Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2196523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2196669Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2196675Z 2025-08-26T20:36:29.2196829Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2197043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2197124Z return mod(**inputs) 2025-08-26T20:36:29.2197441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2197521Z outputs = self.bert( 2025-08-26T20:36:29.2197842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2197933Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2198250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2198330Z layer_outputs = layer_module( 2025-08-26T20:36:29.2198585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2198668Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2198992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2199085Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2199376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2199531Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2199888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2200014Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2200330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2200428Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2200432Z 2025-08-26T20:36:29.2200545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2200764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2200845Z return mod(**inputs) 2025-08-26T20:36:29.2201235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2201320Z outputs = self.bert( 2025-08-26T20:36:29.2201635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2201716Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2202039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2202117Z layer_outputs = layer_module( 2025-08-26T20:36:29.2202373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2202489Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2202806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2202908Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2203242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2203336Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2203715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2203836Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2204153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2204281Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2204524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2204602Z return self.act(input) 2025-08-26T20:36:29.2204606Z 2025-08-26T20:36:29.2204728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2204949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2205021Z return mod(**inputs) 2025-08-26T20:36:29.2205349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2205422Z outputs = self.bert( 2025-08-26T20:36:29.2205746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2205829Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2206155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2206234Z layer_outputs = layer_module( 2025-08-26T20:36:29.2206481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2206580Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2206896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2206996Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2207288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2207371Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2207739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2207882Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2208198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2208319Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2208323Z 2025-08-26T20:36:29.2208437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2208641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2208710Z return mod(**inputs) 2025-08-26T20:36:29.2209010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2209077Z outputs = self.bert( 2025-08-26T20:36:29.2209390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2209465Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2209758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2209855Z layer_outputs = layer_module( 2025-08-26T20:36:29.2210081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2210168Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2210478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2210567Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2210836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2210912Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2211243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2211374Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2211677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.2211756Z return input_tensor + hidden_states 2025-08-26T20:36:29.2211761Z 2025-08-26T20:36:29.2211871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2212080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2212149Z return mod(**inputs) 2025-08-26T20:36:29.2212470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2212541Z outputs = self.bert( 2025-08-26T20:36:29.2212857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2212934Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2213245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2213328Z layer_outputs = layer_module( 2025-08-26T20:36:29.2213570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2213661Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2213981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2214063Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2214364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2214436Z self_outputs = self.self( 2025-08-26T20:36:29.2214691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2214782Z return func(*args, **kwargs) 2025-08-26T20:36:29.2215083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2215167Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2215171Z 2025-08-26T20:36:29.2215275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2215484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2215548Z return mod(**inputs) 2025-08-26T20:36:29.2215865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2215933Z outputs = self.bert( 2025-08-26T20:36:29.2216228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2216330Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2216620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2216722Z layer_outputs = layer_module( 2025-08-26T20:36:29.2216947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2217035Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2217329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2217413Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2217713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2217784Z self_outputs = self.self( 2025-08-26T20:36:29.2218042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2218114Z return func(*args, **kwargs) 2025-08-26T20:36:29.2218405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2218494Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2218498Z 2025-08-26T20:36:29.2218603Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2218812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2218881Z return mod(**inputs) 2025-08-26T20:36:29.2219188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2219255Z outputs = self.bert( 2025-08-26T20:36:29.2219548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2219632Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2219926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2220006Z layer_outputs = layer_module( 2025-08-26T20:36:29.2220229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2220307Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2220610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2220693Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2221006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2221095Z self_outputs = self.self( 2025-08-26T20:36:29.2221348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2221419Z return func(*args, **kwargs) 2025-08-26T20:36:29.2221712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2221800Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2221804Z 2025-08-26T20:36:29.2221886Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2221974Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2222079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2222340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2222430Z return mod(**inputs) 2025-08-26T20:36:29.2222730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2222826Z outputs = self.bert( 2025-08-26T20:36:29.2223138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2223234Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2223544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2223616Z layer_outputs = layer_module( 2025-08-26T20:36:29.2223847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2223928Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2224224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2224307Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2224597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2224738Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2225026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2225116Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2225119Z 2025-08-26T20:36:29.2225223Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2225425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2225497Z return mod(**inputs) 2025-08-26T20:36:29.2225788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2225863Z outputs = self.bert( 2025-08-26T20:36:29.2226152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2226234Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2226521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2226592Z layer_outputs = layer_module( 2025-08-26T20:36:29.2226823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2226901Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2227194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2227284Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2227602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2227698Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2228055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2228174Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2228483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2228575Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2228579Z 2025-08-26T20:36:29.2228713Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2228929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2229006Z return mod(**inputs) 2025-08-26T20:36:29.2229323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2229413Z outputs = self.bert( 2025-08-26T20:36:29.2229706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2229801Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2230115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2230190Z layer_outputs = layer_module( 2025-08-26T20:36:29.2230436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2230522Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2230837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2230930Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2231208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2231298Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2231639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2231755Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2232066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2232189Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2232426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2232501Z return self.act(input) 2025-08-26T20:36:29.2232508Z 2025-08-26T20:36:29.2232625Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2232840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2232917Z return mod(**inputs) 2025-08-26T20:36:29.2233225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2233298Z outputs = self.bert( 2025-08-26T20:36:29.2233617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2233696Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2234011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2234088Z layer_outputs = layer_module( 2025-08-26T20:36:29.2234346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2234439Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2234753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2234850Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2235125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2235211Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2235564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2235709Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2236029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2236139Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2236143Z 2025-08-26T20:36:29.2236285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2236504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2236577Z return mod(**inputs) 2025-08-26T20:36:29.2236903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2236975Z outputs = self.bert( 2025-08-26T20:36:29.2237302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2237384Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2237704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2237783Z layer_outputs = layer_module( 2025-08-26T20:36:29.2238025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2238121Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2238437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2238534Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2238856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2238931Z self_outputs = self.self( 2025-08-26T20:36:29.2239198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2239275Z return func(*args, **kwargs) 2025-08-26T20:36:29.2239681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2239775Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2239782Z 2025-08-26T20:36:29.2239901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2240119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2240190Z return mod(**inputs) 2025-08-26T20:36:29.2240517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2240595Z outputs = self.bert( 2025-08-26T20:36:29.2240918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2240998Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2241342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2241442Z layer_outputs = layer_module( 2025-08-26T20:36:29.2241685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2241779Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2242089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2242179Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2242521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2242599Z self_outputs = self.self( 2025-08-26T20:36:29.2242870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2242964Z return func(*args, **kwargs) 2025-08-26T20:36:29.2243280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2243388Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2243392Z 2025-08-26T20:36:29.2243501Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2243723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2243791Z return mod(**inputs) 2025-08-26T20:36:29.2244107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2244177Z outputs = self.bert( 2025-08-26T20:36:29.2244486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2244569Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2244859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2244937Z layer_outputs = layer_module( 2025-08-26T20:36:29.2245167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2245249Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2245529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2245609Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2245901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2245967Z self_outputs = self.self( 2025-08-26T20:36:29.2246214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2246282Z return func(*args, **kwargs) 2025-08-26T20:36:29.2246562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2246648Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2246652Z 2025-08-26T20:36:29.2246732Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2246819Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2246923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2247134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2247210Z return mod(**inputs) 2025-08-26T20:36:29.2247515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2247621Z outputs = self.bert( 2025-08-26T20:36:29.2247914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2247996Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2248286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2248358Z layer_outputs = layer_module( 2025-08-26T20:36:29.2248593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2248689Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2248986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2249067Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2249378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2249515Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2249827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2249916Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2249920Z 2025-08-26T20:36:29.2250023Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2250233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2250301Z return mod(**inputs) 2025-08-26T20:36:29.2250595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2250669Z outputs = self.bert( 2025-08-26T20:36:29.2250972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2251051Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2251337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2251408Z layer_outputs = layer_module( 2025-08-26T20:36:29.2251642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2251721Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2252023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2252108Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2252381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2252460Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2252792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2252905Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2253189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2253276Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2253279Z 2025-08-26T20:36:29.2253381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2253586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2253650Z return mod(**inputs) 2025-08-26T20:36:29.2253964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2254040Z outputs = self.bert( 2025-08-26T20:36:29.2254320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2254399Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2254687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2254758Z layer_outputs = layer_module( 2025-08-26T20:36:29.2254990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2255086Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2255392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2255478Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2255762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2255867Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2256190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2256302Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2256592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2256714Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2256933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2257006Z return self.act(input) 2025-08-26T20:36:29.2257010Z 2025-08-26T20:36:29.2257125Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2257326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2257401Z return mod(**inputs) 2025-08-26T20:36:29.2257696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2257763Z outputs = self.bert( 2025-08-26T20:36:29.2258060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2258134Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2258443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2258514Z layer_outputs = layer_module( 2025-08-26T20:36:29.2258746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2258823Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2259103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2259194Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2259448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2259532Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2259846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2259976Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2260293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2260377Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2260381Z 2025-08-26T20:36:29.2260491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2260693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2260766Z return mod(**inputs) 2025-08-26T20:36:29.2261057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2261123Z outputs = self.bert( 2025-08-26T20:36:29.2261437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2261514Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2261813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2261906Z layer_outputs = layer_module( 2025-08-26T20:36:29.2262135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2262244Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2262546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2262637Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2262897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2262984Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2263295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2263427Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2263723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.2263807Z return input_tensor + hidden_states 2025-08-26T20:36:29.2263810Z 2025-08-26T20:36:29.2263921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2264118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2264186Z return mod(**inputs) 2025-08-26T20:36:29.2264481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2264551Z outputs = self.bert( 2025-08-26T20:36:29.2264841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2264917Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2265215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2265289Z layer_outputs = layer_module( 2025-08-26T20:36:29.2265513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2265610Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2265918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2266019Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2266339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2266419Z self_outputs = self.self( 2025-08-26T20:36:29.2266707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2266787Z return func(*args, **kwargs) 2025-08-26T20:36:29.2267106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2267196Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2267199Z 2025-08-26T20:36:29.2267318Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2267529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2267599Z return mod(**inputs) 2025-08-26T20:36:29.2267935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2268009Z outputs = self.bert( 2025-08-26T20:36:29.2268323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2268424Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2268732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2268837Z layer_outputs = layer_module( 2025-08-26T20:36:29.2269078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2269169Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2269475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2269572Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2269880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2269955Z self_outputs = self.self( 2025-08-26T20:36:29.2270228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2270302Z return func(*args, **kwargs) 2025-08-26T20:36:29.2270619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2270704Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2270708Z 2025-08-26T20:36:29.2270818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2271036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2271106Z return mod(**inputs) 2025-08-26T20:36:29.2271423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2271493Z outputs = self.bert( 2025-08-26T20:36:29.2271811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2271890Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2272199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2272284Z layer_outputs = layer_module( 2025-08-26T20:36:29.2272519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2272611Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2272926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2273013Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2273327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2273424Z self_outputs = self.self( 2025-08-26T20:36:29.2273705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2273787Z return func(*args, **kwargs) 2025-08-26T20:36:29.2274108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2274205Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2274209Z 2025-08-26T20:36:29.2274297Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2274393Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2274524Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2274755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2274827Z return mod(**inputs) 2025-08-26T20:36:29.2275149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2275250Z outputs = self.bert( 2025-08-26T20:36:29.2275574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2275689Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2276012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2276090Z layer_outputs = layer_module( 2025-08-26T20:36:29.2276350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2276435Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2276762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2276852Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2277178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2277324Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2277639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2277741Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2277745Z 2025-08-26T20:36:29.2277857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2278081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2278154Z return mod(**inputs) 2025-08-26T20:36:29.2278475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2278558Z outputs = self.bert( 2025-08-26T20:36:29.2278877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2278965Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2279286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2279372Z layer_outputs = layer_module( 2025-08-26T20:36:29.2279719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2279817Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2280148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2280240Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2280569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2280657Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2281011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2281136Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2281456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2281555Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2281577Z 2025-08-26T20:36:29.2281693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2281924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2281997Z return mod(**inputs) 2025-08-26T20:36:29.2282344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2282424Z outputs = self.bert( 2025-08-26T20:36:29.2282758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2282845Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2283157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2283238Z layer_outputs = layer_module( 2025-08-26T20:36:29.2283493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2283580Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2283907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2284000Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2284294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2284379Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2284731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2284851Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2285168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2285299Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2285536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2285617Z return self.act(input) 2025-08-26T20:36:29.2285621Z 2025-08-26T20:36:29.2285743Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2285962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2286043Z return mod(**inputs) 2025-08-26T20:36:29.2286363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2286436Z outputs = self.bert( 2025-08-26T20:36:29.2286763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2286844Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2287179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2287257Z layer_outputs = layer_module( 2025-08-26T20:36:29.2287529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2287617Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2287938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2288038Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2288328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2288419Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2288788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2288944Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2289261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2289370Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2289390Z 2025-08-26T20:36:29.2289515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2289735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2289816Z return mod(**inputs) 2025-08-26T20:36:29.2290137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2290211Z outputs = self.bert( 2025-08-26T20:36:29.2290549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2290631Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2290963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2291042Z layer_outputs = layer_module( 2025-08-26T20:36:29.2291288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2291385Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2291704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2291800Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2292119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2292204Z self_outputs = self.self( 2025-08-26T20:36:29.2292481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2292561Z return func(*args, **kwargs) 2025-08-26T20:36:29.2292878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-26T20:36:29.2292966Z query_layer = self.query(hidden_states) 2025-08-26T20:36:29.2292970Z 2025-08-26T20:36:29.2293088Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2293298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2293368Z return mod(**inputs) 2025-08-26T20:36:29.2293686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2293755Z outputs = self.bert( 2025-08-26T20:36:29.2294070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2294163Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2294480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2294557Z layer_outputs = layer_module( 2025-08-26T20:36:29.2294794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2294887Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2295202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2295298Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2295641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2295717Z self_outputs = self.self( 2025-08-26T20:36:29.2295980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2296078Z return func(*args, **kwargs) 2025-08-26T20:36:29.2296719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-26T20:36:29.2296888Z key_layer = self.key(current_states) 2025-08-26T20:36:29.2296892Z 2025-08-26T20:36:29.2297016Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2297237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2297309Z return mod(**inputs) 2025-08-26T20:36:29.2297637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2297710Z outputs = self.bert( 2025-08-26T20:36:29.2298036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2298129Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2298442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2298528Z layer_outputs = layer_module( 2025-08-26T20:36:29.2298768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2298860Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2299169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2299265Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2299573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-26T20:36:29.2299649Z self_outputs = self.self( 2025-08-26T20:36:29.2299917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:36:29.2299993Z return func(*args, **kwargs) 2025-08-26T20:36:29.2300309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-26T20:36:29.2300394Z value_layer = self.value(current_states) 2025-08-26T20:36:29.2300398Z 2025-08-26T20:36:29.2300485Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2300580Z cudagraph partition due to non gpu ops 2025-08-26T20:36:29.2300692Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2300912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2300982Z return mod(**inputs) 2025-08-26T20:36:29.2301350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2301433Z outputs = self.bert( 2025-08-26T20:36:29.2301741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2301829Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2302136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2302218Z layer_outputs = layer_module( 2025-08-26T20:36:29.2302454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2302563Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2302879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-26T20:36:29.2302966Z self_attention_outputs = self.attention( 2025-08-26T20:36:29.2303324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-26T20:36:29.2303461Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:36:29.2303787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-26T20:36:29.2303881Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2303885Z 2025-08-26T20:36:29.2303997Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2304217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2304288Z return mod(**inputs) 2025-08-26T20:36:29.2304605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2304678Z outputs = self.bert( 2025-08-26T20:36:29.2304985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2305071Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2305377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2305460Z layer_outputs = layer_module( 2025-08-26T20:36:29.2305696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2305778Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2306094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2306182Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2306467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2306552Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2306907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2307015Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2307303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-26T20:36:29.2307392Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2307396Z 2025-08-26T20:36:29.2307502Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2307709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2307775Z return mod(**inputs) 2025-08-26T20:36:29.2308081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2308159Z outputs = self.bert( 2025-08-26T20:36:29.2308451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2308536Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2308826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2308906Z layer_outputs = layer_module( 2025-08-26T20:36:29.2309151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2309233Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2309534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2309638Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2309906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2310000Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2310321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-26T20:36:29.2310433Z intermediate_output = self.intermediate(ln_output) 2025-08-26T20:36:29.2310725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-26T20:36:29.2310847Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:36:29.2311062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:36:29.2311141Z return self.act(input) 2025-08-26T20:36:29.2311148Z 2025-08-26T20:36:29.2311252Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2311452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2311527Z return mod(**inputs) 2025-08-26T20:36:29.2311820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2311894Z outputs = self.bert( 2025-08-26T20:36:29.2312186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2312258Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2312562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2312632Z layer_outputs = layer_module( 2025-08-26T20:36:29.2312864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2312945Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2313239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2313323Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2313590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2313672Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2313997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2314145Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2314476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-26T20:36:29.2314565Z hidden_states = self.dense(hidden_states) 2025-08-26T20:36:29.2314575Z 2025-08-26T20:36:29.2314694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2314895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2314969Z return mod(**inputs) 2025-08-26T20:36:29.2315268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-26T20:36:29.2315344Z outputs = self.bert( 2025-08-26T20:36:29.2315652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-26T20:36:29.2315728Z encoder_outputs = self.encoder( 2025-08-26T20:36:29.2316036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-26T20:36:29.2316130Z layer_outputs = layer_module( 2025-08-26T20:36:29.2316379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:36:29.2316481Z return super().__call__(*args, **kwargs) 2025-08-26T20:36:29.2316808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-26T20:36:29.2316907Z layer_output = apply_chunking_to_forward( 2025-08-26T20:36:29.2317200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:36:29.2317293Z return forward_fn(*input_tensors) 2025-08-26T20:36:29.2317662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-26T20:36:29.2317815Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:36:29.2318143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-26T20:36:29.2318231Z return input_tensor + hidden_states 2025-08-26T20:36:29.2318235Z 2025-08-26T20:36:29.2318357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2318573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2318652Z return mod(**inputs) 2025-08-26T20:36:29.2318984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1611, in forward 2025-08-26T20:36:29.2319077Z logits = self.qa_outputs(sequence_output) 2025-08-26T20:36:29.2319089Z 2025-08-26T20:36:29.2319201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2319476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2319568Z return mod(**inputs) 2025-08-26T20:36:29.2319897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1629, in forward 2025-08-26T20:36:29.2320022Z start_loss = loss_fct(start_logits, start_positions) 2025-08-26T20:36:29.2320027Z 2025-08-26T20:36:29.2320139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:36:29.2320354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:36:29.2320433Z return mod(**inputs) 2025-08-26T20:36:29.2320788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1630, in forward 2025-08-26T20:36:29.2320892Z end_loss = loss_fct(end_logits, end_positions) 2025-08-26T20:36:29.2320896Z 2025-08-26T20:36:39.3920486Z Compilation time (from dynamo_timed): 23.705046264 2025-08-26T20:36:39.3921897Z pass 2025-08-26T20:36:39.3922228Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:36:39.3923113Z TIMING: _recursive_pre_grad_passes:0.01152 _recursive_joint_graph_passes:1.17146 _recursive_post_grad_passes:0.13668 async_compile.wait:0.00343 code_gen:8.83083 inductor_compile:11.25438 backend_compile:17.91533 gc:0.00155 entire_frame_compile:23.70505 total_wall_time:23.70505 2025-08-26T20:36:39.3924107Z STATS: call_* op count: 724 | FakeTensorMode.__torch_dispatch__:28470 | FakeTensor.__torch_dispatch__:8283 | ProxyTorchDispatchMode.__torch_dispatch__:10973 2025-08-26T20:36:39.3930840Z Dynamo produced 1 graphs covering 724 ops with 0 graph breaks (0 unique) 2025-08-26T20:36:45.2197339Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:36:45.2198676Z from pkg_resources import resource_filename 2025-08-26T20:36:45.8245247Z 2025-08-26T20:36:46.5848054Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:36:46.5848749Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:36:46.5919984Z cpu eval MobileBertForMaskedLM 2025-08-26T20:36:46.8681427Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:36:47.0383054Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:36:47.2045611Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:37:14.9725954Z cudagraph partition due to non gpu ops 2025-08-26T20:37:14.9732100Z cudagraph partition due to non gpu ops 2025-08-26T20:37:14.9734640Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9735177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9735590Z return mod(**inputs) 2025-08-26T20:37:14.9736215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9738966Z outputs = self.mobilebert( 2025-08-26T20:37:14.9739887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-26T20:37:14.9740578Z embedding_output = self.embeddings( 2025-08-26T20:37:14.9741198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-08-26T20:37:14.9741709Z inputs_embeds = torch.cat( 2025-08-26T20:37:14.9741857Z 2025-08-26T20:37:14.9741990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9742426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9742801Z return mod(**inputs) 2025-08-26T20:37:14.9743244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-26T20:37:14.9743743Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:37:14.9744231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-26T20:37:14.9744738Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:37:14.9745237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-08-26T20:37:14.9745830Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-08-26T20:37:14.9746107Z 2025-08-26T20:37:14.9746221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9746880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9747223Z return mod(**inputs) 2025-08-26T20:37:14.9747618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9759884Z outputs = self.mobilebert( 2025-08-26T20:37:14.9760436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-26T20:37:14.9760917Z embedding_output = self.embeddings( 2025-08-26T20:37:14.9761556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-08-26T20:37:14.9762101Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-08-26T20:37:14.9762304Z 2025-08-26T20:37:14.9762442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9762942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9763313Z return mod(**inputs) 2025-08-26T20:37:14.9763773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9764296Z outputs = self.mobilebert( 2025-08-26T20:37:14.9764759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-26T20:37:14.9765230Z embedding_output = self.embeddings( 2025-08-26T20:37:14.9765713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-08-26T20:37:14.9766196Z embeddings = self.LayerNorm(embeddings) 2025-08-26T20:37:14.9766672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9767168Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9767337Z 2025-08-26T20:37:14.9767457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9767864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9768237Z return mod(**inputs) 2025-08-26T20:37:14.9768712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9769171Z outputs = self.mobilebert( 2025-08-26T20:37:14.9769616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9770111Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9770562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9771022Z layer_outputs = layer_module( 2025-08-26T20:37:14.9771469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:14.9772266Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:14.9772808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:14.9773285Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:14.9773771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:14.9774220Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:14.9774369Z 2025-08-26T20:37:14.9774491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9774880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9775260Z return mod(**inputs) 2025-08-26T20:37:14.9775683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9776124Z outputs = self.mobilebert( 2025-08-26T20:37:14.9776553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9776992Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9777423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9777895Z layer_outputs = layer_module( 2025-08-26T20:37:14.9778336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9778813Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9779269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:14.9779731Z self_outputs = self.self( 2025-08-26T20:37:14.9780156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:14.9780622Z self.value(value_tensor) 2025-08-26T20:37:14.9780751Z 2025-08-26T20:37:14.9780875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9781273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9781620Z return mod(**inputs) 2025-08-26T20:37:14.9782047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9782498Z outputs = self.mobilebert( 2025-08-26T20:37:14.9782937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9783385Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9783842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9784282Z layer_outputs = layer_module( 2025-08-26T20:37:14.9784711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:14.9785250Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:14.9785783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:14.9786264Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:14.9786741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:14.9787190Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:14.9787338Z 2025-08-26T20:37:14.9787457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9787827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9788159Z return mod(**inputs) 2025-08-26T20:37:14.9788575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9789009Z outputs = self.mobilebert( 2025-08-26T20:37:14.9789427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9789872Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9790311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9790773Z layer_outputs = layer_module( 2025-08-26T20:37:14.9791211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:14.9791741Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:14.9792277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:14.9792765Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:14.9793284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:14.9793771Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:14.9794237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9794737Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9794911Z 2025-08-26T20:37:14.9795026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9795428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9795808Z return mod(**inputs) 2025-08-26T20:37:14.9796591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9797065Z outputs = self.mobilebert( 2025-08-26T20:37:14.9797512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9797975Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9798430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9798880Z layer_outputs = layer_module( 2025-08-26T20:37:14.9799332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9799872Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9800351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:14.9800797Z self_outputs = self.self( 2025-08-26T20:37:14.9801242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:14.9801679Z self.query(query_tensor) 2025-08-26T20:37:14.9801804Z 2025-08-26T20:37:14.9801931Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9802321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9802662Z return mod(**inputs) 2025-08-26T20:37:14.9803080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9803531Z outputs = self.mobilebert( 2025-08-26T20:37:14.9803979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9804422Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9804875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9805337Z layer_outputs = layer_module( 2025-08-26T20:37:14.9805786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9806254Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9806764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:14.9807219Z self_outputs = self.self( 2025-08-26T20:37:14.9807658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:14.9808103Z self.key(key_tensor) 2025-08-26T20:37:14.9808220Z 2025-08-26T20:37:14.9808322Z cudagraph partition due to non gpu ops 2025-08-26T20:37:14.9808555Z cudagraph partition due to non gpu ops 2025-08-26T20:37:14.9808820Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9809225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9809589Z return mod(**inputs) 2025-08-26T20:37:14.9810053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9810508Z outputs = self.mobilebert( 2025-08-26T20:37:14.9810953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9811441Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9811895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9812389Z layer_outputs = layer_module( 2025-08-26T20:37:14.9812839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9813311Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9813795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:14.9814317Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:14.9814823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:14.9815302Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:14.9815470Z 2025-08-26T20:37:14.9815591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9816000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9816367Z return mod(**inputs) 2025-08-26T20:37:14.9816792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9817250Z outputs = self.mobilebert( 2025-08-26T20:37:14.9817701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9818173Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9818632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9819095Z layer_outputs = layer_module( 2025-08-26T20:37:14.9819556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9820018Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9820474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:14.9820965Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:14.9821461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:14.9821973Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:14.9822473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9822961Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9823126Z 2025-08-26T20:37:14.9823237Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9823627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9823988Z return mod(**inputs) 2025-08-26T20:37:14.9824413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9824858Z outputs = self.mobilebert( 2025-08-26T20:37:14.9825284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9825753Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9826188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9826628Z layer_outputs = layer_module( 2025-08-26T20:37:14.9827079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9827556Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9828046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:14.9828532Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:14.9829012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:14.9829459Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:14.9829619Z 2025-08-26T20:37:14.9829731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9830118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9830467Z return mod(**inputs) 2025-08-26T20:37:14.9830888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9831393Z outputs = self.mobilebert( 2025-08-26T20:37:14.9831832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9832298Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9832748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9833204Z layer_outputs = layer_module( 2025-08-26T20:37:14.9833636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9834115Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9834594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:14.9835089Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:14.9835576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:14.9836076Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:14.9836283Z 2025-08-26T20:37:14.9836397Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9836794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9837151Z return mod(**inputs) 2025-08-26T20:37:14.9837584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9838027Z outputs = self.mobilebert( 2025-08-26T20:37:14.9838505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9838963Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9839407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9839945Z layer_outputs = layer_module( 2025-08-26T20:37:14.9840395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9840879Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9841382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:14.9841901Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:14.9842405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:14.9842901Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:14.9843064Z 2025-08-26T20:37:14.9843179Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9843601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9843961Z return mod(**inputs) 2025-08-26T20:37:14.9844383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9844844Z outputs = self.mobilebert( 2025-08-26T20:37:14.9845290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9845745Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9846193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9846647Z layer_outputs = layer_module( 2025-08-26T20:37:14.9847095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9847579Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9848056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:14.9848574Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:14.9849090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:14.9849604Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:14.9850111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9850590Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9850756Z 2025-08-26T20:37:14.9850877Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9851273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9851637Z return mod(**inputs) 2025-08-26T20:37:14.9852072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9852526Z outputs = self.mobilebert( 2025-08-26T20:37:14.9852969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9853433Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9853888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9854329Z layer_outputs = layer_module( 2025-08-26T20:37:14.9854791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9855255Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9855729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:14.9856212Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:14.9856710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:14.9857168Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:14.9857342Z 2025-08-26T20:37:14.9857457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9857846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9858198Z return mod(**inputs) 2025-08-26T20:37:14.9858642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9859087Z outputs = self.mobilebert( 2025-08-26T20:37:14.9859542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9859988Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9860430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9860876Z layer_outputs = layer_module( 2025-08-26T20:37:14.9861313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9861787Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9862262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:14.9862757Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:14.9863248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:14.9863730Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:14.9863923Z 2025-08-26T20:37:14.9864041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9864441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9864801Z return mod(**inputs) 2025-08-26T20:37:14.9865229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9865688Z outputs = self.mobilebert( 2025-08-26T20:37:14.9866144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9866596Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9867034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9867494Z layer_outputs = layer_module( 2025-08-26T20:37:14.9867926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9868394Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9868864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:14.9869364Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:14.9869860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:14.9870338Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:14.9870498Z 2025-08-26T20:37:14.9870610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9871000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9871347Z return mod(**inputs) 2025-08-26T20:37:14.9871759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9872197Z outputs = self.mobilebert( 2025-08-26T20:37:14.9872647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9873093Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9873529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9874868Z layer_outputs = layer_module( 2025-08-26T20:37:14.9875323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9875833Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9876315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:14.9876832Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:14.9877336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:14.9877844Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:14.9878348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9878820Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9878989Z 2025-08-26T20:37:14.9879110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9879598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9879981Z return mod(**inputs) 2025-08-26T20:37:14.9880412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9880870Z outputs = self.mobilebert( 2025-08-26T20:37:14.9881318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9881779Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9882233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9882688Z layer_outputs = layer_module( 2025-08-26T20:37:14.9883136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9883608Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9884100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:14.9884602Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:14.9885097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:14.9885568Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:14.9885723Z 2025-08-26T20:37:14.9885841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9886248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9886610Z return mod(**inputs) 2025-08-26T20:37:14.9887069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9887523Z outputs = self.mobilebert( 2025-08-26T20:37:14.9887960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9888423Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9888871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9889309Z layer_outputs = layer_module( 2025-08-26T20:37:14.9889766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9890232Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9890703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:14.9891217Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:14.9891699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:14.9892209Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:14.9892401Z 2025-08-26T20:37:14.9892524Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9892922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9893278Z return mod(**inputs) 2025-08-26T20:37:14.9893690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9894122Z outputs = self.mobilebert( 2025-08-26T20:37:14.9894553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9894999Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9895435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9895881Z layer_outputs = layer_module( 2025-08-26T20:37:14.9896477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9896953Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9897446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:14.9897944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:14.9898452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:14.9898912Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:14.9899071Z 2025-08-26T20:37:14.9899183Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9899581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9899935Z return mod(**inputs) 2025-08-26T20:37:14.9900345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9900807Z outputs = self.mobilebert( 2025-08-26T20:37:14.9901259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9901702Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9902155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9902661Z layer_outputs = layer_module( 2025-08-26T20:37:14.9903097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:14.9903582Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:14.9904050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:14.9904573Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:14.9905084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:14.9905610Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:14.9906117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9906605Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9906835Z 2025-08-26T20:37:14.9906954Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9907358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9907766Z return mod(**inputs) 2025-08-26T20:37:14.9908196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9908654Z outputs = self.mobilebert( 2025-08-26T20:37:14.9909090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9909545Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9909978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9910417Z layer_outputs = layer_module( 2025-08-26T20:37:14.9910862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:14.9911361Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:14.9911867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:14.9912338Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:14.9912494Z 2025-08-26T20:37:14.9912616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9913014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9913372Z return mod(**inputs) 2025-08-26T20:37:14.9913814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9914269Z outputs = self.mobilebert( 2025-08-26T20:37:14.9914720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9915176Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9915625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9916080Z layer_outputs = layer_module( 2025-08-26T20:37:14.9916536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:14.9917045Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:14.9917545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:14.9918047Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:14.9918236Z 2025-08-26T20:37:14.9918354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9918776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9919147Z return mod(**inputs) 2025-08-26T20:37:14.9919635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9920107Z outputs = self.mobilebert( 2025-08-26T20:37:14.9920546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9921006Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9921490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9921937Z layer_outputs = layer_module( 2025-08-26T20:37:14.9922385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:14.9922968Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:14.9923529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:14.9924032Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:14.9924199Z 2025-08-26T20:37:14.9924316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9924715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9925077Z return mod(**inputs) 2025-08-26T20:37:14.9925510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9925958Z outputs = self.mobilebert( 2025-08-26T20:37:14.9926398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9926848Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9927295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9927745Z layer_outputs = layer_module( 2025-08-26T20:37:14.9928177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:14.9928712Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:14.9929254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:14.9929755Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:14.9930281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9930763Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9930935Z 2025-08-26T20:37:14.9931050Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9931459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9931834Z return mod(**inputs) 2025-08-26T20:37:14.9932273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9932718Z outputs = self.mobilebert( 2025-08-26T20:37:14.9933162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9933621Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9934060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9934532Z layer_outputs = layer_module( 2025-08-26T20:37:14.9934974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:14.9935505Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:14.9936065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:14.9936592Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:14.9937130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:14.9937597Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:14.9937757Z 2025-08-26T20:37:14.9937869Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9938265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9938643Z return mod(**inputs) 2025-08-26T20:37:14.9939062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9939532Z outputs = self.mobilebert( 2025-08-26T20:37:14.9939970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9940430Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9940879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9941316Z layer_outputs = layer_module( 2025-08-26T20:37:14.9941754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:14.9942295Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:14.9942828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:14.9943315Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:14.9943800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:14.9944296Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:14.9944788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9945275Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9945440Z 2025-08-26T20:37:14.9945565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9945960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9946328Z return mod(**inputs) 2025-08-26T20:37:14.9946754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9947211Z outputs = self.mobilebert( 2025-08-26T20:37:14.9947654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9948095Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9948542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9948997Z layer_outputs = layer_module( 2025-08-26T20:37:14.9949436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:14.9949971Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:14.9950552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:14.9951048Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:14.9951554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:14.9952019Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:14.9952173Z 2025-08-26T20:37:14.9952286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9952691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9953071Z return mod(**inputs) 2025-08-26T20:37:14.9953501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9953958Z outputs = self.mobilebert( 2025-08-26T20:37:14.9954425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9954885Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9955364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9955823Z layer_outputs = layer_module( 2025-08-26T20:37:14.9956272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9956747Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9957220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:14.9957672Z self_outputs = self.self( 2025-08-26T20:37:14.9958123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:14.9958582Z self.value(value_tensor) 2025-08-26T20:37:14.9958710Z 2025-08-26T20:37:14.9958827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9959237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9959682Z return mod(**inputs) 2025-08-26T20:37:14.9960136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9960601Z outputs = self.mobilebert( 2025-08-26T20:37:14.9961067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9961535Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9961998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9962465Z layer_outputs = layer_module( 2025-08-26T20:37:14.9962917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:14.9963477Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:14.9964047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:14.9964567Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:14.9965083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:14.9965552Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:14.9965717Z 2025-08-26T20:37:14.9965834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9966267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9966630Z return mod(**inputs) 2025-08-26T20:37:14.9967067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9967519Z outputs = self.mobilebert( 2025-08-26T20:37:14.9967966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9968434Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9968900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9969376Z layer_outputs = layer_module( 2025-08-26T20:37:14.9969845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:14.9970405Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:14.9970991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:14.9971521Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:14.9972010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:14.9972483Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:14.9972953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:14.9973444Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:14.9973611Z 2025-08-26T20:37:14.9973732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9974123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9974490Z return mod(**inputs) 2025-08-26T20:37:14.9974926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9975390Z outputs = self.mobilebert( 2025-08-26T20:37:14.9975835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9976281Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9976733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9977190Z layer_outputs = layer_module( 2025-08-26T20:37:14.9977642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9978114Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9978586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:14.9979050Z self_outputs = self.self( 2025-08-26T20:37:14.9979502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:14.9979956Z self.query(query_tensor) 2025-08-26T20:37:14.9980084Z 2025-08-26T20:37:14.9980200Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9980603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9980963Z return mod(**inputs) 2025-08-26T20:37:14.9981400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9981855Z outputs = self.mobilebert( 2025-08-26T20:37:14.9982330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9982789Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9983235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9983688Z layer_outputs = layer_module( 2025-08-26T20:37:14.9984133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9984603Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9985085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:14.9985539Z self_outputs = self.self( 2025-08-26T20:37:14.9985977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:14.9986415Z self.key(key_tensor) 2025-08-26T20:37:14.9986615Z 2025-08-26T20:37:14.9986706Z cudagraph partition due to non gpu ops 2025-08-26T20:37:14.9986946Z cudagraph partition due to non gpu ops 2025-08-26T20:37:14.9987208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9987631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9987984Z return mod(**inputs) 2025-08-26T20:37:14.9988414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9988873Z outputs = self.mobilebert( 2025-08-26T20:37:14.9989323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9989781Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9990251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9990697Z layer_outputs = layer_module( 2025-08-26T20:37:14.9991147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9991625Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9992085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:14.9992602Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:14.9993124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:14.9993611Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:14.9993766Z 2025-08-26T20:37:14.9993887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:14.9994279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:14.9994639Z return mod(**inputs) 2025-08-26T20:37:14.9995070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:14.9995533Z outputs = self.mobilebert( 2025-08-26T20:37:14.9995969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:14.9996666Z encoder_outputs = self.encoder( 2025-08-26T20:37:14.9997131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:14.9997597Z layer_outputs = layer_module( 2025-08-26T20:37:14.9998046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:14.9998503Z self_attention_outputs = self.attention( 2025-08-26T20:37:14.9999035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:14.9999606Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0000126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0000655Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0001160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0001681Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0001850Z 2025-08-26T20:37:15.0001964Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0002353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0002739Z return mod(**inputs) 2025-08-26T20:37:15.0003149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0003618Z outputs = self.mobilebert( 2025-08-26T20:37:15.0004101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0004511Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0004914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0005330Z layer_outputs = layer_module( 2025-08-26T20:37:15.0005741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0006182Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0006621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0007098Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0007576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0008005Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0008147Z 2025-08-26T20:37:15.0008260Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0008623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0008947Z return mod(**inputs) 2025-08-26T20:37:15.0009369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0009831Z outputs = self.mobilebert( 2025-08-26T20:37:15.0010262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0010706Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0011142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0011565Z layer_outputs = layer_module( 2025-08-26T20:37:15.0011978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0012422Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0012859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0013318Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0013793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0014254Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0014422Z 2025-08-26T20:37:15.0014538Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0014894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0015224Z return mod(**inputs) 2025-08-26T20:37:15.0015617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0016032Z outputs = self.mobilebert( 2025-08-26T20:37:15.0016451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0016867Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0017273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0017712Z layer_outputs = layer_module( 2025-08-26T20:37:15.0018126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0018573Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0018997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0019469Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0019934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0020363Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0020505Z 2025-08-26T20:37:15.0020620Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0020992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0021347Z return mod(**inputs) 2025-08-26T20:37:15.0021760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0022199Z outputs = self.mobilebert( 2025-08-26T20:37:15.0022617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0023052Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0023460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0023878Z layer_outputs = layer_module( 2025-08-26T20:37:15.0024285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0024713Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0025157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0025652Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0026133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0026597Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0027051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0027482Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0027639Z 2025-08-26T20:37:15.0027745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0028106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0028461Z return mod(**inputs) 2025-08-26T20:37:15.0028852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0029273Z outputs = self.mobilebert( 2025-08-26T20:37:15.0029678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0030094Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0030496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0030917Z layer_outputs = layer_module( 2025-08-26T20:37:15.0031341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0031786Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0032226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0032693Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0033155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0033606Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0033746Z 2025-08-26T20:37:15.0033862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0034231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0034552Z return mod(**inputs) 2025-08-26T20:37:15.0034957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0035394Z outputs = self.mobilebert( 2025-08-26T20:37:15.0035825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0036265Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0036692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0037129Z layer_outputs = layer_module( 2025-08-26T20:37:15.0037560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0038027Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0038485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0038969Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0039542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0040062Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0040248Z 2025-08-26T20:37:15.0040373Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0040774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0041127Z return mod(**inputs) 2025-08-26T20:37:15.0041548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0041991Z outputs = self.mobilebert( 2025-08-26T20:37:15.0042424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0042857Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0043292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0043760Z layer_outputs = layer_module( 2025-08-26T20:37:15.0044194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0044662Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0045120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0045620Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0046122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0046595Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0046750Z 2025-08-26T20:37:15.0046862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0047262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0047631Z return mod(**inputs) 2025-08-26T20:37:15.0048051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0048522Z outputs = self.mobilebert( 2025-08-26T20:37:15.0048943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0049385Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0049825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0050270Z layer_outputs = layer_module( 2025-08-26T20:37:15.0050684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0051114Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0051551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0052024Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0052494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0052960Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0053427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0053872Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0054027Z 2025-08-26T20:37:15.0054134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0054498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0054823Z return mod(**inputs) 2025-08-26T20:37:15.0055242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0055684Z outputs = self.mobilebert( 2025-08-26T20:37:15.0056119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0056555Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0056985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0057415Z layer_outputs = layer_module( 2025-08-26T20:37:15.0057826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0058262Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0058723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0059193Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0059681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0060115Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0060259Z 2025-08-26T20:37:15.0060381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0060768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0061108Z return mod(**inputs) 2025-08-26T20:37:15.0061520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0061938Z outputs = self.mobilebert( 2025-08-26T20:37:15.0062359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0062814Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0063250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0063713Z layer_outputs = layer_module( 2025-08-26T20:37:15.0064146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0064587Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0065018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0065473Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0065929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0066389Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0066554Z 2025-08-26T20:37:15.0066667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0067029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0067361Z return mod(**inputs) 2025-08-26T20:37:15.0067751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0068167Z outputs = self.mobilebert( 2025-08-26T20:37:15.0068591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0069034Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0069472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0069916Z layer_outputs = layer_module( 2025-08-26T20:37:15.0070347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0070817Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0071283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0071780Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0072280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0072737Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0072889Z 2025-08-26T20:37:15.0073001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0073391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0073762Z return mod(**inputs) 2025-08-26T20:37:15.0074185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0074629Z outputs = self.mobilebert( 2025-08-26T20:37:15.0075097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0075565Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0076024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0076504Z layer_outputs = layer_module( 2025-08-26T20:37:15.0076956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0077440Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0077943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0078460Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0078997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0079613Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0080149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0080634Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0080805Z 2025-08-26T20:37:15.0080922Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0081289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0081619Z return mod(**inputs) 2025-08-26T20:37:15.0082021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0082443Z outputs = self.mobilebert( 2025-08-26T20:37:15.0082849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0083268Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0083672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0084114Z layer_outputs = layer_module( 2025-08-26T20:37:15.0084544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0085036Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0085526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0085978Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0086134Z 2025-08-26T20:37:15.0086248Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0086633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0086985Z return mod(**inputs) 2025-08-26T20:37:15.0087395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0087840Z outputs = self.mobilebert( 2025-08-26T20:37:15.0088275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0088716Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0089182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0089625Z layer_outputs = layer_module( 2025-08-26T20:37:15.0090068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0090562Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0091052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0091535Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0091716Z 2025-08-26T20:37:15.0091831Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0092240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0092596Z return mod(**inputs) 2025-08-26T20:37:15.0093013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0093476Z outputs = self.mobilebert( 2025-08-26T20:37:15.0093899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0094362Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0094802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0095241Z layer_outputs = layer_module( 2025-08-26T20:37:15.0095663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0096381Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0096934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0097408Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0097572Z 2025-08-26T20:37:15.0097693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0098076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0098435Z return mod(**inputs) 2025-08-26T20:37:15.0098854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0099302Z outputs = self.mobilebert( 2025-08-26T20:37:15.0099737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0100175Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0100614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0101060Z layer_outputs = layer_module( 2025-08-26T20:37:15.0101492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0102027Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0102559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0103055Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0103548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0104021Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0104172Z 2025-08-26T20:37:15.0104285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0104703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0105042Z return mod(**inputs) 2025-08-26T20:37:15.0105439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0105862Z outputs = self.mobilebert( 2025-08-26T20:37:15.0106271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0106700Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0107123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0107585Z layer_outputs = layer_module( 2025-08-26T20:37:15.0108001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0108504Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0109053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0109559Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0110080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0110539Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0110690Z 2025-08-26T20:37:15.0110803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0111192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0111539Z return mod(**inputs) 2025-08-26T20:37:15.0111934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0112347Z outputs = self.mobilebert( 2025-08-26T20:37:15.0112755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0113172Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0113581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0113998Z layer_outputs = layer_module( 2025-08-26T20:37:15.0114399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0114897Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0115401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0115885Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0116374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0116852Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0117339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0117800Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0117957Z 2025-08-26T20:37:15.0118080Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0118466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0118811Z return mod(**inputs) 2025-08-26T20:37:15.0119232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0119736Z outputs = self.mobilebert( 2025-08-26T20:37:15.0120199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0120641Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0121069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0121509Z layer_outputs = layer_module( 2025-08-26T20:37:15.0121948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0122493Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0123051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0123539Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0124028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0124502Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0124668Z 2025-08-26T20:37:15.0124788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0125165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0125513Z return mod(**inputs) 2025-08-26T20:37:15.0125929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0126370Z outputs = self.mobilebert( 2025-08-26T20:37:15.0126798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0127235Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0127678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0128129Z layer_outputs = layer_module( 2025-08-26T20:37:15.0128544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0128981Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0129407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0129826Z self_outputs = self.self( 2025-08-26T20:37:15.0130236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0130660Z self.value(value_tensor) 2025-08-26T20:37:15.0130785Z 2025-08-26T20:37:15.0130897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0131299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0131652Z return mod(**inputs) 2025-08-26T20:37:15.0132048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0132497Z outputs = self.mobilebert( 2025-08-26T20:37:15.0132914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0133336Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0133751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0134167Z layer_outputs = layer_module( 2025-08-26T20:37:15.0134581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0135113Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0135632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0136100Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0136570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0137039Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0137194Z 2025-08-26T20:37:15.0137297Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0137682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0138022Z return mod(**inputs) 2025-08-26T20:37:15.0138416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0138854Z outputs = self.mobilebert( 2025-08-26T20:37:15.0139253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0139690Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0140104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0140541Z layer_outputs = layer_module( 2025-08-26T20:37:15.0140963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0141467Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0141975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0142428Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0142883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0143308Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0143740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0144178Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0144326Z 2025-08-26T20:37:15.0144438Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0144801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0145122Z return mod(**inputs) 2025-08-26T20:37:15.0145516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0145931Z outputs = self.mobilebert( 2025-08-26T20:37:15.0146337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0146753Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0147159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0147581Z layer_outputs = layer_module( 2025-08-26T20:37:15.0147989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0148424Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0148873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0149309Z self_outputs = self.self( 2025-08-26T20:37:15.0149736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0150182Z self.query(query_tensor) 2025-08-26T20:37:15.0150306Z 2025-08-26T20:37:15.0150437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0150803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0151144Z return mod(**inputs) 2025-08-26T20:37:15.0151547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0151973Z outputs = self.mobilebert( 2025-08-26T20:37:15.0152401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0152810Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0153220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0153699Z layer_outputs = layer_module( 2025-08-26T20:37:15.0154112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0154567Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0154996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0155413Z self_outputs = self.self( 2025-08-26T20:37:15.0155824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0156239Z self.key(key_tensor) 2025-08-26T20:37:15.0156347Z 2025-08-26T20:37:15.0156432Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0156658Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0156899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0157267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0157594Z return mod(**inputs) 2025-08-26T20:37:15.0158002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0158445Z outputs = self.mobilebert( 2025-08-26T20:37:15.0158875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0159315Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0159827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0160281Z layer_outputs = layer_module( 2025-08-26T20:37:15.0160727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0161193Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0161656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0162156Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0162658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0163121Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0163274Z 2025-08-26T20:37:15.0163399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0163801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0164152Z return mod(**inputs) 2025-08-26T20:37:15.0164575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0165047Z outputs = self.mobilebert( 2025-08-26T20:37:15.0165476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0165912Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0166353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0166792Z layer_outputs = layer_module( 2025-08-26T20:37:15.0167226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0167677Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0168153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0168651Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0169147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0169643Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0170138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0170571Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0170726Z 2025-08-26T20:37:15.0170833Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0171200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0171535Z return mod(**inputs) 2025-08-26T20:37:15.0171927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0172340Z outputs = self.mobilebert( 2025-08-26T20:37:15.0172747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0173170Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0173589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0174002Z layer_outputs = layer_module( 2025-08-26T20:37:15.0174418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0174865Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0175305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0175762Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0176215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0176655Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0176805Z 2025-08-26T20:37:15.0176910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0177279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0177611Z return mod(**inputs) 2025-08-26T20:37:15.0177997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0178417Z outputs = self.mobilebert( 2025-08-26T20:37:15.0178826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0179247Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0179676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0180097Z layer_outputs = layer_module( 2025-08-26T20:37:15.0180531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0180997Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0181468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0181916Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0182386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0182851Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0183026Z 2025-08-26T20:37:15.0183136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0183503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0183845Z return mod(**inputs) 2025-08-26T20:37:15.0184243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0184681Z outputs = self.mobilebert( 2025-08-26T20:37:15.0185090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0185512Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0185923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0186343Z layer_outputs = layer_module( 2025-08-26T20:37:15.0186760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0187190Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0187630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0188099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0188572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0189005Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0189146Z 2025-08-26T20:37:15.0189261Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0189628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0189963Z return mod(**inputs) 2025-08-26T20:37:15.0190351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0190764Z outputs = self.mobilebert( 2025-08-26T20:37:15.0191170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0191581Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0191998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0192425Z layer_outputs = layer_module( 2025-08-26T20:37:15.0192861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0193324Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0193784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0194281Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0194805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0195298Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0195789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0196436Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0196608Z 2025-08-26T20:37:15.0196725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0197113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0197524Z return mod(**inputs) 2025-08-26T20:37:15.0197949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0198386Z outputs = self.mobilebert( 2025-08-26T20:37:15.0198819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0199291Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0199803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0200310Z layer_outputs = layer_module( 2025-08-26T20:37:15.0200771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0201273Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0201759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0202231Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0202667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0203098Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0203256Z 2025-08-26T20:37:15.0203370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0203758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0204107Z return mod(**inputs) 2025-08-26T20:37:15.0204513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0204958Z outputs = self.mobilebert( 2025-08-26T20:37:15.0205359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0205779Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0206192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0206598Z layer_outputs = layer_module( 2025-08-26T20:37:15.0206995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0207428Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0207864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0208311Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0208780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0209264Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0209439Z 2025-08-26T20:37:15.0209557Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0209984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0210318Z return mod(**inputs) 2025-08-26T20:37:15.0210713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0211128Z outputs = self.mobilebert( 2025-08-26T20:37:15.0211534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0211953Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0212359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0212792Z layer_outputs = layer_module( 2025-08-26T20:37:15.0213206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0213645Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0214099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0214571Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0215068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0215511Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0215653Z 2025-08-26T20:37:15.0215769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0216130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0216462Z return mod(**inputs) 2025-08-26T20:37:15.0216861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0217284Z outputs = self.mobilebert( 2025-08-26T20:37:15.0217699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0218116Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0218534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0218954Z layer_outputs = layer_module( 2025-08-26T20:37:15.0219370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0219815Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0220278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0220778Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0221284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0221788Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0222293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0222751Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0222917Z 2025-08-26T20:37:15.0223027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0223417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0223774Z return mod(**inputs) 2025-08-26T20:37:15.0224194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0224638Z outputs = self.mobilebert( 2025-08-26T20:37:15.0225105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0225553Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0225988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0226431Z layer_outputs = layer_module( 2025-08-26T20:37:15.0226872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0227336Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0227889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0228382Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0228874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0229373Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0229532Z 2025-08-26T20:37:15.0229645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0230058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0230410Z return mod(**inputs) 2025-08-26T20:37:15.0230820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0231266Z outputs = self.mobilebert( 2025-08-26T20:37:15.0231701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0232141Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0232581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0233034Z layer_outputs = layer_module( 2025-08-26T20:37:15.0233482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0233968Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0234446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0234952Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0235457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0235976Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0236160Z 2025-08-26T20:37:15.0236286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0236692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0237049Z return mod(**inputs) 2025-08-26T20:37:15.0237481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0237943Z outputs = self.mobilebert( 2025-08-26T20:37:15.0238390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0238844Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0239298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0239847Z layer_outputs = layer_module( 2025-08-26T20:37:15.0240322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0240819Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0241329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0241831Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0242325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0242784Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0242935Z 2025-08-26T20:37:15.0243055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0243438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0243809Z return mod(**inputs) 2025-08-26T20:37:15.0244226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0244644Z outputs = self.mobilebert( 2025-08-26T20:37:15.0245075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0245479Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0245905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0246313Z layer_outputs = layer_module( 2025-08-26T20:37:15.0246715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0247137Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0247565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0248022Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0248495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0248964Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0249422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0249864Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0250021Z 2025-08-26T20:37:15.0250126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0250503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0250849Z return mod(**inputs) 2025-08-26T20:37:15.0251249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0251668Z outputs = self.mobilebert( 2025-08-26T20:37:15.0252079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0252524Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0252957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0253387Z layer_outputs = layer_module( 2025-08-26T20:37:15.0253815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0254282Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0254752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0255180Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0255323Z 2025-08-26T20:37:15.0255431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0255835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0256170Z return mod(**inputs) 2025-08-26T20:37:15.0256565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0256979Z outputs = self.mobilebert( 2025-08-26T20:37:15.0257388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0257808Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0258242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0258664Z layer_outputs = layer_module( 2025-08-26T20:37:15.0259075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0259592Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0260085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0260590Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0260771Z 2025-08-26T20:37:15.0260892Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0261273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0261625Z return mod(**inputs) 2025-08-26T20:37:15.0262046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0262490Z outputs = self.mobilebert( 2025-08-26T20:37:15.0262914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0263357Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0263796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0264239Z layer_outputs = layer_module( 2025-08-26T20:37:15.0264668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0265186Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0265724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0266195Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0266362Z 2025-08-26T20:37:15.0266481Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0266865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0267220Z return mod(**inputs) 2025-08-26T20:37:15.0267639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0268083Z outputs = self.mobilebert( 2025-08-26T20:37:15.0268511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0268948Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0269375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0270671Z layer_outputs = layer_module( 2025-08-26T20:37:15.0271113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0271647Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0272224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0272724Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0273222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0273687Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0273848Z 2025-08-26T20:37:15.0273973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0274385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0274749Z return mod(**inputs) 2025-08-26T20:37:15.0275187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0275641Z outputs = self.mobilebert( 2025-08-26T20:37:15.0276109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0276555Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0277047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0277508Z layer_outputs = layer_module( 2025-08-26T20:37:15.0277958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0278513Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0279071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0279656Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0280193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0280676Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0280836Z 2025-08-26T20:37:15.0280959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0281357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0281726Z return mod(**inputs) 2025-08-26T20:37:15.0282181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0282646Z outputs = self.mobilebert( 2025-08-26T20:37:15.0283095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0283561Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0284025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0284494Z layer_outputs = layer_module( 2025-08-26T20:37:15.0284955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0285510Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0286089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0286607Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0287130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0287643Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0288200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0288735Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0288904Z 2025-08-26T20:37:15.0289017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0289401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0289751Z return mod(**inputs) 2025-08-26T20:37:15.0290163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0290610Z outputs = self.mobilebert( 2025-08-26T20:37:15.0291074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0291521Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0291960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0292413Z layer_outputs = layer_module( 2025-08-26T20:37:15.0292859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0293442Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0293977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0294449Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0294930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0295382Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0295531Z 2025-08-26T20:37:15.0295652Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0296044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0296644Z return mod(**inputs) 2025-08-26T20:37:15.0297074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0297516Z outputs = self.mobilebert( 2025-08-26T20:37:15.0297950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0298393Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0298829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0299281Z layer_outputs = layer_module( 2025-08-26T20:37:15.0299714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0300178Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0300637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0301074Z self_outputs = self.self( 2025-08-26T20:37:15.0301505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0301937Z self.value(value_tensor) 2025-08-26T20:37:15.0302061Z 2025-08-26T20:37:15.0302180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0302559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0302909Z return mod(**inputs) 2025-08-26T20:37:15.0303324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0303841Z outputs = self.mobilebert( 2025-08-26T20:37:15.0304275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0304714Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0305154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0305601Z layer_outputs = layer_module( 2025-08-26T20:37:15.0306037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0306597Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0307139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0307619Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0308096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0308524Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0308699Z 2025-08-26T20:37:15.0308810Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0309156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0309477Z return mod(**inputs) 2025-08-26T20:37:15.0309855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0310259Z outputs = self.mobilebert( 2025-08-26T20:37:15.0310642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0311052Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0311457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0311865Z layer_outputs = layer_module( 2025-08-26T20:37:15.0312271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0312764Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0313273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0313731Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0314203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0314657Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0315103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0315560Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0315724Z 2025-08-26T20:37:15.0315838Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0316223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0316568Z return mod(**inputs) 2025-08-26T20:37:15.0316974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0317415Z outputs = self.mobilebert( 2025-08-26T20:37:15.0317841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0318293Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0318758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0319221Z layer_outputs = layer_module( 2025-08-26T20:37:15.0319930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0320419Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0320884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0321328Z self_outputs = self.self( 2025-08-26T20:37:15.0321747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0322161Z self.query(query_tensor) 2025-08-26T20:37:15.0322275Z 2025-08-26T20:37:15.0322387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0322752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0323094Z return mod(**inputs) 2025-08-26T20:37:15.0323490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0323928Z outputs = self.mobilebert( 2025-08-26T20:37:15.0324333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0324747Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0325161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0325580Z layer_outputs = layer_module( 2025-08-26T20:37:15.0325990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0326427Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0326848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0327265Z self_outputs = self.self( 2025-08-26T20:37:15.0327669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0328084Z self.key(key_tensor) 2025-08-26T20:37:15.0328190Z 2025-08-26T20:37:15.0328280Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0328494Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0328735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0329101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0329429Z return mod(**inputs) 2025-08-26T20:37:15.0329816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0330234Z outputs = self.mobilebert( 2025-08-26T20:37:15.0330635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0331056Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0331489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0331924Z layer_outputs = layer_module( 2025-08-26T20:37:15.0332356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0332824Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0333253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0333715Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0334200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0334632Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0334782Z 2025-08-26T20:37:15.0334887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0335247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0335568Z return mod(**inputs) 2025-08-26T20:37:15.0335964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0336406Z outputs = self.mobilebert( 2025-08-26T20:37:15.0336814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0337238Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0337645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0338095Z layer_outputs = layer_module( 2025-08-26T20:37:15.0338507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0338964Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0339396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0339863Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0340335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0340811Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0341287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0341739Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0341893Z 2025-08-26T20:37:15.0342005Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0342381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0342720Z return mod(**inputs) 2025-08-26T20:37:15.0343122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0343579Z outputs = self.mobilebert( 2025-08-26T20:37:15.0343994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0344417Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0344836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0345272Z layer_outputs = layer_module( 2025-08-26T20:37:15.0345685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0346140Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0346590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0347060Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0347525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0347957Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0348109Z 2025-08-26T20:37:15.0348217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0348611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0348946Z return mod(**inputs) 2025-08-26T20:37:15.0349333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0349755Z outputs = self.mobilebert( 2025-08-26T20:37:15.0350164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0350597Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0351112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0351521Z layer_outputs = layer_module( 2025-08-26T20:37:15.0351935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0352378Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0352836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0353338Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0353815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0354297Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0354485Z 2025-08-26T20:37:15.0354598Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0354984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0355333Z return mod(**inputs) 2025-08-26T20:37:15.0355744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0356189Z outputs = self.mobilebert( 2025-08-26T20:37:15.0356620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0357062Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0357492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0357938Z layer_outputs = layer_module( 2025-08-26T20:37:15.0358370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0358838Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0359306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0359884Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0360407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0360871Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0361024Z 2025-08-26T20:37:15.0361149Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0361541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0361892Z return mod(**inputs) 2025-08-26T20:37:15.0362317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0362767Z outputs = self.mobilebert( 2025-08-26T20:37:15.0363203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0363649Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0364070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0364483Z layer_outputs = layer_module( 2025-08-26T20:37:15.0364883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0365311Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0365731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0366185Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0366657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0367112Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0367566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0368003Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0368162Z 2025-08-26T20:37:15.0368265Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0368648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0368978Z return mod(**inputs) 2025-08-26T20:37:15.0369373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0369780Z outputs = self.mobilebert( 2025-08-26T20:37:15.0370183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0370618Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0371054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0371497Z layer_outputs = layer_module( 2025-08-26T20:37:15.0371919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0372385Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0372847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0373316Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0373762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0374193Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0374350Z 2025-08-26T20:37:15.0374461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0374848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0375200Z return mod(**inputs) 2025-08-26T20:37:15.0375605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0376056Z outputs = self.mobilebert( 2025-08-26T20:37:15.0376458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0376877Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0377289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0377697Z layer_outputs = layer_module( 2025-08-26T20:37:15.0378114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0378568Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0378996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0379438Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0379880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0380339Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0380523Z 2025-08-26T20:37:15.0380630Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0381007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0381325Z return mod(**inputs) 2025-08-26T20:37:15.0381710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0382147Z outputs = self.mobilebert( 2025-08-26T20:37:15.0382553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0382994Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0383401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0383814Z layer_outputs = layer_module( 2025-08-26T20:37:15.0384225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0384661Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0385100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0385560Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0386032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0386460Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0386605Z 2025-08-26T20:37:15.0386722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0387106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0387447Z return mod(**inputs) 2025-08-26T20:37:15.0387839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0388251Z outputs = self.mobilebert( 2025-08-26T20:37:15.0388666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0389112Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0389522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0389949Z layer_outputs = layer_module( 2025-08-26T20:37:15.0390380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0390846Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0391299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0391803Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0392272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0392735Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0393227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0393671Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0393840Z 2025-08-26T20:37:15.0393952Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0394336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0394683Z return mod(**inputs) 2025-08-26T20:37:15.0395103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0395540Z outputs = self.mobilebert( 2025-08-26T20:37:15.0396000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0396661Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0397105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0397596Z layer_outputs = layer_module( 2025-08-26T20:37:15.0398034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0398527Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0398998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0399532Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0400026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0400486Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0400644Z 2025-08-26T20:37:15.0400761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0402587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0402937Z return mod(**inputs) 2025-08-26T20:37:15.0403349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0403798Z outputs = self.mobilebert( 2025-08-26T20:37:15.0404227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0404671Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0405108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0405542Z layer_outputs = layer_module( 2025-08-26T20:37:15.0405979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0406447Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0406975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0407456Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0407927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0408406Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0408589Z 2025-08-26T20:37:15.0408703Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0409091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0409432Z return mod(**inputs) 2025-08-26T20:37:15.0409852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0410295Z outputs = self.mobilebert( 2025-08-26T20:37:15.0410772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0411220Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0411652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0412091Z layer_outputs = layer_module( 2025-08-26T20:37:15.0412504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0412944Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0413400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0413524Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0413802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0413914Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0413938Z 2025-08-26T20:37:15.0414044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0414246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0414312Z return mod(**inputs) 2025-08-26T20:37:15.0414590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0414673Z outputs = self.mobilebert( 2025-08-26T20:37:15.0414957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0415041Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0415327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0415409Z layer_outputs = layer_module( 2025-08-26T20:37:15.0415690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0415789Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0416080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0416213Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0416497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0416616Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0416888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0416989Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0416993Z 2025-08-26T20:37:15.0417095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0417299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0417365Z return mod(**inputs) 2025-08-26T20:37:15.0417643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0417715Z outputs = self.mobilebert( 2025-08-26T20:37:15.0417987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0418067Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0418359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0418441Z layer_outputs = layer_module( 2025-08-26T20:37:15.0418714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0418836Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0419122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0419208Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0419212Z 2025-08-26T20:37:15.0419323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0419536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0419611Z return mod(**inputs) 2025-08-26T20:37:15.0419891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0419985Z outputs = self.mobilebert( 2025-08-26T20:37:15.0420275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0420368Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0420656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0420729Z layer_outputs = layer_module( 2025-08-26T20:37:15.0421012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0421141Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0421437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0421567Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0421572Z 2025-08-26T20:37:15.0421684Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0421898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0421970Z return mod(**inputs) 2025-08-26T20:37:15.0422267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0422352Z outputs = self.mobilebert( 2025-08-26T20:37:15.0422651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0422737Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0423034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0423108Z layer_outputs = layer_module( 2025-08-26T20:37:15.0423418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0423588Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0423895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0423997Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0424001Z 2025-08-26T20:37:15.0424116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0424327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0424398Z return mod(**inputs) 2025-08-26T20:37:15.0424702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0424774Z outputs = self.mobilebert( 2025-08-26T20:37:15.0425079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0425153Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0425437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0425516Z layer_outputs = layer_module( 2025-08-26T20:37:15.0425795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0425961Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0426263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0426399Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0426700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0426792Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0426814Z 2025-08-26T20:37:15.0426926Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0427124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0427196Z return mod(**inputs) 2025-08-26T20:37:15.0427476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0427548Z outputs = self.mobilebert( 2025-08-26T20:37:15.0427833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0427906Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0428205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0428282Z layer_outputs = layer_module( 2025-08-26T20:37:15.0428584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0428757Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0429039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0429171Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0429453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0429547Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0429550Z 2025-08-26T20:37:15.0429656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0429865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0429932Z return mod(**inputs) 2025-08-26T20:37:15.0430217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0430298Z outputs = self.mobilebert( 2025-08-26T20:37:15.0430578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0430659Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0430940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0431012Z layer_outputs = layer_module( 2025-08-26T20:37:15.0431344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0431513Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0431825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0431950Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0432234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0432354Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0432650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0432752Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0432755Z 2025-08-26T20:37:15.0432861Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0433088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0433155Z return mod(**inputs) 2025-08-26T20:37:15.0433454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0433534Z outputs = self.mobilebert( 2025-08-26T20:37:15.0433824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0433909Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0434207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0434289Z layer_outputs = layer_module( 2025-08-26T20:37:15.0434587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0434762Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0435069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0435189Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0435491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0435581Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0435585Z 2025-08-26T20:37:15.0435695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0435916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0435987Z return mod(**inputs) 2025-08-26T20:37:15.0436293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0436371Z outputs = self.mobilebert( 2025-08-26T20:37:15.0436677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0436757Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0437055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0437139Z layer_outputs = layer_module( 2025-08-26T20:37:15.0437439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0437540Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0437842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0437919Z self_outputs = self.self( 2025-08-26T20:37:15.0438281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0438361Z self.value(value_tensor) 2025-08-26T20:37:15.0438366Z 2025-08-26T20:37:15.0438486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0438700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0438779Z return mod(**inputs) 2025-08-26T20:37:15.0439077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0439998Z outputs = self.mobilebert( 2025-08-26T20:37:15.0440347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0440427Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0440744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0440856Z layer_outputs = layer_module( 2025-08-26T20:37:15.0441155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0441326Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0441608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0441729Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0442015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0442107Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0442111Z 2025-08-26T20:37:15.0442217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0442415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0442490Z return mod(**inputs) 2025-08-26T20:37:15.0442767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0442847Z outputs = self.mobilebert( 2025-08-26T20:37:15.0443129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0443202Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0443494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0443566Z layer_outputs = layer_module( 2025-08-26T20:37:15.0443855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0444015Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0444305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0444418Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0444701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0444797Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0445083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0445184Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0445188Z 2025-08-26T20:37:15.0445292Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0445514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0445591Z return mod(**inputs) 2025-08-26T20:37:15.0445870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0445952Z outputs = self.mobilebert( 2025-08-26T20:37:15.0446230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0446310Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0446607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0446682Z layer_outputs = layer_module( 2025-08-26T20:37:15.0446972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0447080Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0447367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0447457Z self_outputs = self.self( 2025-08-26T20:37:15.0447740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0447820Z self.query(query_tensor) 2025-08-26T20:37:15.0447823Z 2025-08-26T20:37:15.0447929Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0448138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0448204Z return mod(**inputs) 2025-08-26T20:37:15.0448489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0448558Z outputs = self.mobilebert( 2025-08-26T20:37:15.0448823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0448900Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0449172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0449250Z layer_outputs = layer_module( 2025-08-26T20:37:15.0449524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0449609Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0449902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0449976Z self_outputs = self.self( 2025-08-26T20:37:15.0450265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0450334Z self.key(key_tensor) 2025-08-26T20:37:15.0450338Z 2025-08-26T20:37:15.0450421Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0450511Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0450615Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0450821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0450886Z return mod(**inputs) 2025-08-26T20:37:15.0451175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0451248Z outputs = self.mobilebert( 2025-08-26T20:37:15.0451528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0451609Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0451903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0451985Z layer_outputs = layer_module( 2025-08-26T20:37:15.0452266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0452351Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0452636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0452760Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0453066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0453155Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0453158Z 2025-08-26T20:37:15.0453288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0453487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0453553Z return mod(**inputs) 2025-08-26T20:37:15.0453870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0453943Z outputs = self.mobilebert( 2025-08-26T20:37:15.0454227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0454301Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0454584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0454665Z layer_outputs = layer_module( 2025-08-26T20:37:15.0454949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0455041Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0455322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0455446Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0455733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0455860Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0456156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0456247Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0456251Z 2025-08-26T20:37:15.0456357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0456555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0456620Z return mod(**inputs) 2025-08-26T20:37:15.0456901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0456971Z outputs = self.mobilebert( 2025-08-26T20:37:15.0457247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0457319Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0457594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0457670Z layer_outputs = layer_module( 2025-08-26T20:37:15.0457948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0458079Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0458348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0458465Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0458729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0458809Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0458813Z 2025-08-26T20:37:15.0458917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0459121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0459193Z return mod(**inputs) 2025-08-26T20:37:15.0459471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0459567Z outputs = self.mobilebert( 2025-08-26T20:37:15.0459838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0459928Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0460210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0460279Z layer_outputs = layer_module( 2025-08-26T20:37:15.0460567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0460663Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0460949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0461076Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0461379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0461506Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0461512Z 2025-08-26T20:37:15.0461621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0461839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0461909Z return mod(**inputs) 2025-08-26T20:37:15.0462213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0462296Z outputs = self.mobilebert( 2025-08-26T20:37:15.0462581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0462661Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0462956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0463026Z layer_outputs = layer_module( 2025-08-26T20:37:15.0463310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0463403Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0463686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0463808Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0464097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0464180Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0464184Z 2025-08-26T20:37:15.0464301Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0464512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0464580Z return mod(**inputs) 2025-08-26T20:37:15.0464873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0464943Z outputs = self.mobilebert( 2025-08-26T20:37:15.0465217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0465299Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0465596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0465678Z layer_outputs = layer_module( 2025-08-26T20:37:15.0465960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0466074Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0466363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0466506Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0466792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0466913Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0467197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0467290Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0467294Z 2025-08-26T20:37:15.0467396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0467607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0467673Z return mod(**inputs) 2025-08-26T20:37:15.0467957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0468031Z outputs = self.mobilebert( 2025-08-26T20:37:15.0468314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0468386Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0468664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0468744Z layer_outputs = layer_module( 2025-08-26T20:37:15.0469027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0469129Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0469415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0469530Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0469831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0469919Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0469923Z 2025-08-26T20:37:15.0470039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0470257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0470332Z return mod(**inputs) 2025-08-26T20:37:15.0470610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0470702Z outputs = self.mobilebert( 2025-08-26T20:37:15.0470990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0471067Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0471354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0471427Z layer_outputs = layer_module( 2025-08-26T20:37:15.0471707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0471824Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0472104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0472223Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0472526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0472647Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0472668Z 2025-08-26T20:37:15.0472771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0472974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0473048Z return mod(**inputs) 2025-08-26T20:37:15.0473325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0473404Z outputs = self.mobilebert( 2025-08-26T20:37:15.0473688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0473761Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0474054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0474126Z layer_outputs = layer_module( 2025-08-26T20:37:15.0474415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0474511Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0474795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0474922Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0475201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0475293Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0475296Z 2025-08-26T20:37:15.0475401Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0475608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0475675Z return mod(**inputs) 2025-08-26T20:37:15.0475956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0476034Z outputs = self.mobilebert( 2025-08-26T20:37:15.0476313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0476393Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0476676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0476747Z layer_outputs = layer_module( 2025-08-26T20:37:15.0477050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0477148Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0477435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0477563Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0477865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0477994Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0478306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0478416Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0478420Z 2025-08-26T20:37:15.0478531Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0478752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0478850Z return mod(**inputs) 2025-08-26T20:37:15.0479161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0479260Z outputs = self.mobilebert( 2025-08-26T20:37:15.0479660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0479758Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0480079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0480167Z layer_outputs = layer_module( 2025-08-26T20:37:15.0480472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0480582Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0480898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0481024Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0481338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0481431Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0481435Z 2025-08-26T20:37:15.0481561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0481760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0481829Z return mod(**inputs) 2025-08-26T20:37:15.0482119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0482196Z outputs = self.mobilebert( 2025-08-26T20:37:15.0482495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0482570Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0482854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0482931Z layer_outputs = layer_module( 2025-08-26T20:37:15.0483207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0483308Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0483593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0483709Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0484024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0484137Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0484142Z 2025-08-26T20:37:15.0484250Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0484443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0484517Z return mod(**inputs) 2025-08-26T20:37:15.0484796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0484870Z outputs = self.mobilebert( 2025-08-26T20:37:15.0485187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0485261Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0485543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0485632Z layer_outputs = layer_module( 2025-08-26T20:37:15.0485914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0486027Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0486300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0486429Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0486703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0486794Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0486797Z 2025-08-26T20:37:15.0486897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0487093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0487165Z return mod(**inputs) 2025-08-26T20:37:15.0487439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0487518Z outputs = self.mobilebert( 2025-08-26T20:37:15.0487787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0487865Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0488197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0488272Z layer_outputs = layer_module( 2025-08-26T20:37:15.0488573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0488674Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0488978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0489104Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0489383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0489515Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0489809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0489913Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0489918Z 2025-08-26T20:37:15.0490027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0490265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0490333Z return mod(**inputs) 2025-08-26T20:37:15.0490604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0490681Z outputs = self.mobilebert( 2025-08-26T20:37:15.0490950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0491028Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0491316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0491389Z layer_outputs = layer_module( 2025-08-26T20:37:15.0491677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0491800Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0492111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0492218Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0492222Z 2025-08-26T20:37:15.0492337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0492547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0492616Z return mod(**inputs) 2025-08-26T20:37:15.0492915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0492987Z outputs = self.mobilebert( 2025-08-26T20:37:15.0493271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0493354Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0493628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0493706Z layer_outputs = layer_module( 2025-08-26T20:37:15.0493981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0494104Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0494378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0494495Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0494499Z 2025-08-26T20:37:15.0494598Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0494791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0494868Z return mod(**inputs) 2025-08-26T20:37:15.0495143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0495221Z outputs = self.mobilebert( 2025-08-26T20:37:15.0495497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0495568Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0495857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0495929Z layer_outputs = layer_module( 2025-08-26T20:37:15.0496376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0496542Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0496875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0496976Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0496982Z 2025-08-26T20:37:15.0497088Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0497295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0497364Z return mod(**inputs) 2025-08-26T20:37:15.0497651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0497724Z outputs = self.mobilebert( 2025-08-26T20:37:15.0498032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0498119Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0498403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0498515Z layer_outputs = layer_module( 2025-08-26T20:37:15.0498796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0498991Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0499275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0499403Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0499693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0499787Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0499791Z 2025-08-26T20:37:15.0499901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0500101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0500167Z return mod(**inputs) 2025-08-26T20:37:15.0500453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0500526Z outputs = self.mobilebert( 2025-08-26T20:37:15.0500820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0500894Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0501181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0501253Z layer_outputs = layer_module( 2025-08-26T20:37:15.0501536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0501701Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0501980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0502112Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0502393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0502479Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0502492Z 2025-08-26T20:37:15.0502594Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0502792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0502866Z return mod(**inputs) 2025-08-26T20:37:15.0503162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0503246Z outputs = self.mobilebert( 2025-08-26T20:37:15.0503527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0503602Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0503891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0503962Z layer_outputs = layer_module( 2025-08-26T20:37:15.0504273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0504431Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0504715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0504866Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0505148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0505298Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0505579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0505677Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0505681Z 2025-08-26T20:37:15.0505784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0505983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0506056Z return mod(**inputs) 2025-08-26T20:37:15.0506339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0506419Z outputs = self.mobilebert( 2025-08-26T20:37:15.0506699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0506774Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0507059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0507130Z layer_outputs = layer_module( 2025-08-26T20:37:15.0507418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0507583Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0507872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0507986Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0508269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0508364Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0508368Z 2025-08-26T20:37:15.0508470Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0508675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0508740Z return mod(**inputs) 2025-08-26T20:37:15.0509027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0509100Z outputs = self.mobilebert( 2025-08-26T20:37:15.0509379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0509484Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0509764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0509845Z layer_outputs = layer_module( 2025-08-26T20:37:15.0510141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0510234Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0510544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0510618Z self_outputs = self.self( 2025-08-26T20:37:15.0510919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0510994Z self.value(value_tensor) 2025-08-26T20:37:15.0510998Z 2025-08-26T20:37:15.0511132Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0511332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0511399Z return mod(**inputs) 2025-08-26T20:37:15.0511706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0511779Z outputs = self.mobilebert( 2025-08-26T20:37:15.0512068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0512141Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0512423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0512505Z layer_outputs = layer_module( 2025-08-26T20:37:15.0512784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0512953Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0513233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0513346Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0513632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0513717Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0513721Z 2025-08-26T20:37:15.0513833Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0514030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0514104Z return mod(**inputs) 2025-08-26T20:37:15.0514383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0514457Z outputs = self.mobilebert( 2025-08-26T20:37:15.0514741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0514816Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0515101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0515171Z layer_outputs = layer_module( 2025-08-26T20:37:15.0515451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0515616Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0515916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0516036Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0516319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0516415Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0516700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0516792Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0516796Z 2025-08-26T20:37:15.0516904Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0517119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0517195Z return mod(**inputs) 2025-08-26T20:37:15.0517473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0517570Z outputs = self.mobilebert( 2025-08-26T20:37:15.0517848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0517941Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0518252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0518332Z layer_outputs = layer_module( 2025-08-26T20:37:15.0518641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0518740Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0519044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0519130Z self_outputs = self.self( 2025-08-26T20:37:15.0519439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0519609Z self.query(query_tensor) 2025-08-26T20:37:15.0519616Z 2025-08-26T20:37:15.0519730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0519949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0520029Z return mod(**inputs) 2025-08-26T20:37:15.0520331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0520420Z outputs = self.mobilebert( 2025-08-26T20:37:15.0520723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0520813Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0521120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0521201Z layer_outputs = layer_module( 2025-08-26T20:37:15.0521515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0521612Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0521926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0522004Z self_outputs = self.self( 2025-08-26T20:37:15.0522310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0522393Z self.key(key_tensor) 2025-08-26T20:37:15.0522396Z 2025-08-26T20:37:15.0522487Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0522582Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0522727Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0522950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0523033Z return mod(**inputs) 2025-08-26T20:37:15.0523342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0523427Z outputs = self.mobilebert( 2025-08-26T20:37:15.0523758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0523842Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0524162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0524243Z layer_outputs = layer_module( 2025-08-26T20:37:15.0524561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0524672Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0524986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0525142Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0525452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0525559Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0525563Z 2025-08-26T20:37:15.0525679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0525904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0525976Z return mod(**inputs) 2025-08-26T20:37:15.0526314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0526395Z outputs = self.mobilebert( 2025-08-26T20:37:15.0526698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0526788Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0527095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0527181Z layer_outputs = layer_module( 2025-08-26T20:37:15.0527489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0527582Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0527898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0528035Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0528347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0528487Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0528801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0528915Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0528918Z 2025-08-26T20:37:15.0529028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0529249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0529319Z return mod(**inputs) 2025-08-26T20:37:15.0529647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0529728Z outputs = self.mobilebert( 2025-08-26T20:37:15.0530025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0530112Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0530405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0530486Z layer_outputs = layer_module( 2025-08-26T20:37:15.0530772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0530892Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0531173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0531290Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0531600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0531686Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0531708Z 2025-08-26T20:37:15.0531821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0532024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0532090Z return mod(**inputs) 2025-08-26T20:37:15.0532382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0532456Z outputs = self.mobilebert( 2025-08-26T20:37:15.0532745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0532817Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0533110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0533181Z layer_outputs = layer_module( 2025-08-26T20:37:15.0533472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0533581Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0533879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0534005Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0534308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0534428Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0534440Z 2025-08-26T20:37:15.0534554Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0534764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0534841Z return mod(**inputs) 2025-08-26T20:37:15.0535143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0535228Z outputs = self.mobilebert( 2025-08-26T20:37:15.0535528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0535601Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0535894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0535966Z layer_outputs = layer_module( 2025-08-26T20:37:15.0536277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0536375Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0536660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0536802Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0537099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0537198Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0537202Z 2025-08-26T20:37:15.0537337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0537548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0537614Z return mod(**inputs) 2025-08-26T20:37:15.0537897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0537995Z outputs = self.mobilebert( 2025-08-26T20:37:15.0538279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0538411Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0538701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0538777Z layer_outputs = layer_module( 2025-08-26T20:37:15.0539087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0539189Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0539498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0539634Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0539939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0540072Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0540373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0540477Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0540481Z 2025-08-26T20:37:15.0540589Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0540810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0540880Z return mod(**inputs) 2025-08-26T20:37:15.0541180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0541266Z outputs = self.mobilebert( 2025-08-26T20:37:15.0541562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0541652Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0541954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0542034Z layer_outputs = layer_module( 2025-08-26T20:37:15.0542314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0542411Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0542703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0542813Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0543121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0543207Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0543213Z 2025-08-26T20:37:15.0543324Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0543525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0543591Z return mod(**inputs) 2025-08-26T20:37:15.0543879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0543968Z outputs = self.mobilebert( 2025-08-26T20:37:15.0544258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0544337Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0544659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0544746Z layer_outputs = layer_module( 2025-08-26T20:37:15.0545062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0545170Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0545468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0545585Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0545896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0546016Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0546020Z 2025-08-26T20:37:15.0546139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0546352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0546430Z return mod(**inputs) 2025-08-26T20:37:15.0546725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0546800Z outputs = self.mobilebert( 2025-08-26T20:37:15.0547109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0547183Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0547479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0547552Z layer_outputs = layer_module( 2025-08-26T20:37:15.0547831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0547933Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0548211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0548354Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0548651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0548756Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0548759Z 2025-08-26T20:37:15.0548869Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0549078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0549155Z return mod(**inputs) 2025-08-26T20:37:15.0549472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0549562Z outputs = self.mobilebert( 2025-08-26T20:37:15.0549853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0549928Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0550215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0550287Z layer_outputs = layer_module( 2025-08-26T20:37:15.0550592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0550689Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0550977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0551119Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0551397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0551559Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0551844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0551945Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0551949Z 2025-08-26T20:37:15.0552054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0552265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0552333Z return mod(**inputs) 2025-08-26T20:37:15.0552613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0552698Z outputs = self.mobilebert( 2025-08-26T20:37:15.0552984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0553070Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0553354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0553429Z layer_outputs = layer_module( 2025-08-26T20:37:15.0553716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0553813Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0554117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0554240Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0554542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0554642Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0554647Z 2025-08-26T20:37:15.0554757Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0554974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0555044Z return mod(**inputs) 2025-08-26T20:37:15.0555348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0555427Z outputs = self.mobilebert( 2025-08-26T20:37:15.0555721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0555808Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0556127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0556213Z layer_outputs = layer_module( 2025-08-26T20:37:15.0556514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0556613Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0556924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0557045Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0557376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0557499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0557503Z 2025-08-26T20:37:15.0557622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0557859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0557932Z return mod(**inputs) 2025-08-26T20:37:15.0558261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0558339Z outputs = self.mobilebert( 2025-08-26T20:37:15.0558652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0558731Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0559041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0559127Z layer_outputs = layer_module( 2025-08-26T20:37:15.0559435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0559624Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0559937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0560082Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0560393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0560485Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0560489Z 2025-08-26T20:37:15.0560609Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0560829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0560910Z return mod(**inputs) 2025-08-26T20:37:15.0561220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0561302Z outputs = self.mobilebert( 2025-08-26T20:37:15.0561619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0561700Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0562002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0562075Z layer_outputs = layer_module( 2025-08-26T20:37:15.0562378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0562482Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0562790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0562958Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0563274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0563417Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0563722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0563830Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0563834Z 2025-08-26T20:37:15.0563948Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0564181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0564264Z return mod(**inputs) 2025-08-26T20:37:15.0564579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0564686Z outputs = self.mobilebert( 2025-08-26T20:37:15.0564995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0565099Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0565415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0565492Z layer_outputs = layer_module( 2025-08-26T20:37:15.0565809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0565944Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0566250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0566349Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0566357Z 2025-08-26T20:37:15.0566469Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0566697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0566771Z return mod(**inputs) 2025-08-26T20:37:15.0567086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0567165Z outputs = self.mobilebert( 2025-08-26T20:37:15.0567470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0567558Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0567866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0567952Z layer_outputs = layer_module( 2025-08-26T20:37:15.0568262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0568396Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0568713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0568833Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0568838Z 2025-08-26T20:37:15.0568958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0569181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0569258Z return mod(**inputs) 2025-08-26T20:37:15.0569558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0569646Z outputs = self.mobilebert( 2025-08-26T20:37:15.0569949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0570026Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0570313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0570388Z layer_outputs = layer_module( 2025-08-26T20:37:15.0570668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0570835Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0571130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0571237Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0571241Z 2025-08-26T20:37:15.0571346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0571573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0571640Z return mod(**inputs) 2025-08-26T20:37:15.0571936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0572015Z outputs = self.mobilebert( 2025-08-26T20:37:15.0572302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0572380Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0572653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0572723Z layer_outputs = layer_module( 2025-08-26T20:37:15.0573004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0573159Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0573438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0573561Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0573835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0573926Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0573929Z 2025-08-26T20:37:15.0574033Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0574229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0574294Z return mod(**inputs) 2025-08-26T20:37:15.0574572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0574646Z outputs = self.mobilebert( 2025-08-26T20:37:15.0574930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0575005Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0575282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0575362Z layer_outputs = layer_module( 2025-08-26T20:37:15.0575644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0575809Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0576086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0576233Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0576527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0576612Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0576615Z 2025-08-26T20:37:15.0576725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0576918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0576992Z return mod(**inputs) 2025-08-26T20:37:15.0577289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0577364Z outputs = self.mobilebert( 2025-08-26T20:37:15.0577668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0577769Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0578047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0578135Z layer_outputs = layer_module( 2025-08-26T20:37:15.0578410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0578570Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0578855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0578993Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0579289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0579438Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0579720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0579815Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0579819Z 2025-08-26T20:37:15.0579932Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0580132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0580206Z return mod(**inputs) 2025-08-26T20:37:15.0580486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0580559Z outputs = self.mobilebert( 2025-08-26T20:37:15.0580858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0580939Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0581251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0581325Z layer_outputs = layer_module( 2025-08-26T20:37:15.0581612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0581774Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0582054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0582177Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0582454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0582573Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0582578Z 2025-08-26T20:37:15.0582679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0582881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0582946Z return mod(**inputs) 2025-08-26T20:37:15.0583216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0583295Z outputs = self.mobilebert( 2025-08-26T20:37:15.0583573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0583669Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0583948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0584019Z layer_outputs = layer_module( 2025-08-26T20:37:15.0584327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0584416Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0584722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0584796Z self_outputs = self.self( 2025-08-26T20:37:15.0585076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0585158Z self.value(value_tensor) 2025-08-26T20:37:15.0585162Z 2025-08-26T20:37:15.0585268Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0585475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0585543Z return mod(**inputs) 2025-08-26T20:37:15.0585833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0585908Z outputs = self.mobilebert( 2025-08-26T20:37:15.0586191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0586278Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0586559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0586640Z layer_outputs = layer_module( 2025-08-26T20:37:15.0586926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0587088Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0587381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0587495Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0587780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0587866Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0587870Z 2025-08-26T20:37:15.0587981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0588179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0588247Z return mod(**inputs) 2025-08-26T20:37:15.0588535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0588608Z outputs = self.mobilebert( 2025-08-26T20:37:15.0588918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0588996Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0589289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0589374Z layer_outputs = layer_module( 2025-08-26T20:37:15.0589687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0589855Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0590163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0590280Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0590578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0590694Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0590998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0591121Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0591125Z 2025-08-26T20:37:15.0591233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0591432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0591498Z return mod(**inputs) 2025-08-26T20:37:15.0591785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0591857Z outputs = self.mobilebert( 2025-08-26T20:37:15.0592144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0592220Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0592504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0592577Z layer_outputs = layer_module( 2025-08-26T20:37:15.0592854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0592951Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0593231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0593311Z self_outputs = self.self( 2025-08-26T20:37:15.0593594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0593666Z self.query(query_tensor) 2025-08-26T20:37:15.0593676Z 2025-08-26T20:37:15.0593781Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0593981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0594055Z return mod(**inputs) 2025-08-26T20:37:15.0594343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0594422Z outputs = self.mobilebert( 2025-08-26T20:37:15.0594720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0594798Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0595105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0595182Z layer_outputs = layer_module( 2025-08-26T20:37:15.0595507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0595604Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0595903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0595989Z self_outputs = self.self( 2025-08-26T20:37:15.0596513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0596602Z self.key(key_tensor) 2025-08-26T20:37:15.0596606Z 2025-08-26T20:37:15.0596698Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0596787Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0596960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0597180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0597261Z return mod(**inputs) 2025-08-26T20:37:15.0597605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0597692Z outputs = self.mobilebert( 2025-08-26T20:37:15.0598039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0598120Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0598433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0598515Z layer_outputs = layer_module( 2025-08-26T20:37:15.0598839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0598932Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0599248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0599397Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0599771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0599883Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0599887Z 2025-08-26T20:37:15.0599999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0600237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0600309Z return mod(**inputs) 2025-08-26T20:37:15.0600617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0600703Z outputs = self.mobilebert( 2025-08-26T20:37:15.0601020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0601110Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0601428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0601510Z layer_outputs = layer_module( 2025-08-26T20:37:15.0601831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0601917Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0602257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0602395Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0602721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0602895Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0603217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0603329Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0603333Z 2025-08-26T20:37:15.0603446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0603670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0603743Z return mod(**inputs) 2025-08-26T20:37:15.0604084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0604172Z outputs = self.mobilebert( 2025-08-26T20:37:15.0604479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0604569Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0604895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0604980Z layer_outputs = layer_module( 2025-08-26T20:37:15.0605309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0605416Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0605731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0605856Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0606170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0606261Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0606265Z 2025-08-26T20:37:15.0606381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0606606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0606680Z return mod(**inputs) 2025-08-26T20:37:15.0606995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0607072Z outputs = self.mobilebert( 2025-08-26T20:37:15.0607391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0607470Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0607779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0607865Z layer_outputs = layer_module( 2025-08-26T20:37:15.0608172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0608283Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0608590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0608712Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0609019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0609133Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0609137Z 2025-08-26T20:37:15.0609247Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0609446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0609518Z return mod(**inputs) 2025-08-26T20:37:15.0609816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0609893Z outputs = self.mobilebert( 2025-08-26T20:37:15.0610185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0610261Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0610540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0610611Z layer_outputs = layer_module( 2025-08-26T20:37:15.0610906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0611008Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0611282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0611436Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0611713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0611823Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0611826Z 2025-08-26T20:37:15.0611927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0612122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0612195Z return mod(**inputs) 2025-08-26T20:37:15.0612469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0612547Z outputs = self.mobilebert( 2025-08-26T20:37:15.0612819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0612894Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0613178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0613254Z layer_outputs = layer_module( 2025-08-26T20:37:15.0613559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0613660Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0613966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0614101Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0614399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0614539Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0614837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0614943Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0614947Z 2025-08-26T20:37:15.0615057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0615270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0615346Z return mod(**inputs) 2025-08-26T20:37:15.0615628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0615708Z outputs = self.mobilebert( 2025-08-26T20:37:15.0615992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0616078Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0616375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0616449Z layer_outputs = layer_module( 2025-08-26T20:37:15.0616734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0616825Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0617128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0617247Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0617564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0617664Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0617667Z 2025-08-26T20:37:15.0617802Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0618022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0618112Z return mod(**inputs) 2025-08-26T20:37:15.0618424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0618501Z outputs = self.mobilebert( 2025-08-26T20:37:15.0618801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0618889Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0619192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0619273Z layer_outputs = layer_module( 2025-08-26T20:37:15.0619564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0619659Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0619952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0620067Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0620369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0620490Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0620494Z 2025-08-26T20:37:15.0620613Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0620827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0620899Z return mod(**inputs) 2025-08-26T20:37:15.0621213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0621290Z outputs = self.mobilebert( 2025-08-26T20:37:15.0621603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0621682Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0621989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0622071Z layer_outputs = layer_module( 2025-08-26T20:37:15.0622378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0622488Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0622792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0622958Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0623254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0623344Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0623348Z 2025-08-26T20:37:15.0623467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0623675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0623753Z return mod(**inputs) 2025-08-26T20:37:15.0624074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0624148Z outputs = self.mobilebert( 2025-08-26T20:37:15.0624434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0624526Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0624811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0624901Z layer_outputs = layer_module( 2025-08-26T20:37:15.0625188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0625282Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0625565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0625696Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0625980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0626118Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0626418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0626517Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0626528Z 2025-08-26T20:37:15.0626637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0626846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0626925Z return mod(**inputs) 2025-08-26T20:37:15.0627225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0627310Z outputs = self.mobilebert( 2025-08-26T20:37:15.0627607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0627685Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0627996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0628067Z layer_outputs = layer_module( 2025-08-26T20:37:15.0628356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0628450Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0628732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0628850Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0629135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0629226Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0629229Z 2025-08-26T20:37:15.0629364Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0629573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0629642Z return mod(**inputs) 2025-08-26T20:37:15.0629926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0630008Z outputs = self.mobilebert( 2025-08-26T20:37:15.0630294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0630379Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0630682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0630756Z layer_outputs = layer_module( 2025-08-26T20:37:15.0631058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0631179Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0631483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0631627Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0631915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0632027Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0632030Z 2025-08-26T20:37:15.0632134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0632341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0632409Z return mod(**inputs) 2025-08-26T20:37:15.0632700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0632775Z outputs = self.mobilebert( 2025-08-26T20:37:15.0633065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0633153Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0633447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0633531Z layer_outputs = layer_module( 2025-08-26T20:37:15.0633829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0633932Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0634225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0634362Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0634663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0634756Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0634759Z 2025-08-26T20:37:15.0634874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0635082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0635153Z return mod(**inputs) 2025-08-26T20:37:15.0635458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0635534Z outputs = self.mobilebert( 2025-08-26T20:37:15.0635837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0635937Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0636243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0636323Z layer_outputs = layer_module( 2025-08-26T20:37:15.0636617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0636723Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0637018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0637176Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0637483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0637617Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0637951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0638051Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0638075Z 2025-08-26T20:37:15.0638240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0638467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0638548Z return mod(**inputs) 2025-08-26T20:37:15.0638854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0638939Z outputs = self.mobilebert( 2025-08-26T20:37:15.0639251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0639330Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0639729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0639815Z layer_outputs = layer_module( 2025-08-26T20:37:15.0640125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0640268Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0640574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0640676Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0640680Z 2025-08-26T20:37:15.0640795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0641021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0641106Z return mod(**inputs) 2025-08-26T20:37:15.0641406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0641493Z outputs = self.mobilebert( 2025-08-26T20:37:15.0641799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0641889Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0642197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0642276Z layer_outputs = layer_module( 2025-08-26T20:37:15.0642593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0642729Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0643068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0643193Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0643196Z 2025-08-26T20:37:15.0643313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0643527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0643599Z return mod(**inputs) 2025-08-26T20:37:15.0643909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0643985Z outputs = self.mobilebert( 2025-08-26T20:37:15.0644316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0644399Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0644704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0644847Z layer_outputs = layer_module( 2025-08-26T20:37:15.0645157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0645354Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0645660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0645770Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0645774Z 2025-08-26T20:37:15.0645889Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0646106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0646186Z return mod(**inputs) 2025-08-26T20:37:15.0646497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0646582Z outputs = self.mobilebert( 2025-08-26T20:37:15.0646886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0646966Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0647280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0647358Z layer_outputs = layer_module( 2025-08-26T20:37:15.0647682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0647855Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0648181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0648320Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0648624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0648740Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0648744Z 2025-08-26T20:37:15.0648855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0649080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0649153Z return mod(**inputs) 2025-08-26T20:37:15.0649470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0649555Z outputs = self.mobilebert( 2025-08-26T20:37:15.0649863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0649977Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0650285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0650374Z layer_outputs = layer_module( 2025-08-26T20:37:15.0650680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0650852Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0651166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0651320Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0651638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0651757Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0651761Z 2025-08-26T20:37:15.0651882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0652100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0652204Z return mod(**inputs) 2025-08-26T20:37:15.0652519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0652598Z outputs = self.mobilebert( 2025-08-26T20:37:15.0652921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0653003Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0653308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0653394Z layer_outputs = layer_module( 2025-08-26T20:37:15.0653702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0653876Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0654183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0654324Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0654635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0654771Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0655088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0655192Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0655197Z 2025-08-26T20:37:15.0655316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0655535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0655610Z return mod(**inputs) 2025-08-26T20:37:15.0655920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0656000Z outputs = self.mobilebert( 2025-08-26T20:37:15.0656315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0656397Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0656708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0656787Z layer_outputs = layer_module( 2025-08-26T20:37:15.0657121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0657313Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0657623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0657753Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0658062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0658178Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0658191Z 2025-08-26T20:37:15.0658305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0658522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0658622Z return mod(**inputs) 2025-08-26T20:37:15.0658930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0659013Z outputs = self.mobilebert( 2025-08-26T20:37:15.0659341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0659420Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0659734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0659812Z layer_outputs = layer_module( 2025-08-26T20:37:15.0660125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0660220Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0660528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0660615Z self_outputs = self.self( 2025-08-26T20:37:15.0660921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0661011Z self.value(value_tensor) 2025-08-26T20:37:15.0661015Z 2025-08-26T20:37:15.0661128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0661353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0661426Z return mod(**inputs) 2025-08-26T20:37:15.0661734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0661819Z outputs = self.mobilebert( 2025-08-26T20:37:15.0662126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0662214Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0662519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0662599Z layer_outputs = layer_module( 2025-08-26T20:37:15.0662906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0663082Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0663390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0663510Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0663817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0663923Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0663929Z 2025-08-26T20:37:15.0664041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0664264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0664340Z return mod(**inputs) 2025-08-26T20:37:15.0664656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0664735Z outputs = self.mobilebert( 2025-08-26T20:37:15.0665065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0665155Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0665462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0665549Z layer_outputs = layer_module( 2025-08-26T20:37:15.0665872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0666053Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0666382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0666499Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0666812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0666909Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0667221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0667322Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0667329Z 2025-08-26T20:37:15.0667440Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0667663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0667738Z return mod(**inputs) 2025-08-26T20:37:15.0668049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0668128Z outputs = self.mobilebert( 2025-08-26T20:37:15.0668449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0668529Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0668834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0668921Z layer_outputs = layer_module( 2025-08-26T20:37:15.0669227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0669327Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0669637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0669712Z self_outputs = self.self( 2025-08-26T20:37:15.0670029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0670104Z self.query(query_tensor) 2025-08-26T20:37:15.0670108Z 2025-08-26T20:37:15.0670223Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0670435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0670505Z return mod(**inputs) 2025-08-26T20:37:15.0670843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0670922Z outputs = self.mobilebert( 2025-08-26T20:37:15.0671232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0671314Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0671626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0671705Z layer_outputs = layer_module( 2025-08-26T20:37:15.0672041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0672146Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0672453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0672563Z self_outputs = self.self( 2025-08-26T20:37:15.0672875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0672966Z self.key(key_tensor) 2025-08-26T20:37:15.0672979Z 2025-08-26T20:37:15.0673068Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0673154Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0673275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0673494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0673565Z return mod(**inputs) 2025-08-26T20:37:15.0673887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0673965Z outputs = self.mobilebert( 2025-08-26T20:37:15.0674285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0674365Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0674680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0674762Z layer_outputs = layer_module( 2025-08-26T20:37:15.0675072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0675171Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0675489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0675632Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0675945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0676040Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0676051Z 2025-08-26T20:37:15.0676165Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0676381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0676464Z return mod(**inputs) 2025-08-26T20:37:15.0676773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0676860Z outputs = self.mobilebert( 2025-08-26T20:37:15.0677166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0677243Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0677562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0677662Z layer_outputs = layer_module( 2025-08-26T20:37:15.0677974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0678063Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0678365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0678505Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0678807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0678969Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0679276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0679386Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0679421Z 2025-08-26T20:37:15.0679619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0679845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0679955Z return mod(**inputs) 2025-08-26T20:37:15.0680270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0680359Z outputs = self.mobilebert( 2025-08-26T20:37:15.0680665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0680747Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0681060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0681137Z layer_outputs = layer_module( 2025-08-26T20:37:15.0681451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0681562Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0681871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0681996Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0682302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0682404Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0682407Z 2025-08-26T20:37:15.0682522Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0682740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0682812Z return mod(**inputs) 2025-08-26T20:37:15.0683116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0683202Z outputs = self.mobilebert( 2025-08-26T20:37:15.0683505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0683590Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0683892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0683973Z layer_outputs = layer_module( 2025-08-26T20:37:15.0684279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0684384Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0684720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0684846Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0685163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0685289Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0685292Z 2025-08-26T20:37:15.0685427Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0685648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0685722Z return mod(**inputs) 2025-08-26T20:37:15.0686053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0686135Z outputs = self.mobilebert( 2025-08-26T20:37:15.0686449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0686547Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0686853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0686961Z layer_outputs = layer_module( 2025-08-26T20:37:15.0687272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0687386Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0687705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0687845Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0688161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0688259Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0688263Z 2025-08-26T20:37:15.0688385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0688605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0688696Z return mod(**inputs) 2025-08-26T20:37:15.0688981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0689054Z outputs = self.mobilebert( 2025-08-26T20:37:15.0689345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0689421Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0689711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0689784Z layer_outputs = layer_module( 2025-08-26T20:37:15.0690067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0690171Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0690458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0690590Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0690878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0691010Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0691293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0691386Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0691411Z 2025-08-26T20:37:15.0691525Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0691720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0691796Z return mod(**inputs) 2025-08-26T20:37:15.0692072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0692144Z outputs = self.mobilebert( 2025-08-26T20:37:15.0692429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0692503Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0692804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0692879Z layer_outputs = layer_module( 2025-08-26T20:37:15.0693162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0693273Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0693552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0693693Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0722004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0722280Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0722288Z 2025-08-26T20:37:15.0722459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0722702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0722781Z return mod(**inputs) 2025-08-26T20:37:15.0723139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0723233Z outputs = self.mobilebert( 2025-08-26T20:37:15.0723543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0723629Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0723937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0724015Z layer_outputs = layer_module( 2025-08-26T20:37:15.0724316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0724428Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0724729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0724860Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0725161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0725272Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0725282Z 2025-08-26T20:37:15.0725387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0725584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0725662Z return mod(**inputs) 2025-08-26T20:37:15.0725939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0726024Z outputs = self.mobilebert( 2025-08-26T20:37:15.0726483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0726565Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0726850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0726937Z layer_outputs = layer_module( 2025-08-26T20:37:15.0727215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0727311Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0727584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0727779Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0728056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0728153Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0728196Z 2025-08-26T20:37:15.0728305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0728517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0728647Z return mod(**inputs) 2025-08-26T20:37:15.0728927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0729011Z outputs = self.mobilebert( 2025-08-26T20:37:15.0729287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0729366Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0729642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0729710Z layer_outputs = layer_module( 2025-08-26T20:37:15.0729989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0730080Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0730358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0730481Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0730761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0730880Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0731155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0731260Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0731265Z 2025-08-26T20:37:15.0731372Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0731578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0731646Z return mod(**inputs) 2025-08-26T20:37:15.0731919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0731998Z outputs = self.mobilebert( 2025-08-26T20:37:15.0732271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0732353Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0732628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0732701Z layer_outputs = layer_module( 2025-08-26T20:37:15.0733003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0733099Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0733372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0733493Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0733761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0733852Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0733856Z 2025-08-26T20:37:15.0733993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0734192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0734267Z return mod(**inputs) 2025-08-26T20:37:15.0734549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0734651Z outputs = self.mobilebert( 2025-08-26T20:37:15.0734930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0735031Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0735321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0735393Z layer_outputs = layer_module( 2025-08-26T20:37:15.0735689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0735782Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0736081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0736195Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0736474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0736595Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0736599Z 2025-08-26T20:37:15.0736702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0736903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0736969Z return mod(**inputs) 2025-08-26T20:37:15.0737262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0737336Z outputs = self.mobilebert( 2025-08-26T20:37:15.0737623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0737707Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0737994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0738075Z layer_outputs = layer_module( 2025-08-26T20:37:15.0738360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0738454Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0738748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0738876Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0739169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0739272Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0739278Z 2025-08-26T20:37:15.0739390Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0739589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0739657Z return mod(**inputs) 2025-08-26T20:37:15.0739945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0740019Z outputs = self.mobilebert( 2025-08-26T20:37:15.0740305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0740397Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0740673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0740751Z layer_outputs = layer_module( 2025-08-26T20:37:15.0741052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0741154Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0741453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0741580Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0741860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0741980Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0742266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0742360Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0742366Z 2025-08-26T20:37:15.0742478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0742677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0742747Z return mod(**inputs) 2025-08-26T20:37:15.0743031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0743105Z outputs = self.mobilebert( 2025-08-26T20:37:15.0743388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0743461Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0743748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0743820Z layer_outputs = layer_module( 2025-08-26T20:37:15.0744100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0744233Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0744511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0744606Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0744610Z 2025-08-26T20:37:15.0744712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0744911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0744984Z return mod(**inputs) 2025-08-26T20:37:15.0745266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0745346Z outputs = self.mobilebert( 2025-08-26T20:37:15.0745641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0745725Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0746009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0746082Z layer_outputs = layer_module( 2025-08-26T20:37:15.0746372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0746493Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0746798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0746915Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0746919Z 2025-08-26T20:37:15.0747021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0747242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0747306Z return mod(**inputs) 2025-08-26T20:37:15.0747590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0747681Z outputs = self.mobilebert( 2025-08-26T20:37:15.0747960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0748035Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0748328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0748411Z layer_outputs = layer_module( 2025-08-26T20:37:15.0748702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0748887Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0749185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0749289Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0749301Z 2025-08-26T20:37:15.0749410Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0749618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0749700Z return mod(**inputs) 2025-08-26T20:37:15.0749997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0750080Z outputs = self.mobilebert( 2025-08-26T20:37:15.0750381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0750464Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0750769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0750848Z layer_outputs = layer_module( 2025-08-26T20:37:15.0751150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0751329Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0751610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0751744Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0752031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0752152Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0752158Z 2025-08-26T20:37:15.0752264Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0752468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0752536Z return mod(**inputs) 2025-08-26T20:37:15.0752814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0752891Z outputs = self.mobilebert( 2025-08-26T20:37:15.0753181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0753283Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0753583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0753660Z layer_outputs = layer_module( 2025-08-26T20:37:15.0754033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0754204Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0754538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0754670Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0754976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0755069Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0755073Z 2025-08-26T20:37:15.0755184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0755414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0755487Z return mod(**inputs) 2025-08-26T20:37:15.0755796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0755871Z outputs = self.mobilebert( 2025-08-26T20:37:15.0756173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0756257Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0756564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0756647Z layer_outputs = layer_module( 2025-08-26T20:37:15.0756955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0757128Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0757442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0757574Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0757894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0758023Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0758335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0758435Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0758439Z 2025-08-26T20:37:15.0758557Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0758768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0758867Z return mod(**inputs) 2025-08-26T20:37:15.0759169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0759245Z outputs = self.mobilebert( 2025-08-26T20:37:15.0759646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0759728Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0760020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0760101Z layer_outputs = layer_module( 2025-08-26T20:37:15.0760423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0760613Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0760923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0761073Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0761408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0761494Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0761499Z 2025-08-26T20:37:15.0761616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0761826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0761907Z return mod(**inputs) 2025-08-26T20:37:15.0762204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0762281Z outputs = self.mobilebert( 2025-08-26T20:37:15.0762582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0762661Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0762970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0763048Z layer_outputs = layer_module( 2025-08-26T20:37:15.0763345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0763439Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0763734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0763822Z self_outputs = self.self( 2025-08-26T20:37:15.0764119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0764208Z self.value(value_tensor) 2025-08-26T20:37:15.0764212Z 2025-08-26T20:37:15.0764323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0764533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0764615Z return mod(**inputs) 2025-08-26T20:37:15.0764912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0764995Z outputs = self.mobilebert( 2025-08-26T20:37:15.0765293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0765370Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0765673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0765768Z layer_outputs = layer_module( 2025-08-26T20:37:15.0766075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0766247Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0766556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0766674Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0766976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0767091Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0767096Z 2025-08-26T20:37:15.0767207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0767420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0767515Z return mod(**inputs) 2025-08-26T20:37:15.0767820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0767913Z outputs = self.mobilebert( 2025-08-26T20:37:15.0768207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0768291Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0768586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0768668Z layer_outputs = layer_module( 2025-08-26T20:37:15.0768967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0769138Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0769444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0769557Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0769863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0769958Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0770261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0770357Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0770363Z 2025-08-26T20:37:15.0770473Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0770689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0770758Z return mod(**inputs) 2025-08-26T20:37:15.0771057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0771130Z outputs = self.mobilebert( 2025-08-26T20:37:15.0771425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0771507Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0771801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0771882Z layer_outputs = layer_module( 2025-08-26T20:37:15.0772179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0772276Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0772590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0772679Z self_outputs = self.self( 2025-08-26T20:37:15.0772967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0773040Z self.query(query_tensor) 2025-08-26T20:37:15.0773044Z 2025-08-26T20:37:15.0773152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0773357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0773428Z return mod(**inputs) 2025-08-26T20:37:15.0773749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0773827Z outputs = self.mobilebert( 2025-08-26T20:37:15.0774128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0774226Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0774520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0774625Z layer_outputs = layer_module( 2025-08-26T20:37:15.0774923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0775021Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0775327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0775406Z self_outputs = self.self( 2025-08-26T20:37:15.0775684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0775752Z self.key(key_tensor) 2025-08-26T20:37:15.0775755Z 2025-08-26T20:37:15.0775850Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0775930Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0776042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0776240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0776305Z return mod(**inputs) 2025-08-26T20:37:15.0776589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0776659Z outputs = self.mobilebert( 2025-08-26T20:37:15.0776947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0777021Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0777301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0777384Z layer_outputs = layer_module( 2025-08-26T20:37:15.0777664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0777763Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0778058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0778199Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0778498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0778591Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0778595Z 2025-08-26T20:37:15.0778712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0778920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0779017Z return mod(**inputs) 2025-08-26T20:37:15.0779316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0779395Z outputs = self.mobilebert( 2025-08-26T20:37:15.0779698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0779777Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0780080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0780157Z layer_outputs = layer_module( 2025-08-26T20:37:15.0780517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0780606Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0780904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0781055Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0781378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0781516Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0781813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0781907Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0781916Z 2025-08-26T20:37:15.0782028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0782233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0782312Z return mod(**inputs) 2025-08-26T20:37:15.0782616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0782700Z outputs = self.mobilebert( 2025-08-26T20:37:15.0782997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0783073Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0783374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0783448Z layer_outputs = layer_module( 2025-08-26T20:37:15.0783755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0783858Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0784154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0784285Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0784584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0784685Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0784690Z 2025-08-26T20:37:15.0784798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0785014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0785085Z return mod(**inputs) 2025-08-26T20:37:15.0785383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0785470Z outputs = self.mobilebert( 2025-08-26T20:37:15.0785785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0785872Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0786169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0786247Z layer_outputs = layer_module( 2025-08-26T20:37:15.0786562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0786662Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0786988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0787112Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0787415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0787557Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0787561Z 2025-08-26T20:37:15.0787673Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0787892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0787982Z return mod(**inputs) 2025-08-26T20:37:15.0788287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0788363Z outputs = self.mobilebert( 2025-08-26T20:37:15.0788662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0788747Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0789041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0789126Z layer_outputs = layer_module( 2025-08-26T20:37:15.0789424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0789529Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0789827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0789958Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0790260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0790350Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0790354Z 2025-08-26T20:37:15.0790469Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0790679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0790753Z return mod(**inputs) 2025-08-26T20:37:15.0791055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0791134Z outputs = self.mobilebert( 2025-08-26T20:37:15.0791438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0791513Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0791815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0791892Z layer_outputs = layer_module( 2025-08-26T20:37:15.0792189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0792298Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0792613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0792757Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0793056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0793186Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0793490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0793590Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0793593Z 2025-08-26T20:37:15.0793730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0793945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0794022Z return mod(**inputs) 2025-08-26T20:37:15.0794340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0794413Z outputs = self.mobilebert( 2025-08-26T20:37:15.0794732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0794807Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0795109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0795182Z layer_outputs = layer_module( 2025-08-26T20:37:15.0795481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0795586Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0795885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0796013Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0796481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0796588Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0796593Z 2025-08-26T20:37:15.0796703Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0796912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0796993Z return mod(**inputs) 2025-08-26T20:37:15.0797294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0797380Z outputs = self.mobilebert( 2025-08-26T20:37:15.0797681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0797761Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0798063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0798139Z layer_outputs = layer_module( 2025-08-26T20:37:15.0798443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0798542Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0798842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0798963Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0799258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0799489Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0799498Z 2025-08-26T20:37:15.0799617Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0799838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0799912Z return mod(**inputs) 2025-08-26T20:37:15.0800229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0800307Z outputs = self.mobilebert( 2025-08-26T20:37:15.0800610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0800729Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0801043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0801127Z layer_outputs = layer_module( 2025-08-26T20:37:15.0801450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0801550Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0801882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0802015Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0802319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0802411Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0802415Z 2025-08-26T20:37:15.0802532Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0802739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0802813Z return mod(**inputs) 2025-08-26T20:37:15.0803117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0803196Z outputs = self.mobilebert( 2025-08-26T20:37:15.0803495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0803566Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0803838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0803913Z layer_outputs = layer_module( 2025-08-26T20:37:15.0804192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0804291Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0804574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0804707Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0804991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0805112Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0805398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0805490Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0805493Z 2025-08-26T20:37:15.0805607Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0805804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0805871Z return mod(**inputs) 2025-08-26T20:37:15.0806174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0806249Z outputs = self.mobilebert( 2025-08-26T20:37:15.0806538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0806611Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0806899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0806969Z layer_outputs = layer_module( 2025-08-26T20:37:15.0807273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0807380Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0807661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0807808Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0808089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0808192Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0808200Z 2025-08-26T20:37:15.0808302Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0808500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0808574Z return mod(**inputs) 2025-08-26T20:37:15.0808856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0808934Z outputs = self.mobilebert( 2025-08-26T20:37:15.0809217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0809291Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0809576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0809649Z layer_outputs = layer_module( 2025-08-26T20:37:15.0809936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0810031Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0810319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0810446Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0810740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0810870Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0810875Z 2025-08-26T20:37:15.0810984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0811200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0811272Z return mod(**inputs) 2025-08-26T20:37:15.0811572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0811651Z outputs = self.mobilebert( 2025-08-26T20:37:15.0811926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0812001Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0812281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0812355Z layer_outputs = layer_module( 2025-08-26T20:37:15.0812659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0812756Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0813045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0813173Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0813472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0813589Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0813593Z 2025-08-26T20:37:15.0813704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0813923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0814010Z return mod(**inputs) 2025-08-26T20:37:15.0814298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0814370Z outputs = self.mobilebert( 2025-08-26T20:37:15.0814666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0814747Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0815028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0815108Z layer_outputs = layer_module( 2025-08-26T20:37:15.0815387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0815484Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0815762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0815885Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0816168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0816292Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0816599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0816696Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0816700Z 2025-08-26T20:37:15.0816817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0817034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0817102Z return mod(**inputs) 2025-08-26T20:37:15.0817390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0817465Z outputs = self.mobilebert( 2025-08-26T20:37:15.0817755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0817831Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0818115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0818197Z layer_outputs = layer_module( 2025-08-26T20:37:15.0818497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0818629Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0818948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0819043Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0819054Z 2025-08-26T20:37:15.0819163Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0819371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0819447Z return mod(**inputs) 2025-08-26T20:37:15.0819745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0819826Z outputs = self.mobilebert( 2025-08-26T20:37:15.0820139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0820215Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0820521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0820620Z layer_outputs = layer_module( 2025-08-26T20:37:15.0820925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0821072Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0821367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0821496Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0821500Z 2025-08-26T20:37:15.0821610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0821825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0821893Z return mod(**inputs) 2025-08-26T20:37:15.0822195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0822276Z outputs = self.mobilebert( 2025-08-26T20:37:15.0822570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0822657Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0822953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0823031Z layer_outputs = layer_module( 2025-08-26T20:37:15.0823328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0823500Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0823806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0823909Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0823915Z 2025-08-26T20:37:15.0824032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0824239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0824319Z return mod(**inputs) 2025-08-26T20:37:15.0824615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0824691Z outputs = self.mobilebert( 2025-08-26T20:37:15.0824993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0825071Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0825376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0825452Z layer_outputs = layer_module( 2025-08-26T20:37:15.0825771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0825950Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0826249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0826388Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0826684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0826806Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0826811Z 2025-08-26T20:37:15.0826919Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0827129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0827221Z return mod(**inputs) 2025-08-26T20:37:15.0827511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0827612Z outputs = self.mobilebert( 2025-08-26T20:37:15.0827908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0827986Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0828288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0828365Z layer_outputs = layer_module( 2025-08-26T20:37:15.0828670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0828834Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0829136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0829267Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0829564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0829665Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0829669Z 2025-08-26T20:37:15.0829781Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0830003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0830078Z return mod(**inputs) 2025-08-26T20:37:15.0830388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0830463Z outputs = self.mobilebert( 2025-08-26T20:37:15.0830767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0830849Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0831160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0831241Z layer_outputs = layer_module( 2025-08-26T20:37:15.0831533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0831697Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0831998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0832128Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0832451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0832584Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0832887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0832985Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0832989Z 2025-08-26T20:37:15.0833098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0833312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0833383Z return mod(**inputs) 2025-08-26T20:37:15.0833716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0833795Z outputs = self.mobilebert( 2025-08-26T20:37:15.0834089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0834192Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0834490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0834597Z layer_outputs = layer_module( 2025-08-26T20:37:15.0834890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0835068Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0835365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0835482Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0835785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0835874Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0835878Z 2025-08-26T20:37:15.0835994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0836203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0836273Z return mod(**inputs) 2025-08-26T20:37:15.0836573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0836651Z outputs = self.mobilebert( 2025-08-26T20:37:15.0836949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0837024Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0837324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0837399Z layer_outputs = layer_module( 2025-08-26T20:37:15.0837689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0837788Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0838081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0838165Z self_outputs = self.self( 2025-08-26T20:37:15.0838458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0838537Z self.value(value_tensor) 2025-08-26T20:37:15.0838550Z 2025-08-26T20:37:15.0838660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0838869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0838966Z return mod(**inputs) 2025-08-26T20:37:15.0839273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0839360Z outputs = self.mobilebert( 2025-08-26T20:37:15.0839762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0839850Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0840166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0840242Z layer_outputs = layer_module( 2025-08-26T20:37:15.0840582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0840759Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0841068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0841187Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0841487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0841579Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0841583Z 2025-08-26T20:37:15.0841687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0841890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0841960Z return mod(**inputs) 2025-08-26T20:37:15.0842229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0842311Z outputs = self.mobilebert( 2025-08-26T20:37:15.0842584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0842659Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0842934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0843005Z layer_outputs = layer_module( 2025-08-26T20:37:15.0843286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0843445Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0843733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0843844Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0844131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0844219Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0844497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0844595Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0844599Z 2025-08-26T20:37:15.0844698Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0844902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0844969Z return mod(**inputs) 2025-08-26T20:37:15.0845250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0845329Z outputs = self.mobilebert( 2025-08-26T20:37:15.0845630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0845709Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0845976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0846056Z layer_outputs = layer_module( 2025-08-26T20:37:15.0846331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0846415Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0846714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0846789Z self_outputs = self.self( 2025-08-26T20:37:15.0847076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0847170Z self.query(query_tensor) 2025-08-26T20:37:15.0847173Z 2025-08-26T20:37:15.0847275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0847479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0847566Z return mod(**inputs) 2025-08-26T20:37:15.0847851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0847923Z outputs = self.mobilebert( 2025-08-26T20:37:15.0848205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0848281Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0848558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0848638Z layer_outputs = layer_module( 2025-08-26T20:37:15.0848919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0849012Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0849294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0849361Z self_outputs = self.self( 2025-08-26T20:37:15.0849645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0849713Z self.key(key_tensor) 2025-08-26T20:37:15.0849716Z 2025-08-26T20:37:15.0849812Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0849894Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0849998Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0850204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0850273Z return mod(**inputs) 2025-08-26T20:37:15.0850558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0850632Z outputs = self.mobilebert( 2025-08-26T20:37:15.0850919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0850993Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0851272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0851353Z layer_outputs = layer_module( 2025-08-26T20:37:15.0851633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0851724Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0852019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0852149Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0852439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0852524Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0852527Z 2025-08-26T20:37:15.0852638Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0852835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0852925Z return mod(**inputs) 2025-08-26T20:37:15.0853204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0853276Z outputs = self.mobilebert( 2025-08-26T20:37:15.0853579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0853650Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0853953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0854027Z layer_outputs = layer_module( 2025-08-26T20:37:15.0854304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0854396Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0854677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0854806Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0855087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0855222Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0855516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0855623Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0855626Z 2025-08-26T20:37:15.0855744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0855951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0856028Z return mod(**inputs) 2025-08-26T20:37:15.0856327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0856405Z outputs = self.mobilebert( 2025-08-26T20:37:15.0856711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0856790Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0857093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0857170Z layer_outputs = layer_module( 2025-08-26T20:37:15.0857464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0857574Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0857871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0858003Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0858279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0858400Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0858405Z 2025-08-26T20:37:15.0858511Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0858713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0858788Z return mod(**inputs) 2025-08-26T20:37:15.0859070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0859156Z outputs = self.mobilebert( 2025-08-26T20:37:15.0859468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0859549Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0859856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0859953Z layer_outputs = layer_module( 2025-08-26T20:37:15.0860259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0860380Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0860683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0860803Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0861105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0861228Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0861232Z 2025-08-26T20:37:15.0861336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0861538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0861608Z return mod(**inputs) 2025-08-26T20:37:15.0861903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0861989Z outputs = self.mobilebert( 2025-08-26T20:37:15.0862282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0862373Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0862651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0862731Z layer_outputs = layer_module( 2025-08-26T20:37:15.0863009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0863103Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0863389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0863514Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0863799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0863883Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0863886Z 2025-08-26T20:37:15.0863995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0864193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0864262Z return mod(**inputs) 2025-08-26T20:37:15.0864551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0864623Z outputs = self.mobilebert( 2025-08-26T20:37:15.0864925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0865001Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0865287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0865366Z layer_outputs = layer_module( 2025-08-26T20:37:15.0865650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0865751Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0866053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0866180Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0866466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0866605Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0866901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0867014Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0867018Z 2025-08-26T20:37:15.0867131Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0867330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0867397Z return mod(**inputs) 2025-08-26T20:37:15.0867689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0867762Z outputs = self.mobilebert( 2025-08-26T20:37:15.0868052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0868128Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0868407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0868489Z layer_outputs = layer_module( 2025-08-26T20:37:15.0868770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0868873Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0869155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0869276Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0869561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0869649Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0869653Z 2025-08-26T20:37:15.0869766Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0869965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0870039Z return mod(**inputs) 2025-08-26T20:37:15.0870319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0870394Z outputs = self.mobilebert( 2025-08-26T20:37:15.0870702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0870782Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0871084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0871184Z layer_outputs = layer_module( 2025-08-26T20:37:15.0871487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0871589Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0871884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0872009Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0872305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0872448Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0872453Z 2025-08-26T20:37:15.0872564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0872778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0872872Z return mod(**inputs) 2025-08-26T20:37:15.0873175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0873279Z outputs = self.mobilebert( 2025-08-26T20:37:15.0873583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0873667Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0873969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0874046Z layer_outputs = layer_module( 2025-08-26T20:37:15.0874359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0874462Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0874782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0874923Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0875236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0875335Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0875338Z 2025-08-26T20:37:15.0875459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0875679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0875751Z return mod(**inputs) 2025-08-26T20:37:15.0876061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0876138Z outputs = self.mobilebert( 2025-08-26T20:37:15.0876438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0876526Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0876830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0876914Z layer_outputs = layer_module( 2025-08-26T20:37:15.0877216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0877316Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0877628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0877758Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0878090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0878227Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0878536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0878639Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0878643Z 2025-08-26T20:37:15.0878756Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0878979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0879050Z return mod(**inputs) 2025-08-26T20:37:15.0879388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0879554Z outputs = self.mobilebert( 2025-08-26T20:37:15.0879882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0879991Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0880299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0880416Z layer_outputs = layer_module( 2025-08-26T20:37:15.0880726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0880834Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0881131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0881253Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0881558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0881651Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0881655Z 2025-08-26T20:37:15.0881775Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0881982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0882056Z return mod(**inputs) 2025-08-26T20:37:15.0882362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0882439Z outputs = self.mobilebert( 2025-08-26T20:37:15.0882749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0882829Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0883133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0883210Z layer_outputs = layer_module( 2025-08-26T20:37:15.0883506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0883612Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0883912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0884036Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0884334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0884454Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0884464Z 2025-08-26T20:37:15.0884573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0884783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0884915Z return mod(**inputs) 2025-08-26T20:37:15.0885217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0885305Z outputs = self.mobilebert( 2025-08-26T20:37:15.0885648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0885728Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0886035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0886110Z layer_outputs = layer_module( 2025-08-26T20:37:15.0886438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0886542Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0886858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0887011Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0887312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0887405Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0887408Z 2025-08-26T20:37:15.0887513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0887721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0887789Z return mod(**inputs) 2025-08-26T20:37:15.0888107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0888190Z outputs = self.mobilebert( 2025-08-26T20:37:15.0888465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0888543Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0888815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0888887Z layer_outputs = layer_module( 2025-08-26T20:37:15.0889168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0889260Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0889539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0889660Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0889969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0890091Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0890362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0890460Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0890464Z 2025-08-26T20:37:15.0890565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0890768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0890833Z return mod(**inputs) 2025-08-26T20:37:15.0891145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0891218Z outputs = self.mobilebert( 2025-08-26T20:37:15.0891517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0891602Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0891891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0891968Z layer_outputs = layer_module( 2025-08-26T20:37:15.0892240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0892358Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0892659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0892745Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0892748Z 2025-08-26T20:37:15.0892855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0893049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0893141Z return mod(**inputs) 2025-08-26T20:37:15.0893412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0893503Z outputs = self.mobilebert( 2025-08-26T20:37:15.0893792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0893867Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0894154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0894228Z layer_outputs = layer_module( 2025-08-26T20:37:15.0894507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0894636Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0894920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0895038Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0895042Z 2025-08-26T20:37:15.0895143Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0895341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0895406Z return mod(**inputs) 2025-08-26T20:37:15.0895676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0895756Z outputs = self.mobilebert( 2025-08-26T20:37:15.0896028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0896104Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0896565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0896642Z layer_outputs = layer_module( 2025-08-26T20:37:15.0896925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0897085Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0897364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0897460Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0897464Z 2025-08-26T20:37:15.0897575Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0897769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0897888Z return mod(**inputs) 2025-08-26T20:37:15.0898171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0898240Z outputs = self.mobilebert( 2025-08-26T20:37:15.0898518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0898589Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0898859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0898937Z layer_outputs = layer_module( 2025-08-26T20:37:15.0899238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0899406Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0899682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0899841Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0900137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0900228Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0900231Z 2025-08-26T20:37:15.0900338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0900528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0900602Z return mod(**inputs) 2025-08-26T20:37:15.0900874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0900945Z outputs = self.mobilebert( 2025-08-26T20:37:15.0901223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0901298Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0901586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0901658Z layer_outputs = layer_module( 2025-08-26T20:37:15.0901954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0902113Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0902395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0902529Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0902817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0902910Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0902914Z 2025-08-26T20:37:15.0903017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0903207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0903280Z return mod(**inputs) 2025-08-26T20:37:15.0903556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0903633Z outputs = self.mobilebert( 2025-08-26T20:37:15.0903912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0903988Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0904276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0904349Z layer_outputs = layer_module( 2025-08-26T20:37:15.0904632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0904789Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0905073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0905195Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0905483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0905611Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0905884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0906005Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0906009Z 2025-08-26T20:37:15.0906110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0906327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0906392Z return mod(**inputs) 2025-08-26T20:37:15.0906662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0906740Z outputs = self.mobilebert( 2025-08-26T20:37:15.0907010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0907088Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0907361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0907433Z layer_outputs = layer_module( 2025-08-26T20:37:15.0907709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0907868Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0908145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0908257Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0908543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0908626Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0908630Z 2025-08-26T20:37:15.0908734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0908940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0909007Z return mod(**inputs) 2025-08-26T20:37:15.0909295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0909371Z outputs = self.mobilebert( 2025-08-26T20:37:15.0909646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0909728Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0910007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0910087Z layer_outputs = layer_module( 2025-08-26T20:37:15.0910364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0910489Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0910776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0910849Z self_outputs = self.self( 2025-08-26T20:37:15.0911139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0911211Z self.value(value_tensor) 2025-08-26T20:37:15.0911214Z 2025-08-26T20:37:15.0911325Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0911520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0911604Z return mod(**inputs) 2025-08-26T20:37:15.0911893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0911964Z outputs = self.mobilebert( 2025-08-26T20:37:15.0912272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0912346Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0912650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0912720Z layer_outputs = layer_module( 2025-08-26T20:37:15.0912999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0913167Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0913450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0913571Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0913862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0913952Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0913964Z 2025-08-26T20:37:15.0914074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0914284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0914362Z return mod(**inputs) 2025-08-26T20:37:15.0914657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0914738Z outputs = self.mobilebert( 2025-08-26T20:37:15.0915036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0915112Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0915417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0915493Z layer_outputs = layer_module( 2025-08-26T20:37:15.0915795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0915964Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0916266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0916388Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0916687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0916788Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0917100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0917213Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0917217Z 2025-08-26T20:37:15.0917327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0917540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0917621Z return mod(**inputs) 2025-08-26T20:37:15.0917939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0918027Z outputs = self.mobilebert( 2025-08-26T20:37:15.0918358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0918440Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0918755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0918854Z layer_outputs = layer_module( 2025-08-26T20:37:15.0919179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0919295Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0919686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0919772Z self_outputs = self.self( 2025-08-26T20:37:15.0920092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0920181Z self.query(query_tensor) 2025-08-26T20:37:15.0920185Z 2025-08-26T20:37:15.0920300Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0920525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0920603Z return mod(**inputs) 2025-08-26T20:37:15.0920909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0920998Z outputs = self.mobilebert( 2025-08-26T20:37:15.0921305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0921396Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0921705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0921787Z layer_outputs = layer_module( 2025-08-26T20:37:15.0922137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0922231Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0922527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0922599Z self_outputs = self.self( 2025-08-26T20:37:15.0922886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0922957Z self.key(key_tensor) 2025-08-26T20:37:15.0922961Z 2025-08-26T20:37:15.0923046Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0923135Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0923244Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0923451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0923518Z return mod(**inputs) 2025-08-26T20:37:15.0923800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0923900Z outputs = self.mobilebert( 2025-08-26T20:37:15.0924187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0924269Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0924554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0924633Z layer_outputs = layer_module( 2025-08-26T20:37:15.0924913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0924998Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0925305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0925433Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0925724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.0925829Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0925851Z 2025-08-26T20:37:15.0925958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0926174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0926238Z return mod(**inputs) 2025-08-26T20:37:15.0926519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0926589Z outputs = self.mobilebert( 2025-08-26T20:37:15.0926871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0926945Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0927228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0927310Z layer_outputs = layer_module( 2025-08-26T20:37:15.0927590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0927682Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0927964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0928088Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.0928379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.0928506Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0928796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0928895Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0928899Z 2025-08-26T20:37:15.0929008Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0929214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0929280Z return mod(**inputs) 2025-08-26T20:37:15.0929559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0929630Z outputs = self.mobilebert( 2025-08-26T20:37:15.0929910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0929981Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0930253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0930352Z layer_outputs = layer_module( 2025-08-26T20:37:15.0930626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0930731Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0931021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0931150Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0931458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0931568Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0931573Z 2025-08-26T20:37:15.0931686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0931884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0931976Z return mod(**inputs) 2025-08-26T20:37:15.0932264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0932355Z outputs = self.mobilebert( 2025-08-26T20:37:15.0932646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0932720Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0933008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0933082Z layer_outputs = layer_module( 2025-08-26T20:37:15.0933381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0933477Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0933755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0933873Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0934147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0934265Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0934268Z 2025-08-26T20:37:15.0934371Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0934565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0934640Z return mod(**inputs) 2025-08-26T20:37:15.0934922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0935003Z outputs = self.mobilebert( 2025-08-26T20:37:15.0935294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0935375Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0935651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0935721Z layer_outputs = layer_module( 2025-08-26T20:37:15.0936000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0936096Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0936377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0936502Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0936794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0936890Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0936893Z 2025-08-26T20:37:15.0936996Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0937197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0937263Z return mod(**inputs) 2025-08-26T20:37:15.0937544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0937615Z outputs = self.mobilebert( 2025-08-26T20:37:15.0937913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0937995Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0938278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0938374Z layer_outputs = layer_module( 2025-08-26T20:37:15.0938664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0938777Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0939070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0939194Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0939477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0939601Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0939887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0939980Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0939984Z 2025-08-26T20:37:15.0940087Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0940301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0940366Z return mod(**inputs) 2025-08-26T20:37:15.0940643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0940713Z outputs = self.mobilebert( 2025-08-26T20:37:15.0940983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0941063Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0941332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0941412Z layer_outputs = layer_module( 2025-08-26T20:37:15.0941683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0941784Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0942059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0942172Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0942461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0942546Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0942550Z 2025-08-26T20:37:15.0942661Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0942876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0942947Z return mod(**inputs) 2025-08-26T20:37:15.0943240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0943319Z outputs = self.mobilebert( 2025-08-26T20:37:15.0943620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0943698Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0943998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0944091Z layer_outputs = layer_module( 2025-08-26T20:37:15.0944390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0944499Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0944814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0944939Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0945246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0945358Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0945370Z 2025-08-26T20:37:15.0945474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0945672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0945747Z return mod(**inputs) 2025-08-26T20:37:15.0946029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0946109Z outputs = self.mobilebert( 2025-08-26T20:37:15.0946392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0946465Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0946753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0946823Z layer_outputs = layer_module( 2025-08-26T20:37:15.0947108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0947202Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0947482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0947613Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0947892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0947985Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0947989Z 2025-08-26T20:37:15.0948094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0948298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0948365Z return mod(**inputs) 2025-08-26T20:37:15.0948640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0948721Z outputs = self.mobilebert( 2025-08-26T20:37:15.0949000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0949082Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0949377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0949452Z layer_outputs = layer_module( 2025-08-26T20:37:15.0949743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0949842Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0950133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0950259Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0950565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0950691Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0950972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0951092Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0951096Z 2025-08-26T20:37:15.0951209Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0951443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0951515Z return mod(**inputs) 2025-08-26T20:37:15.0951809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0951894Z outputs = self.mobilebert( 2025-08-26T20:37:15.0952202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0952288Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0952589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0952676Z layer_outputs = layer_module( 2025-08-26T20:37:15.0952977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0953081Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0953392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0953512Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0953821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0953911Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0953915Z 2025-08-26T20:37:15.0954027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0954250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0954325Z return mod(**inputs) 2025-08-26T20:37:15.0954635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0954716Z outputs = self.mobilebert( 2025-08-26T20:37:15.0955027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0955105Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0955404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0955489Z layer_outputs = layer_module( 2025-08-26T20:37:15.0955797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0955906Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0956240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.0956364Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.0956677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0956799Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0956803Z 2025-08-26T20:37:15.0956924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0957139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0957237Z return mod(**inputs) 2025-08-26T20:37:15.0957544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0957624Z outputs = self.mobilebert( 2025-08-26T20:37:15.0957960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0958040Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0958375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0958455Z layer_outputs = layer_module( 2025-08-26T20:37:15.0958772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0958885Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0959204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0959348Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0959913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.0960023Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0960027Z 2025-08-26T20:37:15.0960141Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0960361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0960446Z return mod(**inputs) 2025-08-26T20:37:15.0960754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0960842Z outputs = self.mobilebert( 2025-08-26T20:37:15.0961147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0961227Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0961541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0961622Z layer_outputs = layer_module( 2025-08-26T20:37:15.0961935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.0962040Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.0962353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.0962488Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.0962792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.0962932Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0963234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0963372Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0963376Z 2025-08-26T20:37:15.0963490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0963708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0963788Z return mod(**inputs) 2025-08-26T20:37:15.0964094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0964182Z outputs = self.mobilebert( 2025-08-26T20:37:15.0964509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0964597Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0964902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0965003Z layer_outputs = layer_module( 2025-08-26T20:37:15.0965315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0965472Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0965789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.0965882Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.0965886Z 2025-08-26T20:37:15.0965997Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0966222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0966295Z return mod(**inputs) 2025-08-26T20:37:15.0966607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0966691Z outputs = self.mobilebert( 2025-08-26T20:37:15.0967008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0967090Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0967397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0967485Z layer_outputs = layer_module( 2025-08-26T20:37:15.0967798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.0967935Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.0968237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.0968355Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.0968368Z 2025-08-26T20:37:15.0968479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0968690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0968771Z return mod(**inputs) 2025-08-26T20:37:15.0969067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0969152Z outputs = self.mobilebert( 2025-08-26T20:37:15.0969451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0969530Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0969837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0969914Z layer_outputs = layer_module( 2025-08-26T20:37:15.0970233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0970408Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0970709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.0970819Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.0970823Z 2025-08-26T20:37:15.0970932Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0971152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0971242Z return mod(**inputs) 2025-08-26T20:37:15.0971550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0971627Z outputs = self.mobilebert( 2025-08-26T20:37:15.0971926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0972032Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0972390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0972493Z layer_outputs = layer_module( 2025-08-26T20:37:15.0972791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0972958Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0973269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.0973400Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.0973704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0973802Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0973806Z 2025-08-26T20:37:15.0973924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0974134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0974206Z return mod(**inputs) 2025-08-26T20:37:15.0974511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0974588Z outputs = self.mobilebert( 2025-08-26T20:37:15.0974895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0974973Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0975275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0975359Z layer_outputs = layer_module( 2025-08-26T20:37:15.0975657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0975834Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0976130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0976268Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0976564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.0976654Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.0976658Z 2025-08-26T20:37:15.0976774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0977003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0977087Z return mod(**inputs) 2025-08-26T20:37:15.0977389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0977469Z outputs = self.mobilebert( 2025-08-26T20:37:15.0977771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0977849Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0978174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0978253Z layer_outputs = layer_module( 2025-08-26T20:37:15.0978562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.0978751Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.0979047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.0979573Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.0979877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.0980009Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.0980291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0980390Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0980394Z 2025-08-26T20:37:15.0980497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0980698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0980774Z return mod(**inputs) 2025-08-26T20:37:15.0981058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0981141Z outputs = self.mobilebert( 2025-08-26T20:37:15.0981420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0981494Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0981784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0981855Z layer_outputs = layer_module( 2025-08-26T20:37:15.0982140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0982304Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0982593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0982707Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0982986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0983078Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0983081Z 2025-08-26T20:37:15.0983184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0983389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0983456Z return mod(**inputs) 2025-08-26T20:37:15.0983741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0983836Z outputs = self.mobilebert( 2025-08-26T20:37:15.0984119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0984200Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0984477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0984554Z layer_outputs = layer_module( 2025-08-26T20:37:15.0984836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0984938Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0985226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0985300Z self_outputs = self.self( 2025-08-26T20:37:15.0985608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.0985681Z self.value(value_tensor) 2025-08-26T20:37:15.0985730Z 2025-08-26T20:37:15.0985835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0986040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0986108Z return mod(**inputs) 2025-08-26T20:37:15.0986394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0986467Z outputs = self.mobilebert( 2025-08-26T20:37:15.0986756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0986829Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0987109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0987191Z layer_outputs = layer_module( 2025-08-26T20:37:15.0987469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0987639Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0987918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.0988029Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.0988320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.0988404Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.0988408Z 2025-08-26T20:37:15.0988516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0988713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0988786Z return mod(**inputs) 2025-08-26T20:37:15.0989067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0989147Z outputs = self.mobilebert( 2025-08-26T20:37:15.0989433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0989506Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0989796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0989867Z layer_outputs = layer_module( 2025-08-26T20:37:15.0990147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.0990335Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.0990615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.0990732Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.0991015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.0991110Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.0991411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.0991509Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.0991512Z 2025-08-26T20:37:15.0991624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0991826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0991919Z return mod(**inputs) 2025-08-26T20:37:15.0992198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0992289Z outputs = self.mobilebert( 2025-08-26T20:37:15.0992578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0992651Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0992938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0993010Z layer_outputs = layer_module( 2025-08-26T20:37:15.0993294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0993382Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0993664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0993746Z self_outputs = self.self( 2025-08-26T20:37:15.0994044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.0994127Z self.query(query_tensor) 2025-08-26T20:37:15.0994131Z 2025-08-26T20:37:15.0994239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0994445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0994525Z return mod(**inputs) 2025-08-26T20:37:15.0994821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0994905Z outputs = self.mobilebert( 2025-08-26T20:37:15.0995205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0995283Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0995587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0995662Z layer_outputs = layer_module( 2025-08-26T20:37:15.0995963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0996054Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0996606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.0996687Z self_outputs = self.self( 2025-08-26T20:37:15.0997036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.0997121Z self.key(key_tensor) 2025-08-26T20:37:15.0997125Z 2025-08-26T20:37:15.0997215Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0997308Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.0997423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.0997633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.0997713Z return mod(**inputs) 2025-08-26T20:37:15.0998018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.0998130Z outputs = self.mobilebert( 2025-08-26T20:37:15.0998427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.0998505Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.0998812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.0998916Z layer_outputs = layer_module( 2025-08-26T20:37:15.0999221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.0999337Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.0999700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.0999840Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1000146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1000249Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1000253Z 2025-08-26T20:37:15.1000364Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1000590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1000663Z return mod(**inputs) 2025-08-26T20:37:15.1000972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1001052Z outputs = self.mobilebert( 2025-08-26T20:37:15.1001355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1001442Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1001747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1001843Z layer_outputs = layer_module( 2025-08-26T20:37:15.1002136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1002228Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1002528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1002660Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1002962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1003098Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1003405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1003507Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1003511Z 2025-08-26T20:37:15.1003618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1003855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1003931Z return mod(**inputs) 2025-08-26T20:37:15.1004233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1004312Z outputs = self.mobilebert( 2025-08-26T20:37:15.1004609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1004692Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1004992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1005092Z layer_outputs = layer_module( 2025-08-26T20:37:15.1005390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1005495Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1005832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1005954Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1006277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1006369Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1006373Z 2025-08-26T20:37:15.1006488Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1006696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1006767Z return mod(**inputs) 2025-08-26T20:37:15.1007068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1007145Z outputs = self.mobilebert( 2025-08-26T20:37:15.1007451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1007528Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1007825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1007908Z layer_outputs = layer_module( 2025-08-26T20:37:15.1008204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1008313Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1008607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1008733Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1009029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1009150Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1009156Z 2025-08-26T20:37:15.1009272Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1009480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1009559Z return mod(**inputs) 2025-08-26T20:37:15.1009855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1009933Z outputs = self.mobilebert( 2025-08-26T20:37:15.1010232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1010311Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1010627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1010702Z layer_outputs = layer_module( 2025-08-26T20:37:15.1010988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1011085Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1011363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1011498Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1011793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1011887Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1011891Z 2025-08-26T20:37:15.1011993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1012217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1012282Z return mod(**inputs) 2025-08-26T20:37:15.1012563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1012668Z outputs = self.mobilebert( 2025-08-26T20:37:15.1012946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1013024Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1013305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1013376Z layer_outputs = layer_module( 2025-08-26T20:37:15.1013660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1013758Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1014040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1014166Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1014449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1014571Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1014850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1014950Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1014954Z 2025-08-26T20:37:15.1015054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1015260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1015327Z return mod(**inputs) 2025-08-26T20:37:15.1015606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1015689Z outputs = self.mobilebert( 2025-08-26T20:37:15.1015966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1016046Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1016337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1016420Z layer_outputs = layer_module( 2025-08-26T20:37:15.1016714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1016822Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1017126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1017241Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1017524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1017609Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1017613Z 2025-08-26T20:37:15.1017714Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1017936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1018007Z return mod(**inputs) 2025-08-26T20:37:15.1018295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1018371Z outputs = self.mobilebert( 2025-08-26T20:37:15.1018678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1018753Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1019048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1019129Z layer_outputs = layer_module( 2025-08-26T20:37:15.1019408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1019511Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1019792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1019904Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1020195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1020310Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1020314Z 2025-08-26T20:37:15.1020425Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1020632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1020710Z return mod(**inputs) 2025-08-26T20:37:15.1021009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1021086Z outputs = self.mobilebert( 2025-08-26T20:37:15.1021391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1021464Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1021756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1021830Z layer_outputs = layer_module( 2025-08-26T20:37:15.1022112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1022217Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1022500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1022633Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1022915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1023008Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1023012Z 2025-08-26T20:37:15.1023115Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1023333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1023411Z return mod(**inputs) 2025-08-26T20:37:15.1023690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1023776Z outputs = self.mobilebert( 2025-08-26T20:37:15.1024077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1024154Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1024481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1024555Z layer_outputs = layer_module( 2025-08-26T20:37:15.1024848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1024963Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1025242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1025393Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1025676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1025813Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1026111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1026214Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1026218Z 2025-08-26T20:37:15.1026327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1026536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1026615Z return mod(**inputs) 2025-08-26T20:37:15.1026908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1026993Z outputs = self.mobilebert( 2025-08-26T20:37:15.1027289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1027371Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1027666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1027742Z layer_outputs = layer_module( 2025-08-26T20:37:15.1028046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1028144Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1028449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1028567Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1028864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1028961Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1028965Z 2025-08-26T20:37:15.1029071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1029284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1029356Z return mod(**inputs) 2025-08-26T20:37:15.1029656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1029732Z outputs = self.mobilebert( 2025-08-26T20:37:15.1030045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1030134Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1030435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1030517Z layer_outputs = layer_module( 2025-08-26T20:37:15.1030813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1030913Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1031246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1031366Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1031668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1031805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1031808Z 2025-08-26T20:37:15.1031942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1032150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1032221Z return mod(**inputs) 2025-08-26T20:37:15.1032524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1032601Z outputs = self.mobilebert( 2025-08-26T20:37:15.1032904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1032980Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1033278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1033363Z layer_outputs = layer_module( 2025-08-26T20:37:15.1033659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1033768Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1034061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1034200Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1034497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1034587Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1034590Z 2025-08-26T20:37:15.1034705Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1034917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1034995Z return mod(**inputs) 2025-08-26T20:37:15.1035288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1035366Z outputs = self.mobilebert( 2025-08-26T20:37:15.1035672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1035750Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1036065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1036142Z layer_outputs = layer_module( 2025-08-26T20:37:15.1036455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1036578Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1036885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1037030Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1037332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1037471Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1037771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1037889Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1037903Z 2025-08-26T20:37:15.1038018Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1038240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1038338Z return mod(**inputs) 2025-08-26T20:37:15.1038644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1038750Z outputs = self.mobilebert( 2025-08-26T20:37:15.1039056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1039137Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1039521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1039613Z layer_outputs = layer_module( 2025-08-26T20:37:15.1039932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1040070Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1040382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1040488Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1040494Z 2025-08-26T20:37:15.1040609Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1040834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1040907Z return mod(**inputs) 2025-08-26T20:37:15.1041234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1041313Z outputs = self.mobilebert( 2025-08-26T20:37:15.1041611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1041696Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1041995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1042079Z layer_outputs = layer_module( 2025-08-26T20:37:15.1042384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1042512Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1042823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1042943Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1042946Z 2025-08-26T20:37:15.1043065Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1043275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1043354Z return mod(**inputs) 2025-08-26T20:37:15.1043683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1043762Z outputs = self.mobilebert( 2025-08-26T20:37:15.1044070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1044148Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1044450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1044526Z layer_outputs = layer_module( 2025-08-26T20:37:15.1044842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1045026Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1045324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1045453Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1045457Z 2025-08-26T20:37:15.1045567Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1045804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1045875Z return mod(**inputs) 2025-08-26T20:37:15.1046169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1046251Z outputs = self.mobilebert( 2025-08-26T20:37:15.1046547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1046633Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1046927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1047002Z layer_outputs = layer_module( 2025-08-26T20:37:15.1047307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1047477Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1047779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1047911Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1048215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1048316Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1048319Z 2025-08-26T20:37:15.1048430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1048650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1048720Z return mod(**inputs) 2025-08-26T20:37:15.1049023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1049102Z outputs = self.mobilebert( 2025-08-26T20:37:15.1049398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1049481Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1049776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1049859Z layer_outputs = layer_module( 2025-08-26T20:37:15.1050155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1050354Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1050656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1050780Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1051065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1051154Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1051158Z 2025-08-26T20:37:15.1051274Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1051501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1051576Z return mod(**inputs) 2025-08-26T20:37:15.1051882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1051979Z outputs = self.mobilebert( 2025-08-26T20:37:15.1052286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1052388Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1052694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1052771Z layer_outputs = layer_module( 2025-08-26T20:37:15.1053066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1053239Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1053535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1053674Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1053973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1054103Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1054481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1054573Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1054577Z 2025-08-26T20:37:15.1054689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1054891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1054964Z return mod(**inputs) 2025-08-26T20:37:15.1055248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1055324Z outputs = self.mobilebert( 2025-08-26T20:37:15.1055614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1055689Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1055980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1056052Z layer_outputs = layer_module( 2025-08-26T20:37:15.1056335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1056506Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1056798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1056949Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1057234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1057327Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1057331Z 2025-08-26T20:37:15.1057434Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1057643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1057719Z return mod(**inputs) 2025-08-26T20:37:15.1058015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1058114Z outputs = self.mobilebert( 2025-08-26T20:37:15.1058415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1058498Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1058821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1058896Z layer_outputs = layer_module( 2025-08-26T20:37:15.1059220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1059312Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1059625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1059699Z self_outputs = self.self( 2025-08-26T20:37:15.1059980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1060062Z self.value(value_tensor) 2025-08-26T20:37:15.1060065Z 2025-08-26T20:37:15.1060168Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1060378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1060445Z return mod(**inputs) 2025-08-26T20:37:15.1060724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1060805Z outputs = self.mobilebert( 2025-08-26T20:37:15.1061087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1061171Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1061464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1061548Z layer_outputs = layer_module( 2025-08-26T20:37:15.1061849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1062021Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1062330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1062448Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1062752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1062839Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1062843Z 2025-08-26T20:37:15.1062957Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1063169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1063239Z return mod(**inputs) 2025-08-26T20:37:15.1063557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1063636Z outputs = self.mobilebert( 2025-08-26T20:37:15.1063936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1064014Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1064312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1064394Z layer_outputs = layer_module( 2025-08-26T20:37:15.1064688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1064877Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1065177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1065324Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1065605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1065716Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1066013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1066112Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1066115Z 2025-08-26T20:37:15.1066229Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1066440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1066510Z return mod(**inputs) 2025-08-26T20:37:15.1066811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1066892Z outputs = self.mobilebert( 2025-08-26T20:37:15.1067194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1067274Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1067570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1067651Z layer_outputs = layer_module( 2025-08-26T20:37:15.1067940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1068040Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1068337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1068419Z self_outputs = self.self( 2025-08-26T20:37:15.1068716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1068794Z self.query(query_tensor) 2025-08-26T20:37:15.1068798Z 2025-08-26T20:37:15.1068918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1069127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1069206Z return mod(**inputs) 2025-08-26T20:37:15.1069502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1069577Z outputs = self.mobilebert( 2025-08-26T20:37:15.1069889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1069967Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1070288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1070368Z layer_outputs = layer_module( 2025-08-26T20:37:15.1070676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1070768Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1071065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1071148Z self_outputs = self.self( 2025-08-26T20:37:15.1071461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1071544Z self.key(key_tensor) 2025-08-26T20:37:15.1071547Z 2025-08-26T20:37:15.1071636Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1071721Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1071843Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1072071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1072147Z return mod(**inputs) 2025-08-26T20:37:15.1072464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1072541Z outputs = self.mobilebert( 2025-08-26T20:37:15.1072843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1072919Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1073233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1073311Z layer_outputs = layer_module( 2025-08-26T20:37:15.1073622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1073717Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1074025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1074169Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1074476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1074575Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1074579Z 2025-08-26T20:37:15.1074691Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1074909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1074989Z return mod(**inputs) 2025-08-26T20:37:15.1075295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1075382Z outputs = self.mobilebert( 2025-08-26T20:37:15.1075686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1075774Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1076076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1076153Z layer_outputs = layer_module( 2025-08-26T20:37:15.1076462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1076555Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1076868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1077018Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1077331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1077479Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1077785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1077896Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1077900Z 2025-08-26T20:37:15.1078014Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1078252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1078328Z return mod(**inputs) 2025-08-26T20:37:15.1078633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1078740Z outputs = self.mobilebert( 2025-08-26T20:37:15.1079049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1079166Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1079543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1079629Z layer_outputs = layer_module( 2025-08-26T20:37:15.1079948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1080058Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1080374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1080499Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1080822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1080914Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1080920Z 2025-08-26T20:37:15.1081030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1081246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1081317Z return mod(**inputs) 2025-08-26T20:37:15.1081622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1081699Z outputs = self.mobilebert( 2025-08-26T20:37:15.1082001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1082087Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1082386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1082471Z layer_outputs = layer_module( 2025-08-26T20:37:15.1082767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1082878Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1083173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1083289Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1083597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1083717Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1083720Z 2025-08-26T20:37:15.1083835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1084077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1084150Z return mod(**inputs) 2025-08-26T20:37:15.1084457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1084533Z outputs = self.mobilebert( 2025-08-26T20:37:15.1084834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1084912Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1085232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1085312Z layer_outputs = layer_module( 2025-08-26T20:37:15.1085606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1085736Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1086035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1086195Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1086491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1086584Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1086595Z 2025-08-26T20:37:15.1086706Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1086916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1086997Z return mod(**inputs) 2025-08-26T20:37:15.1087295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1087380Z outputs = self.mobilebert( 2025-08-26T20:37:15.1087680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1087762Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1088071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1088147Z layer_outputs = layer_module( 2025-08-26T20:37:15.1088448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1088550Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1088843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1088984Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1089286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1089424Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1089721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1089828Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1089832Z 2025-08-26T20:37:15.1089938Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1090151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1090231Z return mod(**inputs) 2025-08-26T20:37:15.1090528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1090631Z outputs = self.mobilebert( 2025-08-26T20:37:15.1090930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1091008Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1091310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1091385Z layer_outputs = layer_module( 2025-08-26T20:37:15.1091686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1091786Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1092105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1092226Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1092580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1092678Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1092699Z 2025-08-26T20:37:15.1092809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1093026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1093099Z return mod(**inputs) 2025-08-26T20:37:15.1093398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1093481Z outputs = self.mobilebert( 2025-08-26T20:37:15.1093780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1093867Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1094165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1094251Z layer_outputs = layer_module( 2025-08-26T20:37:15.1094548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1094650Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1094957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1095075Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1095379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1095499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1095503Z 2025-08-26T20:37:15.1095626Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1095827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1095895Z return mod(**inputs) 2025-08-26T20:37:15.1096370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1096454Z outputs = self.mobilebert( 2025-08-26T20:37:15.1096739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1096814Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1097090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1097170Z layer_outputs = layer_module( 2025-08-26T20:37:15.1097445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1097600Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1097882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1098011Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1098297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1098383Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1098387Z 2025-08-26T20:37:15.1098497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1098719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1098796Z return mod(**inputs) 2025-08-26T20:37:15.1099077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1099173Z outputs = self.mobilebert( 2025-08-26T20:37:15.1099467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1099567Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1099855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1099926Z layer_outputs = layer_module( 2025-08-26T20:37:15.1100207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1100309Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1100586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1100718Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1101000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1101129Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1101414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1101507Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1101511Z 2025-08-26T20:37:15.1101621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1101822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1101897Z return mod(**inputs) 2025-08-26T20:37:15.1102176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1102248Z outputs = self.mobilebert( 2025-08-26T20:37:15.1102536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1102613Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1102916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1102991Z layer_outputs = layer_module( 2025-08-26T20:37:15.1103294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1103393Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1103691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1103820Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1104128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1104229Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1104234Z 2025-08-26T20:37:15.1104343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1104551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1104632Z return mod(**inputs) 2025-08-26T20:37:15.1104928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1105013Z outputs = self.mobilebert( 2025-08-26T20:37:15.1105330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1105416Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1105726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1105823Z layer_outputs = layer_module( 2025-08-26T20:37:15.1106109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1106221Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1106508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1106619Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1106899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1107017Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1107021Z 2025-08-26T20:37:15.1107125Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1107334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1107402Z return mod(**inputs) 2025-08-26T20:37:15.1107687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1107761Z outputs = self.mobilebert( 2025-08-26T20:37:15.1108038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1108120Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1108403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1108480Z layer_outputs = layer_module( 2025-08-26T20:37:15.1108760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1108860Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1109146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1109273Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1109559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1109644Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1109648Z 2025-08-26T20:37:15.1109759Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1109958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1110025Z return mod(**inputs) 2025-08-26T20:37:15.1110311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1110404Z outputs = self.mobilebert( 2025-08-26T20:37:15.1110692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1110767Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1111045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1111123Z layer_outputs = layer_module( 2025-08-26T20:37:15.1111413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1111535Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1111835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1111971Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1112287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1112416Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1112739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1112835Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1112838Z 2025-08-26T20:37:15.1112953Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1113166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1113239Z return mod(**inputs) 2025-08-26T20:37:15.1113551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1113630Z outputs = self.mobilebert( 2025-08-26T20:37:15.1113953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1114031Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1114334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1114407Z layer_outputs = layer_module( 2025-08-26T20:37:15.1114702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1114837Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1115136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1115238Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1115242Z 2025-08-26T20:37:15.1115358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1115584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1115657Z return mod(**inputs) 2025-08-26T20:37:15.1115964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1116049Z outputs = self.mobilebert( 2025-08-26T20:37:15.1116356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1116440Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1116756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1116833Z layer_outputs = layer_module( 2025-08-26T20:37:15.1117176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1117312Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1117634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1117760Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1117764Z 2025-08-26T20:37:15.1117883Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1118099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1118171Z return mod(**inputs) 2025-08-26T20:37:15.1118499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1118579Z outputs = self.mobilebert( 2025-08-26T20:37:15.1118892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1118989Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1119295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1119402Z layer_outputs = layer_module( 2025-08-26T20:37:15.1119815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1120005Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1120311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1120418Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1120430Z 2025-08-26T20:37:15.1120545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1120762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1120845Z return mod(**inputs) 2025-08-26T20:37:15.1121150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1121242Z outputs = self.mobilebert( 2025-08-26T20:37:15.1121551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1121630Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1121939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1122017Z layer_outputs = layer_module( 2025-08-26T20:37:15.1122324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1122495Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1122789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1122932Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1123228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1123337Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1123341Z 2025-08-26T20:37:15.1123451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1123669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1123740Z return mod(**inputs) 2025-08-26T20:37:15.1124061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1124151Z outputs = self.mobilebert( 2025-08-26T20:37:15.1124458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1124544Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1124848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1124925Z layer_outputs = layer_module( 2025-08-26T20:37:15.1125232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1125416Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1125723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1125875Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1126179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1126293Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1126297Z 2025-08-26T20:37:15.1126408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1126624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1126695Z return mod(**inputs) 2025-08-26T20:37:15.1127000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1127077Z outputs = self.mobilebert( 2025-08-26T20:37:15.1127377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1127457Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1127755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1127841Z layer_outputs = layer_module( 2025-08-26T20:37:15.1128134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1128304Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1128601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1128732Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1129039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1129174Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1129456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1129548Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1129551Z 2025-08-26T20:37:15.1129660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1129851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1129917Z return mod(**inputs) 2025-08-26T20:37:15.1130196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1130267Z outputs = self.mobilebert( 2025-08-26T20:37:15.1130545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1130617Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1130905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1130984Z layer_outputs = layer_module( 2025-08-26T20:37:15.1131259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1131422Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1131694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1131831Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1132103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1132183Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1132201Z 2025-08-26T20:37:15.1132311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1132504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1132596Z return mod(**inputs) 2025-08-26T20:37:15.1132867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1132939Z outputs = self.mobilebert( 2025-08-26T20:37:15.1133219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1133291Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1133572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1133644Z layer_outputs = layer_module( 2025-08-26T20:37:15.1133929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1134016Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1134298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1134379Z self_outputs = self.self( 2025-08-26T20:37:15.1134658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1134737Z self.value(value_tensor) 2025-08-26T20:37:15.1134741Z 2025-08-26T20:37:15.1134850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1135060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1135137Z return mod(**inputs) 2025-08-26T20:37:15.1135434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1135518Z outputs = self.mobilebert( 2025-08-26T20:37:15.1135815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1135901Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1136194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1136269Z layer_outputs = layer_module( 2025-08-26T20:37:15.1136574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1136753Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1137040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1137173Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1137452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1137547Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1137551Z 2025-08-26T20:37:15.1137654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1137858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1137925Z return mod(**inputs) 2025-08-26T20:37:15.1138230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1138304Z outputs = self.mobilebert( 2025-08-26T20:37:15.1138584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1138686Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1138968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1139065Z layer_outputs = layer_module( 2025-08-26T20:37:15.1139352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1139513Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1139819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1139928Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1140211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1140299Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1140586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1140679Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1140682Z 2025-08-26T20:37:15.1140783Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1140990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1141056Z return mod(**inputs) 2025-08-26T20:37:15.1141341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1141411Z outputs = self.mobilebert( 2025-08-26T20:37:15.1141689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1141767Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1142049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1142126Z layer_outputs = layer_module( 2025-08-26T20:37:15.1142408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1142498Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1142776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1142846Z self_outputs = self.self( 2025-08-26T20:37:15.1143141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1143213Z self.query(query_tensor) 2025-08-26T20:37:15.1143216Z 2025-08-26T20:37:15.1143328Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1143551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1143623Z return mod(**inputs) 2025-08-26T20:37:15.1143928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1144004Z outputs = self.mobilebert( 2025-08-26T20:37:15.1144308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1144386Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1144707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1144787Z layer_outputs = layer_module( 2025-08-26T20:37:15.1145086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1145207Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1145513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1145610Z self_outputs = self.self( 2025-08-26T20:37:15.1145890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1145959Z self.key(key_tensor) 2025-08-26T20:37:15.1145962Z 2025-08-26T20:37:15.1146055Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1146139Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1146261Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1146473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1146542Z return mod(**inputs) 2025-08-26T20:37:15.1146846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1146923Z outputs = self.mobilebert( 2025-08-26T20:37:15.1147228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1147305Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1147610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1147684Z layer_outputs = layer_module( 2025-08-26T20:37:15.1147985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1148080Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1148376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1148517Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1148815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1148909Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1148913Z 2025-08-26T20:37:15.1149028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1149238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1149315Z return mod(**inputs) 2025-08-26T20:37:15.1149614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1149699Z outputs = self.mobilebert( 2025-08-26T20:37:15.1149998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1150095Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1150402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1150478Z layer_outputs = layer_module( 2025-08-26T20:37:15.1150781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1150869Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1151166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1151322Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1151620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1151758Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1152077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1152185Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1152204Z 2025-08-26T20:37:15.1152318Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1152528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1152607Z return mod(**inputs) 2025-08-26T20:37:15.1152907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1152993Z outputs = self.mobilebert( 2025-08-26T20:37:15.1153291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1153369Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1153675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1153752Z layer_outputs = layer_module( 2025-08-26T20:37:15.1154062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1154166Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1154468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1154588Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1154889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1154987Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1154991Z 2025-08-26T20:37:15.1155100Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1155321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1155392Z return mod(**inputs) 2025-08-26T20:37:15.1155696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1155782Z outputs = self.mobilebert( 2025-08-26T20:37:15.1156078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1156164Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1156464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1156546Z layer_outputs = layer_module( 2025-08-26T20:37:15.1156869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1156975Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1157278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1157400Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1157707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1157828Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1157832Z 2025-08-26T20:37:15.1157943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1158178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1158250Z return mod(**inputs) 2025-08-26T20:37:15.1158557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1158651Z outputs = self.mobilebert( 2025-08-26T20:37:15.1158957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1159056Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1159355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1159519Z layer_outputs = layer_module( 2025-08-26T20:37:15.1159831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1159947Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1160251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1160391Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1160711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1160803Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1160807Z 2025-08-26T20:37:15.1160925Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1161142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1161217Z return mod(**inputs) 2025-08-26T20:37:15.1161498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1161573Z outputs = self.mobilebert( 2025-08-26T20:37:15.1161860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1161936Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1162222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1162293Z layer_outputs = layer_module( 2025-08-26T20:37:15.1162575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1162681Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1162962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1163096Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1163389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1163526Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1163853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1163957Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1163963Z 2025-08-26T20:37:15.1164079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1164290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1164369Z return mod(**inputs) 2025-08-26T20:37:15.1164668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1164759Z outputs = self.mobilebert( 2025-08-26T20:37:15.1165063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1165150Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1165455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1165526Z layer_outputs = layer_module( 2025-08-26T20:37:15.1165827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1165921Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1166202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1166319Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1166600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1166688Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1166691Z 2025-08-26T20:37:15.1166795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1166992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1167066Z return mod(**inputs) 2025-08-26T20:37:15.1167346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1167425Z outputs = self.mobilebert( 2025-08-26T20:37:15.1167704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1167782Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1168067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1168140Z layer_outputs = layer_module( 2025-08-26T20:37:15.1168431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1168525Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1168813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1168925Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1169205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1169324Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1169328Z 2025-08-26T20:37:15.1169431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1169637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1169703Z return mod(**inputs) 2025-08-26T20:37:15.1170006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1170082Z outputs = self.mobilebert( 2025-08-26T20:37:15.1170378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1170468Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1170761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1170846Z layer_outputs = layer_module( 2025-08-26T20:37:15.1171141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1171260Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1171565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1171699Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1172018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1172132Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1172136Z 2025-08-26T20:37:15.1172253Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1172465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1172537Z return mod(**inputs) 2025-08-26T20:37:15.1172840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1172918Z outputs = self.mobilebert( 2025-08-26T20:37:15.1173223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1173302Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1173599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1173681Z layer_outputs = layer_module( 2025-08-26T20:37:15.1173979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1174085Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1174380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1174520Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1174817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1174947Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1175256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1175353Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1175359Z 2025-08-26T20:37:15.1175476Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1175685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1175756Z return mod(**inputs) 2025-08-26T20:37:15.1176058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1176135Z outputs = self.mobilebert( 2025-08-26T20:37:15.1176440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1176519Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1176838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1176917Z layer_outputs = layer_module( 2025-08-26T20:37:15.1177212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1177324Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1177625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1177749Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1178067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1178160Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1178171Z 2025-08-26T20:37:15.1178281Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1178547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1178626Z return mod(**inputs) 2025-08-26T20:37:15.1178958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1179042Z outputs = self.mobilebert( 2025-08-26T20:37:15.1179338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1179418Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1179721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1179797Z layer_outputs = layer_module( 2025-08-26T20:37:15.1180098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1180194Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1180471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1180590Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1180871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1180991Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1180994Z 2025-08-26T20:37:15.1181097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1181305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1181371Z return mod(**inputs) 2025-08-26T20:37:15.1181652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1181733Z outputs = self.mobilebert( 2025-08-26T20:37:15.1182011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1182091Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1182368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1182440Z layer_outputs = layer_module( 2025-08-26T20:37:15.1182742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1182843Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1183143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1183289Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1183602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1183694Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1183698Z 2025-08-26T20:37:15.1183807Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1184027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1184093Z return mod(**inputs) 2025-08-26T20:37:15.1184378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1184467Z outputs = self.mobilebert( 2025-08-26T20:37:15.1184752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1184834Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1185135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1185214Z layer_outputs = layer_module( 2025-08-26T20:37:15.1185513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1185614Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1185897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1186022Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1186309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1186430Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1186718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1186812Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1186817Z 2025-08-26T20:37:15.1186922Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1187139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1187210Z return mod(**inputs) 2025-08-26T20:37:15.1187513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1187603Z outputs = self.mobilebert( 2025-08-26T20:37:15.1187892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1187966Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1188252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1188332Z layer_outputs = layer_module( 2025-08-26T20:37:15.1188614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1188746Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1189027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1189112Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1189122Z 2025-08-26T20:37:15.1189227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1189422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1189497Z return mod(**inputs) 2025-08-26T20:37:15.1189796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1189881Z outputs = self.mobilebert( 2025-08-26T20:37:15.1190161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1190236Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1190526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1190602Z layer_outputs = layer_module( 2025-08-26T20:37:15.1190908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1191032Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1191313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1191451Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1191455Z 2025-08-26T20:37:15.1191558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1191783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1191851Z return mod(**inputs) 2025-08-26T20:37:15.1192136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1192208Z outputs = self.mobilebert( 2025-08-26T20:37:15.1192487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1192570Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1192848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1192929Z layer_outputs = layer_module( 2025-08-26T20:37:15.1193209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1193372Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1193676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1193778Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1193782Z 2025-08-26T20:37:15.1193898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1194116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1194195Z return mod(**inputs) 2025-08-26T20:37:15.1194503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1194581Z outputs = self.mobilebert( 2025-08-26T20:37:15.1194893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1194975Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1195295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1195370Z layer_outputs = layer_module( 2025-08-26T20:37:15.1195667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1195846Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1196146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1196574Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1196875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1196988Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1196992Z 2025-08-26T20:37:15.1197103Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1197314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1197395Z return mod(**inputs) 2025-08-26T20:37:15.1197753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1197840Z outputs = self.mobilebert( 2025-08-26T20:37:15.1198139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1198222Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1198549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1198623Z layer_outputs = layer_module( 2025-08-26T20:37:15.1198952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1199118Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1199423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1199613Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1199918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1200016Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1200023Z 2025-08-26T20:37:15.1200134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1200355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1200430Z return mod(**inputs) 2025-08-26T20:37:15.1200734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1200817Z outputs = self.mobilebert( 2025-08-26T20:37:15.1201122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1201210Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1201526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1201610Z layer_outputs = layer_module( 2025-08-26T20:37:15.1201922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1202093Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1202401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1202531Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1202834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1202963Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1203267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1203365Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1203370Z 2025-08-26T20:37:15.1203503Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1203722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1203794Z return mod(**inputs) 2025-08-26T20:37:15.1204097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1204173Z outputs = self.mobilebert( 2025-08-26T20:37:15.1204468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1204553Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1204872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1204961Z layer_outputs = layer_module( 2025-08-26T20:37:15.1205262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1205473Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1205794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1205913Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1206216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1206305Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1206309Z 2025-08-26T20:37:15.1206428Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1206638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1206708Z return mod(**inputs) 2025-08-26T20:37:15.1207017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1207093Z outputs = self.mobilebert( 2025-08-26T20:37:15.1207399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1207475Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1207780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1207857Z layer_outputs = layer_module( 2025-08-26T20:37:15.1208157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1208257Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1208561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1208644Z self_outputs = self.self( 2025-08-26T20:37:15.1208943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1209021Z self.value(value_tensor) 2025-08-26T20:37:15.1209025Z 2025-08-26T20:37:15.1209142Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1209353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1209429Z return mod(**inputs) 2025-08-26T20:37:15.1209729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1209806Z outputs = self.mobilebert( 2025-08-26T20:37:15.1210110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1210209Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1210515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1210594Z layer_outputs = layer_module( 2025-08-26T20:37:15.1210897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1211075Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1211359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1211497Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1211776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1211870Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1211890Z 2025-08-26T20:37:15.1211995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1212197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1212284Z return mod(**inputs) 2025-08-26T20:37:15.1212563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1212644Z outputs = self.mobilebert( 2025-08-26T20:37:15.1212923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1213003Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1213283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1213353Z layer_outputs = layer_module( 2025-08-26T20:37:15.1213647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1213806Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1214093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1214203Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1214485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1214574Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1214852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1214951Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1214957Z 2025-08-26T20:37:15.1215063Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1215263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1215330Z return mod(**inputs) 2025-08-26T20:37:15.1215608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1215688Z outputs = self.mobilebert( 2025-08-26T20:37:15.1215962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1216041Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1216322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1216400Z layer_outputs = layer_module( 2025-08-26T20:37:15.1216694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1216783Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1217074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1217146Z self_outputs = self.self( 2025-08-26T20:37:15.1217427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1217499Z self.query(query_tensor) 2025-08-26T20:37:15.1217503Z 2025-08-26T20:37:15.1217607Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1217827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1217896Z return mod(**inputs) 2025-08-26T20:37:15.1218186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1218273Z outputs = self.mobilebert( 2025-08-26T20:37:15.1218551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1218648Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1218934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1219017Z layer_outputs = layer_module( 2025-08-26T20:37:15.1219311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1219409Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1219706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1219782Z self_outputs = self.self( 2025-08-26T20:37:15.1220089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1220161Z self.key(key_tensor) 2025-08-26T20:37:15.1220166Z 2025-08-26T20:37:15.1220262Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1220349Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1220461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1220680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1220750Z return mod(**inputs) 2025-08-26T20:37:15.1221050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1221121Z outputs = self.mobilebert( 2025-08-26T20:37:15.1221398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1221484Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1221759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1221842Z layer_outputs = layer_module( 2025-08-26T20:37:15.1222123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1222215Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1222497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1222622Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1222913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1223017Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1223023Z 2025-08-26T20:37:15.1223136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1223336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1223403Z return mod(**inputs) 2025-08-26T20:37:15.1223690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1223762Z outputs = self.mobilebert( 2025-08-26T20:37:15.1224050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1224142Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1224428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1224501Z layer_outputs = layer_module( 2025-08-26T20:37:15.1224804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1224897Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1225196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1225329Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1225609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1225733Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1226023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1226116Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1226119Z 2025-08-26T20:37:15.1226233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1226431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1226507Z return mod(**inputs) 2025-08-26T20:37:15.1226785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1226858Z outputs = self.mobilebert( 2025-08-26T20:37:15.1227160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1227236Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1227542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1227617Z layer_outputs = layer_module( 2025-08-26T20:37:15.1227917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1228028Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1228327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1228450Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1228784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1228882Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1228886Z 2025-08-26T20:37:15.1228996Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1229208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1229286Z return mod(**inputs) 2025-08-26T20:37:15.1229599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1229688Z outputs = self.mobilebert( 2025-08-26T20:37:15.1229983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1230063Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1230373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1230448Z layer_outputs = layer_module( 2025-08-26T20:37:15.1230776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1230881Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1231184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1231323Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1231620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1231766Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1231770Z 2025-08-26T20:37:15.1231878Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1232091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1232161Z return mod(**inputs) 2025-08-26T20:37:15.1232458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1232540Z outputs = self.mobilebert( 2025-08-26T20:37:15.1232836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1232923Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1233217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1233300Z layer_outputs = layer_module( 2025-08-26T20:37:15.1233595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1233695Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1233997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1234132Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1234434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1234527Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1234535Z 2025-08-26T20:37:15.1234653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1234870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1234943Z return mod(**inputs) 2025-08-26T20:37:15.1235255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1235332Z outputs = self.mobilebert( 2025-08-26T20:37:15.1235643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1235723Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1236026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1236114Z layer_outputs = layer_module( 2025-08-26T20:37:15.1236444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1236559Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1236865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1237002Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1237317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1237450Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1237823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1237929Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1237933Z 2025-08-26T20:37:15.1238084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1238300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1238373Z return mod(**inputs) 2025-08-26T20:37:15.1238710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1238789Z outputs = self.mobilebert( 2025-08-26T20:37:15.1239101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1239179Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1239570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1239666Z layer_outputs = layer_module( 2025-08-26T20:37:15.1239973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1240086Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1240392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1240526Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1240831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1240924Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1240928Z 2025-08-26T20:37:15.1241053Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1241268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1241350Z return mod(**inputs) 2025-08-26T20:37:15.1241655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1241737Z outputs = self.mobilebert( 2025-08-26T20:37:15.1242058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1242139Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1242439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1242517Z layer_outputs = layer_module( 2025-08-26T20:37:15.1242824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1242924Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1243221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1243366Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1243650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1243776Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1243780Z 2025-08-26T20:37:15.1243894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1244118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1244193Z return mod(**inputs) 2025-08-26T20:37:15.1244520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1244610Z outputs = self.mobilebert( 2025-08-26T20:37:15.1244912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1245020Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1245326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1245422Z layer_outputs = layer_module( 2025-08-26T20:37:15.1245738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1245842Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1246161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1246296Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1246603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1246696Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1246703Z 2025-08-26T20:37:15.1246806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1247011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1247079Z return mod(**inputs) 2025-08-26T20:37:15.1247368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1247440Z outputs = self.mobilebert( 2025-08-26T20:37:15.1247724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1247805Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1248104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1248184Z layer_outputs = layer_module( 2025-08-26T20:37:15.1248484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1248584Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1248891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1249023Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1249333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1249460Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1249769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1249868Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1249872Z 2025-08-26T20:37:15.1249999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1250218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1250289Z return mod(**inputs) 2025-08-26T20:37:15.1250591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1250667Z outputs = self.mobilebert( 2025-08-26T20:37:15.1250970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1251047Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1251364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1251452Z layer_outputs = layer_module( 2025-08-26T20:37:15.1251747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1251874Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1252168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1252306Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1252609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1252699Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1252703Z 2025-08-26T20:37:15.1252822Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1253032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1253111Z return mod(**inputs) 2025-08-26T20:37:15.1253415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1253494Z outputs = self.mobilebert( 2025-08-26T20:37:15.1253807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1253888Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1254198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1254273Z layer_outputs = layer_module( 2025-08-26T20:37:15.1254576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1254685Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1254989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1255109Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1255396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1255518Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1255522Z 2025-08-26T20:37:15.1255628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1255827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1255903Z return mod(**inputs) 2025-08-26T20:37:15.1256201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1256287Z outputs = self.mobilebert( 2025-08-26T20:37:15.1256588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1256695Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1257008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1257085Z layer_outputs = layer_module( 2025-08-26T20:37:15.1257386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1257485Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1257781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1257937Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1258234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1258332Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1258353Z 2025-08-26T20:37:15.1258469Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1258685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1258774Z return mod(**inputs) 2025-08-26T20:37:15.1259076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1259161Z outputs = self.mobilebert( 2025-08-26T20:37:15.1259461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1259546Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1259852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1259927Z layer_outputs = layer_module( 2025-08-26T20:37:15.1260239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1260341Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1260654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1260789Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1261101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1261231Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1261538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1261645Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1261649Z 2025-08-26T20:37:15.1261761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1261983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1262054Z return mod(**inputs) 2025-08-26T20:37:15.1262365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1262443Z outputs = self.mobilebert( 2025-08-26T20:37:15.1262745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1262828Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1263136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1263221Z layer_outputs = layer_module( 2025-08-26T20:37:15.1263541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1263675Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1263978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1264070Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1264074Z 2025-08-26T20:37:15.1264189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1264401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1264479Z return mod(**inputs) 2025-08-26T20:37:15.1264793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1264873Z outputs = self.mobilebert( 2025-08-26T20:37:15.1265177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1265272Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1265573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1265672Z layer_outputs = layer_module( 2025-08-26T20:37:15.1265970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1266105Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1266404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1266528Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1266532Z 2025-08-26T20:37:15.1266640Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1266855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1266926Z return mod(**inputs) 2025-08-26T20:37:15.1267221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1267309Z outputs = self.mobilebert( 2025-08-26T20:37:15.1267605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1267689Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1267984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1268060Z layer_outputs = layer_module( 2025-08-26T20:37:15.1268367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1268537Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1268847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1268950Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1268954Z 2025-08-26T20:37:15.1269070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1269281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1269350Z return mod(**inputs) 2025-08-26T20:37:15.1269654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1269732Z outputs = self.mobilebert( 2025-08-26T20:37:15.1270037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1270115Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1270431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1270519Z layer_outputs = layer_module( 2025-08-26T20:37:15.1270820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1270992Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1271296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1271451Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1271751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1271850Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1271872Z 2025-08-26T20:37:15.1271990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1272198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1272295Z return mod(**inputs) 2025-08-26T20:37:15.1272591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1272668Z outputs = self.mobilebert( 2025-08-26T20:37:15.1272973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1273052Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1273355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1273430Z layer_outputs = layer_module( 2025-08-26T20:37:15.1273733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1273900Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1274202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1274342Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1274637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1274736Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1274742Z 2025-08-26T20:37:15.1274851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1275057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1275136Z return mod(**inputs) 2025-08-26T20:37:15.1275433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1275516Z outputs = self.mobilebert( 2025-08-26T20:37:15.1275815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1275899Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1276192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1276266Z layer_outputs = layer_module( 2025-08-26T20:37:15.1276571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1276737Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1277054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1277189Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1277484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1277621Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1277917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1278024Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1278028Z 2025-08-26T20:37:15.1278157Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1278381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1278453Z return mod(**inputs) 2025-08-26T20:37:15.1278780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1278869Z outputs = self.mobilebert( 2025-08-26T20:37:15.1279199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1279285Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1279668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1279754Z layer_outputs = layer_module( 2025-08-26T20:37:15.1280076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1280255Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1280580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1280715Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1281025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1281118Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1281122Z 2025-08-26T20:37:15.1281232Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1281452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1281525Z return mod(**inputs) 2025-08-26T20:37:15.1281834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1281911Z outputs = self.mobilebert( 2025-08-26T20:37:15.1282219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1282302Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1282585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1282667Z layer_outputs = layer_module( 2025-08-26T20:37:15.1282951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1283047Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1283333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1283407Z self_outputs = self.self( 2025-08-26T20:37:15.1283699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1283795Z self.value(value_tensor) 2025-08-26T20:37:15.1283801Z 2025-08-26T20:37:15.1283913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1284112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1284180Z return mod(**inputs) 2025-08-26T20:37:15.1284467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1284541Z outputs = self.mobilebert( 2025-08-26T20:37:15.1284827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1284917Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1285204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1285275Z layer_outputs = layer_module( 2025-08-26T20:37:15.1285573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1285748Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1286067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1286194Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1286496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1286586Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1286598Z 2025-08-26T20:37:15.1286707Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1286919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1286996Z return mod(**inputs) 2025-08-26T20:37:15.1287312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1287397Z outputs = self.mobilebert( 2025-08-26T20:37:15.1287723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1287801Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1288112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1288187Z layer_outputs = layer_module( 2025-08-26T20:37:15.1288496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1288666Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1288974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1289107Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1289394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1289490Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1289781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1289883Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1289887Z 2025-08-26T20:37:15.1289992Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1290194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1290271Z return mod(**inputs) 2025-08-26T20:37:15.1290575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1290658Z outputs = self.mobilebert( 2025-08-26T20:37:15.1290946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1291024Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1291328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1291405Z layer_outputs = layer_module( 2025-08-26T20:37:15.1291725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1291819Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1292124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1292220Z self_outputs = self.self( 2025-08-26T20:37:15.1292513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1292616Z self.query(query_tensor) 2025-08-26T20:37:15.1292620Z 2025-08-26T20:37:15.1292729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1292944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1293010Z return mod(**inputs) 2025-08-26T20:37:15.1293290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1293369Z outputs = self.mobilebert( 2025-08-26T20:37:15.1293648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1293730Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1294018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1294097Z layer_outputs = layer_module( 2025-08-26T20:37:15.1294376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1294463Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1294765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1294839Z self_outputs = self.self( 2025-08-26T20:37:15.1295138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1295210Z self.key(key_tensor) 2025-08-26T20:37:15.1295214Z 2025-08-26T20:37:15.1295305Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1295402Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1295513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1295730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1295802Z return mod(**inputs) 2025-08-26T20:37:15.1296100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1296374Z outputs = self.mobilebert( 2025-08-26T20:37:15.1296679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1296770Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1297064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1297148Z layer_outputs = layer_module( 2025-08-26T20:37:15.1297491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1297584Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1297898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1298034Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1298356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1298480Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1298485Z 2025-08-26T20:37:15.1298610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1298824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1298894Z return mod(**inputs) 2025-08-26T20:37:15.1299227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1299305Z outputs = self.mobilebert( 2025-08-26T20:37:15.1299641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1299718Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1300016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1300103Z layer_outputs = layer_module( 2025-08-26T20:37:15.1300398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1300493Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1300788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1300918Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1301219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1301352Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1301650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1301748Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1301752Z 2025-08-26T20:37:15.1301866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1302073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1302143Z return mod(**inputs) 2025-08-26T20:37:15.1302451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1302528Z outputs = self.mobilebert( 2025-08-26T20:37:15.1302831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1302909Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1303202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1303285Z layer_outputs = layer_module( 2025-08-26T20:37:15.1303580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1303688Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1303981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1304160Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1304458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1304551Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1304555Z 2025-08-26T20:37:15.1304671Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1304880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1304958Z return mod(**inputs) 2025-08-26T20:37:15.1305277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1305356Z outputs = self.mobilebert( 2025-08-26T20:37:15.1305659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1305765Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1306073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1306166Z layer_outputs = layer_module( 2025-08-26T20:37:15.1306472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1306573Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1306871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1307002Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1307302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1307432Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1307438Z 2025-08-26T20:37:15.1307551Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1307767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1307849Z return mod(**inputs) 2025-08-26T20:37:15.1308160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1308242Z outputs = self.mobilebert( 2025-08-26T20:37:15.1308541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1308626Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1308922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1308997Z layer_outputs = layer_module( 2025-08-26T20:37:15.1309303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1309404Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1309706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1309837Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1310135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1310234Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1310239Z 2025-08-26T20:37:15.1310348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1310562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1310631Z return mod(**inputs) 2025-08-26T20:37:15.1310953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1311031Z outputs = self.mobilebert( 2025-08-26T20:37:15.1311336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1311421Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1311718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1311798Z layer_outputs = layer_module( 2025-08-26T20:37:15.1312111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1312214Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1312519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1312663Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1312951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1313090Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1313378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1313471Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1313475Z 2025-08-26T20:37:15.1313580Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1313783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1313850Z return mod(**inputs) 2025-08-26T20:37:15.1314133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1314207Z outputs = self.mobilebert( 2025-08-26T20:37:15.1314486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1314569Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1314848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1314926Z layer_outputs = layer_module( 2025-08-26T20:37:15.1315207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1315308Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1315596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1315717Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1316020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1316112Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1316115Z 2025-08-26T20:37:15.1316230Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1316436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1316504Z return mod(**inputs) 2025-08-26T20:37:15.1316805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1316881Z outputs = self.mobilebert( 2025-08-26T20:37:15.1317185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1317280Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1317583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1317661Z layer_outputs = layer_module( 2025-08-26T20:37:15.1317952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1318062Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1318372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1318522Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1318832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1318950Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1318979Z 2025-08-26T20:37:15.1319090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1319296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1319395Z return mod(**inputs) 2025-08-26T20:37:15.1319752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1319844Z outputs = self.mobilebert( 2025-08-26T20:37:15.1320150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1320233Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1320549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1320627Z layer_outputs = layer_module( 2025-08-26T20:37:15.1320942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1321045Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1321361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1321504Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1321802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1321911Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1321917Z 2025-08-26T20:37:15.1322021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1322226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1322294Z return mod(**inputs) 2025-08-26T20:37:15.1322576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1322662Z outputs = self.mobilebert( 2025-08-26T20:37:15.1322947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1323028Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1323312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1323385Z layer_outputs = layer_module( 2025-08-26T20:37:15.1323680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1323776Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1324084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1324211Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1324496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1324621Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1324901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1325000Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1325003Z 2025-08-26T20:37:15.1325124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1325334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1325402Z return mod(**inputs) 2025-08-26T20:37:15.1325699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1325800Z outputs = self.mobilebert( 2025-08-26T20:37:15.1326097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1326200Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1326506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1326589Z layer_outputs = layer_module( 2025-08-26T20:37:15.1326888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1326987Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1327294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1327416Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1327719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1327810Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1327814Z 2025-08-26T20:37:15.1327922Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1328136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1328205Z return mod(**inputs) 2025-08-26T20:37:15.1328507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1328584Z outputs = self.mobilebert( 2025-08-26T20:37:15.1328889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1328969Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1329264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1329349Z layer_outputs = layer_module( 2025-08-26T20:37:15.1329646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1329754Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1330048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1330166Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1330466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1330585Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1330609Z 2025-08-26T20:37:15.1330728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1330937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1331016Z return mod(**inputs) 2025-08-26T20:37:15.1331313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1331390Z outputs = self.mobilebert( 2025-08-26T20:37:15.1331693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1331795Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1332101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1332176Z layer_outputs = layer_module( 2025-08-26T20:37:15.1332474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1332599Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1332915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1333055Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1333360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1333456Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1333459Z 2025-08-26T20:37:15.1333572Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1333783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1333861Z return mod(**inputs) 2025-08-26T20:37:15.1334166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1334248Z outputs = self.mobilebert( 2025-08-26T20:37:15.1334554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1334631Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1334940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1335014Z layer_outputs = layer_module( 2025-08-26T20:37:15.1335324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1335424Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1335733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1335867Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1336170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1336310Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1336614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1336720Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1336724Z 2025-08-26T20:37:15.1336834Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1337046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1337124Z return mod(**inputs) 2025-08-26T20:37:15.1337485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1337576Z outputs = self.mobilebert( 2025-08-26T20:37:15.1337878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1337967Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1338269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1338348Z layer_outputs = layer_module( 2025-08-26T20:37:15.1338675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1338810Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1339118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1339234Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1339238Z 2025-08-26T20:37:15.1339350Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1339580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1339670Z return mod(**inputs) 2025-08-26T20:37:15.1339975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1340052Z outputs = self.mobilebert( 2025-08-26T20:37:15.1340356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1340434Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1340728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1340815Z layer_outputs = layer_module( 2025-08-26T20:37:15.1341112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1341248Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1341551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1341670Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1341682Z 2025-08-26T20:37:15.1341790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1342001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1342078Z return mod(**inputs) 2025-08-26T20:37:15.1342373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1342460Z outputs = self.mobilebert( 2025-08-26T20:37:15.1342753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1342831Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1343135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1343211Z layer_outputs = layer_module( 2025-08-26T20:37:15.1343515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1343687Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1343979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1344088Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1344093Z 2025-08-26T20:37:15.1344218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1344439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1344511Z return mod(**inputs) 2025-08-26T20:37:15.1344816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1344894Z outputs = self.mobilebert( 2025-08-26T20:37:15.1345192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1345279Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1345595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1345681Z layer_outputs = layer_module( 2025-08-26T20:37:15.1345979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1346166Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1346488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1346619Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1346925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1347025Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1347030Z 2025-08-26T20:37:15.1347144Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1347353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1347423Z return mod(**inputs) 2025-08-26T20:37:15.1347735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1347811Z outputs = self.mobilebert( 2025-08-26T20:37:15.1348119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1348195Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1348507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1348592Z layer_outputs = layer_module( 2025-08-26T20:37:15.1348907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1349086Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1349407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1349555Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1349872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1349966Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1349970Z 2025-08-26T20:37:15.1350089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1350319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1350396Z return mod(**inputs) 2025-08-26T20:37:15.1350696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1350779Z outputs = self.mobilebert( 2025-08-26T20:37:15.1351116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1351198Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1351519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1351599Z layer_outputs = layer_module( 2025-08-26T20:37:15.1351922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1352092Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1352424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1352569Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1352896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1353058Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1353374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1353499Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1353503Z 2025-08-26T20:37:15.1353615Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1353829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1353910Z return mod(**inputs) 2025-08-26T20:37:15.1354216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1354302Z outputs = self.mobilebert( 2025-08-26T20:37:15.1354620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1354700Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1355025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1355105Z layer_outputs = layer_module( 2025-08-26T20:37:15.1355425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1355601Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1355928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1356050Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1356357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1356457Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1356461Z 2025-08-26T20:37:15.1356573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1356797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1356867Z return mod(**inputs) 2025-08-26T20:37:15.1357177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1357262Z outputs = self.mobilebert( 2025-08-26T20:37:15.1357572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1357658Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1357963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1358077Z layer_outputs = layer_module( 2025-08-26T20:37:15.1358385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1358482Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1358796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1358875Z self_outputs = self.self( 2025-08-26T20:37:15.1359189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1359286Z self.value(value_tensor) 2025-08-26T20:37:15.1359291Z 2025-08-26T20:37:15.1359408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1359706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1359784Z return mod(**inputs) 2025-08-26T20:37:15.1360128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1360208Z outputs = self.mobilebert( 2025-08-26T20:37:15.1360546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1360628Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1360933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1361022Z layer_outputs = layer_module( 2025-08-26T20:37:15.1361329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1361514Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1361825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1361950Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1362268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1362360Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1362364Z 2025-08-26T20:37:15.1362489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1362705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1362785Z return mod(**inputs) 2025-08-26T20:37:15.1363094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1363172Z outputs = self.mobilebert( 2025-08-26T20:37:15.1363488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1363580Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1363887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1366720Z layer_outputs = layer_module( 2025-08-26T20:37:15.1368248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1368447Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1368756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1368876Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1369187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1369283Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1369589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1369691Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1369733Z 2025-08-26T20:37:15.1369848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1370078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1370152Z return mod(**inputs) 2025-08-26T20:37:15.1370449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1370536Z outputs = self.mobilebert( 2025-08-26T20:37:15.1370836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1370948Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1371244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1371346Z layer_outputs = layer_module( 2025-08-26T20:37:15.1371650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1371746Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1372050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1372128Z self_outputs = self.self( 2025-08-26T20:37:15.1372433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1372510Z self.query(query_tensor) 2025-08-26T20:37:15.1372515Z 2025-08-26T20:37:15.1372624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1372841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1372915Z return mod(**inputs) 2025-08-26T20:37:15.1373216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1373293Z outputs = self.mobilebert( 2025-08-26T20:37:15.1373592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1373678Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1373971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1374054Z layer_outputs = layer_module( 2025-08-26T20:37:15.1374350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1374445Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1374748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1374896Z self_outputs = self.self( 2025-08-26T20:37:15.1375220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1375293Z self.key(key_tensor) 2025-08-26T20:37:15.1375296Z 2025-08-26T20:37:15.1375391Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1375477Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1375588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1375802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1375875Z return mod(**inputs) 2025-08-26T20:37:15.1376178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1376257Z outputs = self.mobilebert( 2025-08-26T20:37:15.1376552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1376636Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1376933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1377018Z layer_outputs = layer_module( 2025-08-26T20:37:15.1377315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1377405Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1377708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1377859Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1378194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1378283Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1378286Z 2025-08-26T20:37:15.1378398Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1378596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1378664Z return mod(**inputs) 2025-08-26T20:37:15.1378951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1379024Z outputs = self.mobilebert( 2025-08-26T20:37:15.1379317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1379397Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1379692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1379778Z layer_outputs = layer_module( 2025-08-26T20:37:15.1380073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1380171Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1380465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1380600Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1380894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1381029Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1381329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1381429Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1381468Z 2025-08-26T20:37:15.1381588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1381822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1381902Z return mod(**inputs) 2025-08-26T20:37:15.1382197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1382273Z outputs = self.mobilebert( 2025-08-26T20:37:15.1382573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1382652Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1382953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1383030Z layer_outputs = layer_module( 2025-08-26T20:37:15.1383333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1383446Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1383748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1383888Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1384180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1384297Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1384300Z 2025-08-26T20:37:15.1384410Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1384618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1384731Z return mod(**inputs) 2025-08-26T20:37:15.1385040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1385126Z outputs = self.mobilebert( 2025-08-26T20:37:15.1385434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1385512Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1385823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1385900Z layer_outputs = layer_module( 2025-08-26T20:37:15.1386215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1386316Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1386624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1386754Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1387060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1387187Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1387191Z 2025-08-26T20:37:15.1387301Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1387522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1387597Z return mod(**inputs) 2025-08-26T20:37:15.1387899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1387985Z outputs = self.mobilebert( 2025-08-26T20:37:15.1388313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1388401Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1388718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1388796Z layer_outputs = layer_module( 2025-08-26T20:37:15.1389104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1389204Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1389505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1389641Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1389942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1390036Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1390040Z 2025-08-26T20:37:15.1390149Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1390367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1390437Z return mod(**inputs) 2025-08-26T20:37:15.1390740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1390824Z outputs = self.mobilebert( 2025-08-26T20:37:15.1391099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1391202Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1391487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1391587Z layer_outputs = layer_module( 2025-08-26T20:37:15.1391869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1391973Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1392253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1392379Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1392679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1392811Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1393115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1393215Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1393219Z 2025-08-26T20:37:15.1393336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1393548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1393617Z return mod(**inputs) 2025-08-26T20:37:15.1393922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1393996Z outputs = self.mobilebert( 2025-08-26T20:37:15.1394297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1394375Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1394670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1394754Z layer_outputs = layer_module( 2025-08-26T20:37:15.1395070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1395198Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1395498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1395622Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1395919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1396009Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1396015Z 2025-08-26T20:37:15.1396133Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1396629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1396718Z return mod(**inputs) 2025-08-26T20:37:15.1397024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1397104Z outputs = self.mobilebert( 2025-08-26T20:37:15.1397420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1397503Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1397817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1397897Z layer_outputs = layer_module( 2025-08-26T20:37:15.1398249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1398360Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1398696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1398835Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1399149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1399278Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1399282Z 2025-08-26T20:37:15.1399394Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1399673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1399750Z return mod(**inputs) 2025-08-26T20:37:15.1400055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1400142Z outputs = self.mobilebert( 2025-08-26T20:37:15.1400449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1400537Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1400862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1400937Z layer_outputs = layer_module( 2025-08-26T20:37:15.1401247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1401345Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1401653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1401787Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1402097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1402218Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1402222Z 2025-08-26T20:37:15.1402336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1402587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1402661Z return mod(**inputs) 2025-08-26T20:37:15.1402965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1403042Z outputs = self.mobilebert( 2025-08-26T20:37:15.1403337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1403424Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1403717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1403804Z layer_outputs = layer_module( 2025-08-26T20:37:15.1404103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1404211Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1404507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1404636Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1404943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1405090Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1405396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1405516Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1405519Z 2025-08-26T20:37:15.1405636Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1405845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1405915Z return mod(**inputs) 2025-08-26T20:37:15.1406219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1406294Z outputs = self.mobilebert( 2025-08-26T20:37:15.1406594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1406672Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1406967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1407052Z layer_outputs = layer_module( 2025-08-26T20:37:15.1407348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1407453Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1407749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1407867Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1408171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1408259Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1408264Z 2025-08-26T20:37:15.1408379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1408588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1408666Z return mod(**inputs) 2025-08-26T20:37:15.1408978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1409056Z outputs = self.mobilebert( 2025-08-26T20:37:15.1409380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1409459Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1409759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1409833Z layer_outputs = layer_module( 2025-08-26T20:37:15.1410129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1410236Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1410531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1410656Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1410950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1411075Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1411079Z 2025-08-26T20:37:15.1411187Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1411393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1411472Z return mod(**inputs) 2025-08-26T20:37:15.1411794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1411878Z outputs = self.mobilebert( 2025-08-26T20:37:15.1412225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1412304Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1412607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1412683Z layer_outputs = layer_module( 2025-08-26T20:37:15.1412984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1413085Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1413390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1413521Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1413817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1413917Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1413921Z 2025-08-26T20:37:15.1414029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1414252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1414321Z return mod(**inputs) 2025-08-26T20:37:15.1414599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1414679Z outputs = self.mobilebert( 2025-08-26T20:37:15.1414968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1415055Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1415352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1415436Z layer_outputs = layer_module( 2025-08-26T20:37:15.1415747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1415866Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1416171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1416302Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1416605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1416735Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1417037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1417136Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1417141Z 2025-08-26T20:37:15.1417249Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1417466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1417536Z return mod(**inputs) 2025-08-26T20:37:15.1417837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1417912Z outputs = self.mobilebert( 2025-08-26T20:37:15.1418206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1418306Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1418586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1418684Z layer_outputs = layer_module( 2025-08-26T20:37:15.1418966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1419088Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1419379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1419464Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1419467Z 2025-08-26T20:37:15.1419579Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1419777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1419853Z return mod(**inputs) 2025-08-26T20:37:15.1420139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1420214Z outputs = self.mobilebert( 2025-08-26T20:37:15.1420506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1420580Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1420870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1420942Z layer_outputs = layer_module( 2025-08-26T20:37:15.1421218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1421348Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1421631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1421753Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1421758Z 2025-08-26T20:37:15.1421866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1422100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1422172Z return mod(**inputs) 2025-08-26T20:37:15.1422487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1422576Z outputs = self.mobilebert( 2025-08-26T20:37:15.1422871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1422956Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1423252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1423331Z layer_outputs = layer_module( 2025-08-26T20:37:15.1423634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1423811Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1424100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1424194Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1424198Z 2025-08-26T20:37:15.1424308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1424507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1424573Z return mod(**inputs) 2025-08-26T20:37:15.1424879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1424951Z outputs = self.mobilebert( 2025-08-26T20:37:15.1425253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1425327Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1425608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1425686Z layer_outputs = layer_module( 2025-08-26T20:37:15.1425967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1426134Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1426413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1426544Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1426826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1426923Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1426927Z 2025-08-26T20:37:15.1427039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1427239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1427314Z return mod(**inputs) 2025-08-26T20:37:15.1427615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1427700Z outputs = self.mobilebert( 2025-08-26T20:37:15.1427995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1428073Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1428381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1428460Z layer_outputs = layer_module( 2025-08-26T20:37:15.1428789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1428967Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1429246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1429381Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1429660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1429756Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1429760Z 2025-08-26T20:37:15.1429864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1430069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1430139Z return mod(**inputs) 2025-08-26T20:37:15.1430420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1430499Z outputs = self.mobilebert( 2025-08-26T20:37:15.1430777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1430858Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1431135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1431228Z layer_outputs = layer_module( 2025-08-26T20:37:15.1431521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1431702Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1432005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1432136Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1432440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1432571Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1432865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1432973Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1432977Z 2025-08-26T20:37:15.1433084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1433301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1433371Z return mod(**inputs) 2025-08-26T20:37:15.1433670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1433755Z outputs = self.mobilebert( 2025-08-26T20:37:15.1434050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1434133Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1434428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1434511Z layer_outputs = layer_module( 2025-08-26T20:37:15.1434804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1434979Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1435302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1435453Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1435762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1435851Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1435855Z 2025-08-26T20:37:15.1435973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1436181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1436255Z return mod(**inputs) 2025-08-26T20:37:15.1436558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1436637Z outputs = self.mobilebert( 2025-08-26T20:37:15.1436948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1437029Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1437332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1437421Z layer_outputs = layer_module( 2025-08-26T20:37:15.1437725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1437826Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1438154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1438232Z self_outputs = self.self( 2025-08-26T20:37:15.1438562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1438642Z self.value(value_tensor) 2025-08-26T20:37:15.1438646Z 2025-08-26T20:37:15.1438766Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1438982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1439061Z return mod(**inputs) 2025-08-26T20:37:15.1439366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1439520Z outputs = self.mobilebert( 2025-08-26T20:37:15.1439843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1439926Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1440236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1440316Z layer_outputs = layer_module( 2025-08-26T20:37:15.1440625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1440814Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1441130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1441257Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1441553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1441650Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1441654Z 2025-08-26T20:37:15.1441763Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1441972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1442076Z return mod(**inputs) 2025-08-26T20:37:15.1442393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1442481Z outputs = self.mobilebert( 2025-08-26T20:37:15.1442778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1442855Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1443163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1443240Z layer_outputs = layer_module( 2025-08-26T20:37:15.1443548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1443719Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1444028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1444148Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1444444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1444546Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1444842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1444968Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1444972Z 2025-08-26T20:37:15.1445081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1445308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1445389Z return mod(**inputs) 2025-08-26T20:37:15.1445684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1445769Z outputs = self.mobilebert( 2025-08-26T20:37:15.1446063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1446147Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1446441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1446520Z layer_outputs = layer_module( 2025-08-26T20:37:15.1446824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1446917Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1447221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1447298Z self_outputs = self.self( 2025-08-26T20:37:15.1447596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1447680Z self.query(query_tensor) 2025-08-26T20:37:15.1447684Z 2025-08-26T20:37:15.1447793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1448006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1448077Z return mod(**inputs) 2025-08-26T20:37:15.1448384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1448459Z outputs = self.mobilebert( 2025-08-26T20:37:15.1448767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1448868Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1449169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1449251Z layer_outputs = layer_module( 2025-08-26T20:37:15.1449530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1449615Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1449905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1449978Z self_outputs = self.self( 2025-08-26T20:37:15.1450261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1450331Z self.key(key_tensor) 2025-08-26T20:37:15.1450334Z 2025-08-26T20:37:15.1450425Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1450505Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1450612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1450814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1450879Z return mod(**inputs) 2025-08-26T20:37:15.1451165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1451236Z outputs = self.mobilebert( 2025-08-26T20:37:15.1451553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1451638Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1451958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1452041Z layer_outputs = layer_module( 2025-08-26T20:37:15.1452321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1452406Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1452694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1452817Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1453106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1453194Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1453198Z 2025-08-26T20:37:15.1453316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1453524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1453597Z return mod(**inputs) 2025-08-26T20:37:15.1453901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1453977Z outputs = self.mobilebert( 2025-08-26T20:37:15.1454277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1454357Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1454650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1454737Z layer_outputs = layer_module( 2025-08-26T20:37:15.1455033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1455132Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1455448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1455605Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1455905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1456036Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1456332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1456432Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1456435Z 2025-08-26T20:37:15.1456555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1456752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1456820Z return mod(**inputs) 2025-08-26T20:37:15.1457107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1457181Z outputs = self.mobilebert( 2025-08-26T20:37:15.1457464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1457539Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1457823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1457916Z layer_outputs = layer_module( 2025-08-26T20:37:15.1458198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1458322Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1458602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1458722Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1459001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1459085Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1459088Z 2025-08-26T20:37:15.1459200Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1459396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1459476Z return mod(**inputs) 2025-08-26T20:37:15.1459757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1459836Z outputs = self.mobilebert( 2025-08-26T20:37:15.1460116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1460188Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1460478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1460551Z layer_outputs = layer_module( 2025-08-26T20:37:15.1460838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1460934Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1461217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1461341Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1461621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1461771Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1461775Z 2025-08-26T20:37:15.1461881Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1462108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1462182Z return mod(**inputs) 2025-08-26T20:37:15.1462478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1462562Z outputs = self.mobilebert( 2025-08-26T20:37:15.1462865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1462951Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1463245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1463325Z layer_outputs = layer_module( 2025-08-26T20:37:15.1463634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1463730Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1464020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1464146Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1464435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1464540Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1464544Z 2025-08-26T20:37:15.1464646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1464872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1464940Z return mod(**inputs) 2025-08-26T20:37:15.1465227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1465300Z outputs = self.mobilebert( 2025-08-26T20:37:15.1465578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1465669Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1465949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1466030Z layer_outputs = layer_module( 2025-08-26T20:37:15.1466310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1466412Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1466697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1466823Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1467111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1467233Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1467521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1467616Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1467620Z 2025-08-26T20:37:15.1467723Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1467928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1467997Z return mod(**inputs) 2025-08-26T20:37:15.1468301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1468393Z outputs = self.mobilebert( 2025-08-26T20:37:15.1468700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1468778Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1469082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1469164Z layer_outputs = layer_module( 2025-08-26T20:37:15.1469444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1469546Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1469831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1469944Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1470233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1470316Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1470320Z 2025-08-26T20:37:15.1470431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1470627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1470721Z return mod(**inputs) 2025-08-26T20:37:15.1471005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1471096Z outputs = self.mobilebert( 2025-08-26T20:37:15.1471382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1471455Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1471741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1471812Z layer_outputs = layer_module( 2025-08-26T20:37:15.1472089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1472194Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1472490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1472617Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1472911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1473040Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1473043Z 2025-08-26T20:37:15.1473153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1473364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1473440Z return mod(**inputs) 2025-08-26T20:37:15.1473735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1473817Z outputs = self.mobilebert( 2025-08-26T20:37:15.1474109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1474189Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1474495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1474571Z layer_outputs = layer_module( 2025-08-26T20:37:15.1474896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1475015Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1475318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1475451Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1475750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1475849Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1475853Z 2025-08-26T20:37:15.1475964Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1476181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1476253Z return mod(**inputs) 2025-08-26T20:37:15.1476552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1476633Z outputs = self.mobilebert( 2025-08-26T20:37:15.1476925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1477007Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1477301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1477405Z layer_outputs = layer_module( 2025-08-26T20:37:15.1477699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1477821Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1478141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1478280Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1478599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1478728Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1479023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1479131Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1479134Z 2025-08-26T20:37:15.1479244Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1479533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1479616Z return mod(**inputs) 2025-08-26T20:37:15.1479933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1480014Z outputs = self.mobilebert( 2025-08-26T20:37:15.1480319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1480410Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1480719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1480806Z layer_outputs = layer_module( 2025-08-26T20:37:15.1481122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1481223Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1481581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1481702Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1482035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1482130Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1482133Z 2025-08-26T20:37:15.1482252Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1482461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1482533Z return mod(**inputs) 2025-08-26T20:37:15.1482835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1482909Z outputs = self.mobilebert( 2025-08-26T20:37:15.1483212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1483290Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1483584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1483667Z layer_outputs = layer_module( 2025-08-26T20:37:15.1483965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1484071Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1484367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1484510Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1484788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1484929Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1484933Z 2025-08-26T20:37:15.1485044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1485241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1485316Z return mod(**inputs) 2025-08-26T20:37:15.1485605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1485676Z outputs = self.mobilebert( 2025-08-26T20:37:15.1485953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1486025Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1486300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1486372Z layer_outputs = layer_module( 2025-08-26T20:37:15.1486649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1486741Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1487010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1487139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1487411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1487508Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1487511Z 2025-08-26T20:37:15.1487613Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1487810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1487883Z return mod(**inputs) 2025-08-26T20:37:15.1488181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1488278Z outputs = self.mobilebert( 2025-08-26T20:37:15.1488556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1488636Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1488914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1488988Z layer_outputs = layer_module( 2025-08-26T20:37:15.1489281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1489373Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1489650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1489773Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1490041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1490164Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1490436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1490552Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1490556Z 2025-08-26T20:37:15.1490658Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1490862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1490945Z return mod(**inputs) 2025-08-26T20:37:15.1491226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1491308Z outputs = self.mobilebert( 2025-08-26T20:37:15.1491588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1491667Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1491944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1492015Z layer_outputs = layer_module( 2025-08-26T20:37:15.1492300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1492421Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1492712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1492795Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1492799Z 2025-08-26T20:37:15.1492908Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1493102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1493169Z return mod(**inputs) 2025-08-26T20:37:15.1493455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1493527Z outputs = self.mobilebert( 2025-08-26T20:37:15.1493814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1493887Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1494168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1494262Z layer_outputs = layer_module( 2025-08-26T20:37:15.1494561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1494693Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1494971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1495091Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1495095Z 2025-08-26T20:37:15.1495199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1495395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1495470Z return mod(**inputs) 2025-08-26T20:37:15.1495754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1495836Z outputs = self.mobilebert( 2025-08-26T20:37:15.1496120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1496386Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1496682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1496756Z layer_outputs = layer_module( 2025-08-26T20:37:15.1497042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1497260Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1497567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1497702Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1497706Z 2025-08-26T20:37:15.1497827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1498031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1498098Z return mod(**inputs) 2025-08-26T20:37:15.1498385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1498457Z outputs = self.mobilebert( 2025-08-26T20:37:15.1498751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1498837Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1499142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1499222Z layer_outputs = layer_module( 2025-08-26T20:37:15.1499502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1499671Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1499948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1500074Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1500371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1500473Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1500477Z 2025-08-26T20:37:15.1500595Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1500803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1500910Z return mod(**inputs) 2025-08-26T20:37:15.1501497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1501573Z outputs = self.mobilebert( 2025-08-26T20:37:15.1501861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1501936Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1502230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1502308Z layer_outputs = layer_module( 2025-08-26T20:37:15.1502602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1502778Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1503074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1503213Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1503507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1503599Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1503612Z 2025-08-26T20:37:15.1503721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1503928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1504039Z return mod(**inputs) 2025-08-26T20:37:15.1504317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1504414Z outputs = self.mobilebert( 2025-08-26T20:37:15.1504696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1504772Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1505056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1505127Z layer_outputs = layer_module( 2025-08-26T20:37:15.1505416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1505571Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1505851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1505984Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1506263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1506391Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1506669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1506767Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1506770Z 2025-08-26T20:37:15.1506874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1507070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1507143Z return mod(**inputs) 2025-08-26T20:37:15.1507423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1507506Z outputs = self.mobilebert( 2025-08-26T20:37:15.1507799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1507880Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1508179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1508252Z layer_outputs = layer_module( 2025-08-26T20:37:15.1508553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1508725Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1509029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1509147Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1509443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1509538Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1509542Z 2025-08-26T20:37:15.1509662Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1509863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1509928Z return mod(**inputs) 2025-08-26T20:37:15.1510212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1510310Z outputs = self.mobilebert( 2025-08-26T20:37:15.1510594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1510694Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1510973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1511051Z layer_outputs = layer_module( 2025-08-26T20:37:15.1511333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1511420Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1511704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1511777Z self_outputs = self.self( 2025-08-26T20:37:15.1512066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1512141Z self.value(value_tensor) 2025-08-26T20:37:15.1512144Z 2025-08-26T20:37:15.1512256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1512453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1512522Z return mod(**inputs) 2025-08-26T20:37:15.1512827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1512904Z outputs = self.mobilebert( 2025-08-26T20:37:15.1513205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1513289Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1513566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1513649Z layer_outputs = layer_module( 2025-08-26T20:37:15.1513927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1514095Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1514394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1514533Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1514835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1514923Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1514927Z 2025-08-26T20:37:15.1515045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1515250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1515331Z return mod(**inputs) 2025-08-26T20:37:15.1515628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1515705Z outputs = self.mobilebert( 2025-08-26T20:37:15.1516007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1516086Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1516388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1516463Z layer_outputs = layer_module( 2025-08-26T20:37:15.1516758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1516988Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1517290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1517431Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1517729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1517829Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1518123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1518223Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1518227Z 2025-08-26T20:37:15.1518348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1518561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1518643Z return mod(**inputs) 2025-08-26T20:37:15.1518947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1519035Z outputs = self.mobilebert( 2025-08-26T20:37:15.1519341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1519422Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1519803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1519887Z layer_outputs = layer_module( 2025-08-26T20:37:15.1520205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1520300Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1520609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1520697Z self_outputs = self.self( 2025-08-26T20:37:15.1521021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1521125Z self.query(query_tensor) 2025-08-26T20:37:15.1521129Z 2025-08-26T20:37:15.1521243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1521477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1521552Z return mod(**inputs) 2025-08-26T20:37:15.1521847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1521931Z outputs = self.mobilebert( 2025-08-26T20:37:15.1522228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1522319Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1522613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1522690Z layer_outputs = layer_module( 2025-08-26T20:37:15.1522997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1523091Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1523391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1523465Z self_outputs = self.self( 2025-08-26T20:37:15.1523759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1523855Z self.key(key_tensor) 2025-08-26T20:37:15.1523859Z 2025-08-26T20:37:15.1523947Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1524037Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1524165Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1524382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1524451Z return mod(**inputs) 2025-08-26T20:37:15.1524751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1524835Z outputs = self.mobilebert( 2025-08-26T20:37:15.1525134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1525218Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1525515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1525594Z layer_outputs = layer_module( 2025-08-26T20:37:15.1525900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1525990Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1526291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1526423Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1526716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1526816Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1526820Z 2025-08-26T20:37:15.1526930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1527145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1527217Z return mod(**inputs) 2025-08-26T20:37:15.1527521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1527598Z outputs = self.mobilebert( 2025-08-26T20:37:15.1527910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1528015Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1528314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1528396Z layer_outputs = layer_module( 2025-08-26T20:37:15.1528695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1528786Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1529094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1529225Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1529531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1529666Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1529971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1530081Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1530085Z 2025-08-26T20:37:15.1530188Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1530394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1530483Z return mod(**inputs) 2025-08-26T20:37:15.1530771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1530865Z outputs = self.mobilebert( 2025-08-26T20:37:15.1531171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1531256Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1531555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1531638Z layer_outputs = layer_module( 2025-08-26T20:37:15.1531937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1532046Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1532346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1532465Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1532778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1532866Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1532869Z 2025-08-26T20:37:15.1532981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1533178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1533244Z return mod(**inputs) 2025-08-26T20:37:15.1533532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1533603Z outputs = self.mobilebert( 2025-08-26T20:37:15.1533890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1533962Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1534251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1534344Z layer_outputs = layer_module( 2025-08-26T20:37:15.1534660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1534764Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1535041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1535160Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1535434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1535547Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1535558Z 2025-08-26T20:37:15.1535659Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1535853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1535927Z return mod(**inputs) 2025-08-26T20:37:15.1536200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1536277Z outputs = self.mobilebert( 2025-08-26T20:37:15.1536547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1536620Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1536898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1536987Z layer_outputs = layer_module( 2025-08-26T20:37:15.1537277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1537390Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1537672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1537808Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1538088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1538180Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1538184Z 2025-08-26T20:37:15.1538287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1538492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1538559Z return mod(**inputs) 2025-08-26T20:37:15.1538841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1538920Z outputs = self.mobilebert( 2025-08-26T20:37:15.1539202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1539283Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1539563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1539634Z layer_outputs = layer_module( 2025-08-26T20:37:15.1539927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1540024Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1540337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1540475Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1540800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1540931Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1541244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1541353Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1541357Z 2025-08-26T20:37:15.1541465Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1541682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1541755Z return mod(**inputs) 2025-08-26T20:37:15.1542056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1542130Z outputs = self.mobilebert( 2025-08-26T20:37:15.1542409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1542491Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1542771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1542850Z layer_outputs = layer_module( 2025-08-26T20:37:15.1543132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1543229Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1543557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1543675Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1543996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1544087Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1544091Z 2025-08-26T20:37:15.1544208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1544420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1544489Z return mod(**inputs) 2025-08-26T20:37:15.1544802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1544878Z outputs = self.mobilebert( 2025-08-26T20:37:15.1545189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1545268Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1545573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1545659Z layer_outputs = layer_module( 2025-08-26T20:37:15.1545962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1546068Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1546372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1546489Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1546802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1546923Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1546927Z 2025-08-26T20:37:15.1547043Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1547259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1547355Z return mod(**inputs) 2025-08-26T20:37:15.1547670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1547750Z outputs = self.mobilebert( 2025-08-26T20:37:15.1548050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1548126Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1548427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1548503Z layer_outputs = layer_module( 2025-08-26T20:37:15.1548797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1548905Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1549198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1549339Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1549634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1549731Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1549735Z 2025-08-26T20:37:15.1549845Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1550052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1550151Z return mod(**inputs) 2025-08-26T20:37:15.1550447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1550553Z outputs = self.mobilebert( 2025-08-26T20:37:15.1550856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1550936Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1551245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1551321Z layer_outputs = layer_module( 2025-08-26T20:37:15.1551634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1551732Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1552043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1552174Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1552482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1552618Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1552922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1553026Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1553030Z 2025-08-26T20:37:15.1553140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1553359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1553434Z return mod(**inputs) 2025-08-26T20:37:15.1553739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1553826Z outputs = self.mobilebert( 2025-08-26T20:37:15.1554144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1554231Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1554545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1554623Z layer_outputs = layer_module( 2025-08-26T20:37:15.1554928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1555028Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1555326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1555445Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1555746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1555837Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1555840Z 2025-08-26T20:37:15.1555948Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1556165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1556235Z return mod(**inputs) 2025-08-26T20:37:15.1556534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1556608Z outputs = self.mobilebert( 2025-08-26T20:37:15.1556910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1557017Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1557329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1557433Z layer_outputs = layer_module( 2025-08-26T20:37:15.1557741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1557842Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1558163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1558283Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1558594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1558715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1558719Z 2025-08-26T20:37:15.1558838Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1559053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1559127Z return mod(**inputs) 2025-08-26T20:37:15.1559502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1559597Z outputs = self.mobilebert( 2025-08-26T20:37:15.1559917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1559997Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1560319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1560410Z layer_outputs = layer_module( 2025-08-26T20:37:15.1560731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1560844Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1561180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1561341Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1561641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1561734Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1561737Z 2025-08-26T20:37:15.1561859Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1562070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1562152Z return mod(**inputs) 2025-08-26T20:37:15.1562449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1562528Z outputs = self.mobilebert( 2025-08-26T20:37:15.1562833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1562910Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1563216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1563291Z layer_outputs = layer_module( 2025-08-26T20:37:15.1563594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1563696Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1564011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1564150Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1564472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1564608Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1564914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1565014Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1565017Z 2025-08-26T20:37:15.1565121Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1565320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1565398Z return mod(**inputs) 2025-08-26T20:37:15.1565689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1565774Z outputs = self.mobilebert( 2025-08-26T20:37:15.1566068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1566144Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1566450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1566525Z layer_outputs = layer_module( 2025-08-26T20:37:15.1566825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1566953Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1567256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1567347Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1567352Z 2025-08-26T20:37:15.1567463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1567698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1567772Z return mod(**inputs) 2025-08-26T20:37:15.1568095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1568174Z outputs = self.mobilebert( 2025-08-26T20:37:15.1568467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1568553Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1568852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1568937Z layer_outputs = layer_module( 2025-08-26T20:37:15.1569231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1569360Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1569665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1569783Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1569787Z 2025-08-26T20:37:15.1569904Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1570118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1570198Z return mod(**inputs) 2025-08-26T20:37:15.1570501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1570600Z outputs = self.mobilebert( 2025-08-26T20:37:15.1570914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1571016Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1571332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1571412Z layer_outputs = layer_module( 2025-08-26T20:37:15.1571719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1571895Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1572193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1572304Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1572307Z 2025-08-26T20:37:15.1572417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1572631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1572702Z return mod(**inputs) 2025-08-26T20:37:15.1573000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1573081Z outputs = self.mobilebert( 2025-08-26T20:37:15.1573374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1573457Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1573751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1573828Z layer_outputs = layer_module( 2025-08-26T20:37:15.1574130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1574300Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1574616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1574766Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1575072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1575170Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1575174Z 2025-08-26T20:37:15.1575284Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1575498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1575572Z return mod(**inputs) 2025-08-26T20:37:15.1575875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1575952Z outputs = self.mobilebert( 2025-08-26T20:37:15.1576453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1576536Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1576830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1576914Z layer_outputs = layer_module( 2025-08-26T20:37:15.1577209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1577384Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1577705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1577857Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1578168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1578262Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1578267Z 2025-08-26T20:37:15.1578387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1578602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1578683Z return mod(**inputs) 2025-08-26T20:37:15.1578990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1579080Z outputs = self.mobilebert( 2025-08-26T20:37:15.1579390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1579470Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1579779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1579855Z layer_outputs = layer_module( 2025-08-26T20:37:15.1580154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1580329Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1580627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1580765Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1581064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1581204Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1581521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1581623Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1581626Z 2025-08-26T20:37:15.1581762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1581974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1582052Z return mod(**inputs) 2025-08-26T20:37:15.1582348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1582425Z outputs = self.mobilebert( 2025-08-26T20:37:15.1582729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1582808Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1583112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1583187Z layer_outputs = layer_module( 2025-08-26T20:37:15.1583487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1583657Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1583953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1584080Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1584395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1584490Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1584524Z 2025-08-26T20:37:15.1584638Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1584856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1584925Z return mod(**inputs) 2025-08-26T20:37:15.1585223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1585307Z outputs = self.mobilebert( 2025-08-26T20:37:15.1585606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1585689Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1585984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1586060Z layer_outputs = layer_module( 2025-08-26T20:37:15.1586363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1586458Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1586762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1586840Z self_outputs = self.self( 2025-08-26T20:37:15.1587136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1587221Z self.value(value_tensor) 2025-08-26T20:37:15.1587224Z 2025-08-26T20:37:15.1587335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1587552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1587625Z return mod(**inputs) 2025-08-26T20:37:15.1587929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1588006Z outputs = self.mobilebert( 2025-08-26T20:37:15.1588325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1588430Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1588730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1588812Z layer_outputs = layer_module( 2025-08-26T20:37:15.1589109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1589281Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1589586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1589705Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1590009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1590097Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1590101Z 2025-08-26T20:37:15.1590217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1590423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1590492Z return mod(**inputs) 2025-08-26T20:37:15.1590795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1590889Z outputs = self.mobilebert( 2025-08-26T20:37:15.1591194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1591292Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1591589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1591672Z layer_outputs = layer_module( 2025-08-26T20:37:15.1591972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1592145Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1592444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1592565Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1592862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1592957Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1593262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1593359Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1593362Z 2025-08-26T20:37:15.1593479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1593683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1593756Z return mod(**inputs) 2025-08-26T20:37:15.1594058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1594134Z outputs = self.mobilebert( 2025-08-26T20:37:15.1594435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1594511Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1594834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1594912Z layer_outputs = layer_module( 2025-08-26T20:37:15.1595226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1595330Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1595634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1595720Z self_outputs = self.self( 2025-08-26T20:37:15.1596031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1596113Z self.query(query_tensor) 2025-08-26T20:37:15.1596124Z 2025-08-26T20:37:15.1596369Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1596592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1596675Z return mod(**inputs) 2025-08-26T20:37:15.1596981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1597070Z outputs = self.mobilebert( 2025-08-26T20:37:15.1597378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1597458Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1597774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1597901Z layer_outputs = layer_module( 2025-08-26T20:37:15.1598227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1598346Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1598657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1598743Z self_outputs = self.self( 2025-08-26T20:37:15.1599052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1599134Z self.key(key_tensor) 2025-08-26T20:37:15.1599138Z 2025-08-26T20:37:15.1599228Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1599322Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1599437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1599872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1599957Z return mod(**inputs) 2025-08-26T20:37:15.1600269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1600364Z outputs = self.mobilebert( 2025-08-26T20:37:15.1600679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1600759Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1601082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1601161Z layer_outputs = layer_module( 2025-08-26T20:37:15.1601487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1601582Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1601900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1602048Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1602393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1602531Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1602536Z 2025-08-26T20:37:15.1602651Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1602870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1602943Z return mod(**inputs) 2025-08-26T20:37:15.1603248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1603337Z outputs = self.mobilebert( 2025-08-26T20:37:15.1603640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1603730Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1604033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1604112Z layer_outputs = layer_module( 2025-08-26T20:37:15.1604433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1604523Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1604841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1604972Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1605306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1605443Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1605783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1605884Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1605887Z 2025-08-26T20:37:15.1605992Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1606194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1606261Z return mod(**inputs) 2025-08-26T20:37:15.1606540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1606624Z outputs = self.mobilebert( 2025-08-26T20:37:15.1606923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1607006Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1607309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1607387Z layer_outputs = layer_module( 2025-08-26T20:37:15.1607663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1607761Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1608047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1608161Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1608452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1608543Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1608547Z 2025-08-26T20:37:15.1608657Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1608889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1608961Z return mod(**inputs) 2025-08-26T20:37:15.1609284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1609362Z outputs = self.mobilebert( 2025-08-26T20:37:15.1609660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1632220Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1632714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1632828Z layer_outputs = layer_module( 2025-08-26T20:37:15.1633158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1633276Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1633591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1633723Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1634030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1634164Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1634170Z 2025-08-26T20:37:15.1634299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1634660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1634738Z return mod(**inputs) 2025-08-26T20:37:15.1635034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1635158Z outputs = self.mobilebert( 2025-08-26T20:37:15.1635444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1635536Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1635826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1635915Z layer_outputs = layer_module( 2025-08-26T20:37:15.1636213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1636321Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1636633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1636777Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1637088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1637184Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1637191Z 2025-08-26T20:37:15.1637317Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1637540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1637614Z return mod(**inputs) 2025-08-26T20:37:15.1637923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1638009Z outputs = self.mobilebert( 2025-08-26T20:37:15.1638318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1638403Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1638731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1638818Z layer_outputs = layer_module( 2025-08-26T20:37:15.1639145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1639259Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1639639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1639791Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1640092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1640223Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1640530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1640633Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1640637Z 2025-08-26T20:37:15.1640762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1640981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1641048Z return mod(**inputs) 2025-08-26T20:37:15.1641337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1641413Z outputs = self.mobilebert( 2025-08-26T20:37:15.1641724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1641803Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1642110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1642186Z layer_outputs = layer_module( 2025-08-26T20:37:15.1642470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1642575Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1642856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1642977Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1643258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1643347Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1643357Z 2025-08-26T20:37:15.1643461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1643674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1643742Z return mod(**inputs) 2025-08-26T20:37:15.1644031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1644104Z outputs = self.mobilebert( 2025-08-26T20:37:15.1644397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1644471Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1644751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1644831Z layer_outputs = layer_module( 2025-08-26T20:37:15.1645108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1645209Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1645506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1645673Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1645971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1646092Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1646096Z 2025-08-26T20:37:15.1646212Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1646428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1646507Z return mod(**inputs) 2025-08-26T20:37:15.1646804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1646882Z outputs = self.mobilebert( 2025-08-26T20:37:15.1647187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1647267Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1647572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1647647Z layer_outputs = layer_module( 2025-08-26T20:37:15.1647942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1648045Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1648346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1648499Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1648782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1648879Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1648884Z 2025-08-26T20:37:15.1648991Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1649193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1649269Z return mod(**inputs) 2025-08-26T20:37:15.1649551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1649631Z outputs = self.mobilebert( 2025-08-26T20:37:15.1649918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1649993Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1650290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1650363Z layer_outputs = layer_module( 2025-08-26T20:37:15.1650656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1650751Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1651051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1651185Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1651491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1651629Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1651953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1652066Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1652070Z 2025-08-26T20:37:15.1652193Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1652405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1652484Z return mod(**inputs) 2025-08-26T20:37:15.1652781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1652865Z outputs = self.mobilebert( 2025-08-26T20:37:15.1653164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1653249Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1653557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1653635Z layer_outputs = layer_module( 2025-08-26T20:37:15.1653948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1654045Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1654351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1654472Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1654781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1654898Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1654902Z 2025-08-26T20:37:15.1655013Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1655248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1655321Z return mod(**inputs) 2025-08-26T20:37:15.1655628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1655705Z outputs = self.mobilebert( 2025-08-26T20:37:15.1656003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1656088Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1656386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1656472Z layer_outputs = layer_module( 2025-08-26T20:37:15.1656769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1656871Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1657178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1657299Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1657602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1657721Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1657725Z 2025-08-26T20:37:15.1657840Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1658050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1658122Z return mod(**inputs) 2025-08-26T20:37:15.1658427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1658506Z outputs = self.mobilebert( 2025-08-26T20:37:15.1658828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1658934Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1659232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1659316Z layer_outputs = layer_module( 2025-08-26T20:37:15.1659612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1659719Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1660018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1660162Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1660459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1660548Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1660552Z 2025-08-26T20:37:15.1660670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1660883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1660960Z return mod(**inputs) 2025-08-26T20:37:15.1661255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1661352Z outputs = self.mobilebert( 2025-08-26T20:37:15.1661660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1661762Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1662068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1662147Z layer_outputs = layer_module( 2025-08-26T20:37:15.1662454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1662549Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1662831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1662964Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1663247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1663374Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1663655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1663751Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1663762Z 2025-08-26T20:37:15.1663868Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1664067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1664142Z return mod(**inputs) 2025-08-26T20:37:15.1664438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1664520Z outputs = self.mobilebert( 2025-08-26T20:37:15.1664824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1664914Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1665204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1665290Z layer_outputs = layer_module( 2025-08-26T20:37:15.1665600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1665726Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1666005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1666099Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1666102Z 2025-08-26T20:37:15.1666212Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1666431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1666502Z return mod(**inputs) 2025-08-26T20:37:15.1666817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1666892Z outputs = self.mobilebert( 2025-08-26T20:37:15.1667173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1667252Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1667533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1667612Z layer_outputs = layer_module( 2025-08-26T20:37:15.1667895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1668037Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1668327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1668460Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1668464Z 2025-08-26T20:37:15.1668580Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1668781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1668856Z return mod(**inputs) 2025-08-26T20:37:15.1669140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1669213Z outputs = self.mobilebert( 2025-08-26T20:37:15.1669520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1669600Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1669904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1669982Z layer_outputs = layer_module( 2025-08-26T20:37:15.1670283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1670465Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1670772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1670875Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1670879Z 2025-08-26T20:37:15.1670981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1671187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1671255Z return mod(**inputs) 2025-08-26T20:37:15.1671535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1671617Z outputs = self.mobilebert( 2025-08-26T20:37:15.1671919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1672001Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1672307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1672382Z layer_outputs = layer_module( 2025-08-26T20:37:15.1672688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1672861Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1673166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1673300Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1673606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1673707Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1673713Z 2025-08-26T20:37:15.1673821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1674040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1674110Z return mod(**inputs) 2025-08-26T20:37:15.1674416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1674511Z outputs = self.mobilebert( 2025-08-26T20:37:15.1674819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1674922Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1675223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1675307Z layer_outputs = layer_module( 2025-08-26T20:37:15.1675610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1675786Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1676084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1676217Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1676529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1676619Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1676625Z 2025-08-26T20:37:15.1676743Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1676954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1677024Z return mod(**inputs) 2025-08-26T20:37:15.1677330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1677407Z outputs = self.mobilebert( 2025-08-26T20:37:15.1677713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1677791Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1678095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1678172Z layer_outputs = layer_module( 2025-08-26T20:37:15.1678481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1678680Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1679003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1679148Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1679539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1679690Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1680006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1680109Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1680115Z 2025-08-26T20:37:15.1680236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1680465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1680544Z return mod(**inputs) 2025-08-26T20:37:15.1680863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1680939Z outputs = self.mobilebert( 2025-08-26T20:37:15.1681248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1681325Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1681633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1681736Z layer_outputs = layer_module( 2025-08-26T20:37:15.1682031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1682239Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1682539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1682665Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1682963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1683061Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1683065Z 2025-08-26T20:37:15.1683179Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1683391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1683466Z return mod(**inputs) 2025-08-26T20:37:15.1683766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1683852Z outputs = self.mobilebert( 2025-08-26T20:37:15.1684148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1684231Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1684529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1684605Z layer_outputs = layer_module( 2025-08-26T20:37:15.1684908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1685004Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1685308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1685388Z self_outputs = self.self( 2025-08-26T20:37:15.1685711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1685820Z self.value(value_tensor) 2025-08-26T20:37:15.1685824Z 2025-08-26T20:37:15.1685936Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1686155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1686226Z return mod(**inputs) 2025-08-26T20:37:15.1686525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1686611Z outputs = self.mobilebert( 2025-08-26T20:37:15.1686907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1686993Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1687294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1687376Z layer_outputs = layer_module( 2025-08-26T20:37:15.1687679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1687852Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1688158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1688277Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1688600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1688705Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1688709Z 2025-08-26T20:37:15.1688827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1689035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1689107Z return mod(**inputs) 2025-08-26T20:37:15.1689414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1689492Z outputs = self.mobilebert( 2025-08-26T20:37:15.1689797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1689872Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1690169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1690253Z layer_outputs = layer_module( 2025-08-26T20:37:15.1690552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1690733Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1691034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1691151Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1691431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1691522Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1691831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1691928Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1691934Z 2025-08-26T20:37:15.1692052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1692278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1692350Z return mod(**inputs) 2025-08-26T20:37:15.1692669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1692748Z outputs = self.mobilebert( 2025-08-26T20:37:15.1693045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1693123Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1693436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1693509Z layer_outputs = layer_module( 2025-08-26T20:37:15.1693788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1693886Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1694166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1694250Z self_outputs = self.self( 2025-08-26T20:37:15.1694530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1694606Z self.query(query_tensor) 2025-08-26T20:37:15.1694609Z 2025-08-26T20:37:15.1694721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1694915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1695007Z return mod(**inputs) 2025-08-26T20:37:15.1695294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1695389Z outputs = self.mobilebert( 2025-08-26T20:37:15.1695676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1695751Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1696040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1696111Z layer_outputs = layer_module( 2025-08-26T20:37:15.1696670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1696763Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1697047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1697129Z self_outputs = self.self( 2025-08-26T20:37:15.1697409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1697489Z self.key(key_tensor) 2025-08-26T20:37:15.1697492Z 2025-08-26T20:37:15.1697581Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1697663Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1697777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1697975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1698050Z return mod(**inputs) 2025-08-26T20:37:15.1698330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1698404Z outputs = self.mobilebert( 2025-08-26T20:37:15.1698707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1698786Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1699150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1699230Z layer_outputs = layer_module( 2025-08-26T20:37:15.1699564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1699657Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1699960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1700102Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1700403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1700505Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1700510Z 2025-08-26T20:37:15.1700621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1700830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1700911Z return mod(**inputs) 2025-08-26T20:37:15.1701214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1701299Z outputs = self.mobilebert( 2025-08-26T20:37:15.1701597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1701691Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1702008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1702081Z layer_outputs = layer_module( 2025-08-26T20:37:15.1702402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1702488Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1702783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1702906Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1703187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1703323Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1703603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1703705Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1703710Z 2025-08-26T20:37:15.1703813Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1704019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1704085Z return mod(**inputs) 2025-08-26T20:37:15.1704367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1704447Z outputs = self.mobilebert( 2025-08-26T20:37:15.1704731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1704817Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1705111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1705189Z layer_outputs = layer_module( 2025-08-26T20:37:15.1705500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1705615Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1705921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1706039Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1706326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1706412Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1706416Z 2025-08-26T20:37:15.1706520Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1706737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1706806Z return mod(**inputs) 2025-08-26T20:37:15.1707114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1707193Z outputs = self.mobilebert( 2025-08-26T20:37:15.1707493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1707581Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1707878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1707964Z layer_outputs = layer_module( 2025-08-26T20:37:15.1708265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1708392Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1708691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1708835Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1709148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1709272Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1709276Z 2025-08-26T20:37:15.1709393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1709602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1709673Z return mod(**inputs) 2025-08-26T20:37:15.1709980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1710059Z outputs = self.mobilebert( 2025-08-26T20:37:15.1710365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1710444Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1710746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1710831Z layer_outputs = layer_module( 2025-08-26T20:37:15.1711131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1711232Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1711534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1711668Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1711980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1712069Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1712075Z 2025-08-26T20:37:15.1712193Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1712423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1712498Z return mod(**inputs) 2025-08-26T20:37:15.1712820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1712900Z outputs = self.mobilebert( 2025-08-26T20:37:15.1713203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1713279Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1713580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1713664Z layer_outputs = layer_module( 2025-08-26T20:37:15.1713962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1714072Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1714368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1714508Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1714804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1714931Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1715233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1715352Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1715374Z 2025-08-26T20:37:15.1715493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1715705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1715784Z return mod(**inputs) 2025-08-26T20:37:15.1716082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1716158Z outputs = self.mobilebert( 2025-08-26T20:37:15.1716462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1716539Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1716844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1716920Z layer_outputs = layer_module( 2025-08-26T20:37:15.1717217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1717324Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1717624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1717752Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1718051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1718148Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1718152Z 2025-08-26T20:37:15.1718260Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1718472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1718552Z return mod(**inputs) 2025-08-26T20:37:15.1718849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1718933Z outputs = self.mobilebert( 2025-08-26T20:37:15.1719251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1719348Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1719731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1719815Z layer_outputs = layer_module( 2025-08-26T20:37:15.1720132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1720239Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1720545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1720678Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1720993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1721124Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1721129Z 2025-08-26T20:37:15.1721239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1721457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1721529Z return mod(**inputs) 2025-08-26T20:37:15.1721829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1721942Z outputs = self.mobilebert( 2025-08-26T20:37:15.1722238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1722351Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1722650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1722728Z layer_outputs = layer_module( 2025-08-26T20:37:15.1723036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1723146Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1723439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1723563Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1723855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1723940Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1723944Z 2025-08-26T20:37:15.1724047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1724253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1724320Z return mod(**inputs) 2025-08-26T20:37:15.1724606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1724678Z outputs = self.mobilebert( 2025-08-26T20:37:15.1724955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1725035Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1725316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1725396Z layer_outputs = layer_module( 2025-08-26T20:37:15.1725677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1725796Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1726095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1726221Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1726511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1726631Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1726917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1727011Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1727015Z 2025-08-26T20:37:15.1727128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1727328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1727395Z return mod(**inputs) 2025-08-26T20:37:15.1727685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1727758Z outputs = self.mobilebert( 2025-08-26T20:37:15.1728045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1728118Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1728396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1728496Z layer_outputs = layer_module( 2025-08-26T20:37:15.1728775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1728892Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1729174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1729293Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1729576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1729661Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1729665Z 2025-08-26T20:37:15.1729776Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1729977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1730051Z return mod(**inputs) 2025-08-26T20:37:15.1730332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1730405Z outputs = self.mobilebert( 2025-08-26T20:37:15.1730696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1730772Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1731064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1731136Z layer_outputs = layer_module( 2025-08-26T20:37:15.1731419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1731527Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1731822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1731949Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1732262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1732384Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1732402Z 2025-08-26T20:37:15.1732508Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1732705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1732780Z return mod(**inputs) 2025-08-26T20:37:15.1733062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1733145Z outputs = self.mobilebert( 2025-08-26T20:37:15.1733424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1733507Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1733785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1733858Z layer_outputs = layer_module( 2025-08-26T20:37:15.1734145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1734239Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1734529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1734654Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1734961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1735056Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1735078Z 2025-08-26T20:37:15.1735184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1735389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1735458Z return mod(**inputs) 2025-08-26T20:37:15.1735743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1735815Z outputs = self.mobilebert( 2025-08-26T20:37:15.1736092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1736173Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1736450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1736529Z layer_outputs = layer_module( 2025-08-26T20:37:15.1736811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1736908Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1737196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1737321Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1737606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1737726Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1738013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1738108Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1738112Z 2025-08-26T20:37:15.1738218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1738436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1738505Z return mod(**inputs) 2025-08-26T20:37:15.1738808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1738887Z outputs = self.mobilebert( 2025-08-26T20:37:15.1739184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1739271Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1739570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1739655Z layer_outputs = layer_module( 2025-08-26T20:37:15.1739952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1740086Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1740382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1740467Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1740471Z 2025-08-26T20:37:15.1740582Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1740779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1740851Z return mod(**inputs) 2025-08-26T20:37:15.1741128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1741218Z outputs = self.mobilebert( 2025-08-26T20:37:15.1741518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1741613Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1741915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1741992Z layer_outputs = layer_module( 2025-08-26T20:37:15.1742292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1742419Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1742716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1742844Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1742848Z 2025-08-26T20:37:15.1742955Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1743166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1743237Z return mod(**inputs) 2025-08-26T20:37:15.1743531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1743616Z outputs = self.mobilebert( 2025-08-26T20:37:15.1743909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1743994Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1744288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1744370Z layer_outputs = layer_module( 2025-08-26T20:37:15.1744668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1744837Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1745160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1745264Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1745285Z 2025-08-26T20:37:15.1745402Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1745610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1745681Z return mod(**inputs) 2025-08-26T20:37:15.1745985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1746063Z outputs = self.mobilebert( 2025-08-26T20:37:15.1746370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1746451Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1746758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1746834Z layer_outputs = layer_module( 2025-08-26T20:37:15.1747131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1747309Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1747605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1747744Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1748061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1748160Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1748188Z 2025-08-26T20:37:15.1748299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1748508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1748588Z return mod(**inputs) 2025-08-26T20:37:15.1748885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1748967Z outputs = self.mobilebert( 2025-08-26T20:37:15.1749262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1749340Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1749644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1749720Z layer_outputs = layer_module( 2025-08-26T20:37:15.1750027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1750195Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1750492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1750632Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1750926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1751024Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1751028Z 2025-08-26T20:37:15.1751140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1751357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1751429Z return mod(**inputs) 2025-08-26T20:37:15.1751781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1751866Z outputs = self.mobilebert( 2025-08-26T20:37:15.1752185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1752273Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1752570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1752647Z layer_outputs = layer_module( 2025-08-26T20:37:15.1752952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1753121Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1753425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1753557Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1753860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1753989Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1754291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1754399Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1754403Z 2025-08-26T20:37:15.1754533Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1754749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1754820Z return mod(**inputs) 2025-08-26T20:37:15.1755133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1755221Z outputs = self.mobilebert( 2025-08-26T20:37:15.1755519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1755607Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1755905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1755988Z layer_outputs = layer_module( 2025-08-26T20:37:15.1756284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1756459Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1756766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1756886Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1757193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1757281Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1757285Z 2025-08-26T20:37:15.1757400Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1757610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1757682Z return mod(**inputs) 2025-08-26T20:37:15.1757994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1758074Z outputs = self.mobilebert( 2025-08-26T20:37:15.1758384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1758466Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1758785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1758899Z layer_outputs = layer_module( 2025-08-26T20:37:15.1759207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1759307Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1759715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1759802Z self_outputs = self.self( 2025-08-26T20:37:15.1760120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1760203Z self.value(value_tensor) 2025-08-26T20:37:15.1760206Z 2025-08-26T20:37:15.1760331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1760547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1760630Z return mod(**inputs) 2025-08-26T20:37:15.1760933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1761011Z outputs = self.mobilebert( 2025-08-26T20:37:15.1761321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1761402Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1761741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1761820Z layer_outputs = layer_module( 2025-08-26T20:37:15.1762145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1762333Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1762641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1762771Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1763074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1763172Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1763177Z 2025-08-26T20:37:15.1763290Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1763505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1763587Z return mod(**inputs) 2025-08-26T20:37:15.1763892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1763977Z outputs = self.mobilebert( 2025-08-26T20:37:15.1764286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1764367Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1764680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1764757Z layer_outputs = layer_module( 2025-08-26T20:37:15.1765071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1765245Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1765561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1765698Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1766025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1766132Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1766438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1766548Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1766552Z 2025-08-26T20:37:15.1766667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1766896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1766971Z return mod(**inputs) 2025-08-26T20:37:15.1767284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1767375Z outputs = self.mobilebert( 2025-08-26T20:37:15.1767685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1767802Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1768102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1768178Z layer_outputs = layer_module( 2025-08-26T20:37:15.1768487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1768599Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1768902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1769002Z self_outputs = self.self( 2025-08-26T20:37:15.1769297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1769383Z self.query(query_tensor) 2025-08-26T20:37:15.1769389Z 2025-08-26T20:37:15.1769499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1769718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1769790Z return mod(**inputs) 2025-08-26T20:37:15.1770091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1770168Z outputs = self.mobilebert( 2025-08-26T20:37:15.1770464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1770551Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1770850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1770933Z layer_outputs = layer_module( 2025-08-26T20:37:15.1771229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1771320Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1771626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1771699Z self_outputs = self.self( 2025-08-26T20:37:15.1772003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1772074Z self.key(key_tensor) 2025-08-26T20:37:15.1772078Z 2025-08-26T20:37:15.1772169Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1772262Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1772390Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1772613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1772701Z return mod(**inputs) 2025-08-26T20:37:15.1773006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1773083Z outputs = self.mobilebert( 2025-08-26T20:37:15.1773374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1773462Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1773753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1773835Z layer_outputs = layer_module( 2025-08-26T20:37:15.1774126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1774217Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1774520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1774652Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1774954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1775045Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1775068Z 2025-08-26T20:37:15.1775186Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1775395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1775482Z return mod(**inputs) 2025-08-26T20:37:15.1775791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1775867Z outputs = self.mobilebert( 2025-08-26T20:37:15.1776169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1776246Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1776541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1776625Z layer_outputs = layer_module( 2025-08-26T20:37:15.1776923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1777022Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1777327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1777464Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1777777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1777913Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1778226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1778328Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1778332Z 2025-08-26T20:37:15.1778453Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1778667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1778740Z return mod(**inputs) 2025-08-26T20:37:15.1779058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1779153Z outputs = self.mobilebert( 2025-08-26T20:37:15.1779477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1779557Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1779852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1779935Z layer_outputs = layer_module( 2025-08-26T20:37:15.1780232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1780343Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1780637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1780768Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1781066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1781158Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1781162Z 2025-08-26T20:37:15.1781277Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1781485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1781563Z return mod(**inputs) 2025-08-26T20:37:15.1781861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1781966Z outputs = self.mobilebert( 2025-08-26T20:37:15.1782262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1782358Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1782666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1782743Z layer_outputs = layer_module( 2025-08-26T20:37:15.1783047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1783150Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1783445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1783574Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1783872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1784003Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1784006Z 2025-08-26T20:37:15.1784116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1784332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1784404Z return mod(**inputs) 2025-08-26T20:37:15.1784703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1784788Z outputs = self.mobilebert( 2025-08-26T20:37:15.1785083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1785168Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1785468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1785543Z layer_outputs = layer_module( 2025-08-26T20:37:15.1785851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1786293Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1786616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1786754Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1787069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1787158Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1787162Z 2025-08-26T20:37:15.1787266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1787472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1787541Z return mod(**inputs) 2025-08-26T20:37:15.1787828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1787900Z outputs = self.mobilebert( 2025-08-26T20:37:15.1788181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1788264Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1788547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1788626Z layer_outputs = layer_module( 2025-08-26T20:37:15.1788906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1789021Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1789306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1789451Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1789740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1789860Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1790149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1790248Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1790252Z 2025-08-26T20:37:15.1790359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1790580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1790651Z return mod(**inputs) 2025-08-26T20:37:15.1790959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1791032Z outputs = self.mobilebert( 2025-08-26T20:37:15.1791318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1791390Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1791671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1791751Z layer_outputs = layer_module( 2025-08-26T20:37:15.1792040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1792148Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1792445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1792566Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1792889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1792998Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1793003Z 2025-08-26T20:37:15.1793122Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1793332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1793409Z return mod(**inputs) 2025-08-26T20:37:15.1793704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1793783Z outputs = self.mobilebert( 2025-08-26T20:37:15.1794089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1794168Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1794473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1794550Z layer_outputs = layer_module( 2025-08-26T20:37:15.1794846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1794953Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1795247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1795372Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1795692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1795817Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1795839Z 2025-08-26T20:37:15.1795949Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1796364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1796453Z return mod(**inputs) 2025-08-26T20:37:15.1796753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1796838Z outputs = self.mobilebert( 2025-08-26T20:37:15.1797137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1797217Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1797534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1797613Z layer_outputs = layer_module( 2025-08-26T20:37:15.1797927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1798032Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1798346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1798481Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1798795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1798895Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1798900Z 2025-08-26T20:37:15.1799011Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1799228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1799300Z return mod(**inputs) 2025-08-26T20:37:15.1799693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1799787Z outputs = self.mobilebert( 2025-08-26T20:37:15.1800113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1800204Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1800511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1800591Z layer_outputs = layer_module( 2025-08-26T20:37:15.1800909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1801012Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1801329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1801459Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1801749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1801872Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1802154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1802262Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1802266Z 2025-08-26T20:37:15.1802377Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1802628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1802698Z return mod(**inputs) 2025-08-26T20:37:15.1803002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1803110Z outputs = self.mobilebert( 2025-08-26T20:37:15.1803410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1803496Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1803792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1803875Z layer_outputs = layer_module( 2025-08-26T20:37:15.1804173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1804276Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1804583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1804705Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1805011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1805101Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1805105Z 2025-08-26T20:37:15.1805223Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1805434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1805503Z return mod(**inputs) 2025-08-26T20:37:15.1805805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1805884Z outputs = self.mobilebert( 2025-08-26T20:37:15.1806189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1806269Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1806586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1806671Z layer_outputs = layer_module( 2025-08-26T20:37:15.1806979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1807089Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1807385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1807509Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1807807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1807925Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1807931Z 2025-08-26T20:37:15.1808046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1808253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1808330Z return mod(**inputs) 2025-08-26T20:37:15.1808624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1808702Z outputs = self.mobilebert( 2025-08-26T20:37:15.1809001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1809078Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1809398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1809475Z layer_outputs = layer_module( 2025-08-26T20:37:15.1809803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1809903Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1810204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1810344Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1810642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1810741Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1810745Z 2025-08-26T20:37:15.1810855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1811064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1811140Z return mod(**inputs) 2025-08-26T20:37:15.1811436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1811519Z outputs = self.mobilebert( 2025-08-26T20:37:15.1811818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1811901Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1812195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1812269Z layer_outputs = layer_module( 2025-08-26T20:37:15.1812573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1812675Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1812978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1813111Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1813419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1813573Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1813868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1813972Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1813976Z 2025-08-26T20:37:15.1814086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1814301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1814371Z return mod(**inputs) 2025-08-26T20:37:15.1814664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1814749Z outputs = self.mobilebert( 2025-08-26T20:37:15.1815045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1815132Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1815428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1815506Z layer_outputs = layer_module( 2025-08-26T20:37:15.1815806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1815954Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1816257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1816369Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1816373Z 2025-08-26T20:37:15.1816492Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1816698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1816769Z return mod(**inputs) 2025-08-26T20:37:15.1817076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1817154Z outputs = self.mobilebert( 2025-08-26T20:37:15.1817465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1817547Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1817852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1817942Z layer_outputs = layer_module( 2025-08-26T20:37:15.1818247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1818386Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1818693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1818822Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1818826Z 2025-08-26T20:37:15.1818938Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1819152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1819235Z return mod(**inputs) 2025-08-26T20:37:15.1819545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1819634Z outputs = self.mobilebert( 2025-08-26T20:37:15.1819959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1820040Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1820370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1820450Z layer_outputs = layer_module( 2025-08-26T20:37:15.1820762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1820936Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1821249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1821352Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1821358Z 2025-08-26T20:37:15.1821471Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1821693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1821766Z return mod(**inputs) 2025-08-26T20:37:15.1822075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1822154Z outputs = self.mobilebert( 2025-08-26T20:37:15.1822461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1822549Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1822851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1822959Z layer_outputs = layer_module( 2025-08-26T20:37:15.1823284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1823468Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1823779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1823915Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1824226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1824328Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1824332Z 2025-08-26T20:37:15.1824453Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1824665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1824739Z return mod(**inputs) 2025-08-26T20:37:15.1825052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1825132Z outputs = self.mobilebert( 2025-08-26T20:37:15.1825444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1825524Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1825834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1825912Z layer_outputs = layer_module( 2025-08-26T20:37:15.1826216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1826397Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1826702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1826891Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1827227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1827323Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1827334Z 2025-08-26T20:37:15.1827447Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1827661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1827741Z return mod(**inputs) 2025-08-26T20:37:15.1828043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1828131Z outputs = self.mobilebert( 2025-08-26T20:37:15.1828433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1828515Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1828829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1828906Z layer_outputs = layer_module( 2025-08-26T20:37:15.1829216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1829387Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1829689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1829852Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1830162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1830323Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1830629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1830739Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1830743Z 2025-08-26T20:37:15.1830855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1831072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1831153Z return mod(**inputs) 2025-08-26T20:37:15.1831458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1831547Z outputs = self.mobilebert( 2025-08-26T20:37:15.1831851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1831932Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1832249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1832330Z layer_outputs = layer_module( 2025-08-26T20:37:15.1832649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1832825Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1833129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1833248Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1833545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1833643Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1833647Z 2025-08-26T20:37:15.1833772Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1834005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1834078Z return mod(**inputs) 2025-08-26T20:37:15.1834385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1834460Z outputs = self.mobilebert( 2025-08-26T20:37:15.1834760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1834847Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1835142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1835228Z layer_outputs = layer_module( 2025-08-26T20:37:15.1835526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1835621Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1835937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1836017Z self_outputs = self.self( 2025-08-26T20:37:15.1836325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1836402Z self.value(value_tensor) 2025-08-26T20:37:15.1836437Z 2025-08-26T20:37:15.1836568Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1836782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1836874Z return mod(**inputs) 2025-08-26T20:37:15.1837186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1837265Z outputs = self.mobilebert( 2025-08-26T20:37:15.1837580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1837660Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1837963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1838050Z layer_outputs = layer_module( 2025-08-26T20:37:15.1838355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1838541Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1838852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1838976Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1839289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1839379Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1839384Z 2025-08-26T20:37:15.1839585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1839802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1839882Z return mod(**inputs) 2025-08-26T20:37:15.1840186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1840265Z outputs = self.mobilebert( 2025-08-26T20:37:15.1840577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1840681Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1841018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1841099Z layer_outputs = layer_module( 2025-08-26T20:37:15.1841414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1841594Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1841892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1842019Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1842316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1842419Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1842718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1842817Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1842821Z 2025-08-26T20:37:15.1842939Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1843148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1843226Z return mod(**inputs) 2025-08-26T20:37:15.1843518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1843620Z outputs = self.mobilebert( 2025-08-26T20:37:15.1843916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1844014Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1844319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1844396Z layer_outputs = layer_module( 2025-08-26T20:37:15.1844701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1844791Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1845088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1845173Z self_outputs = self.self( 2025-08-26T20:37:15.1845467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1845552Z self.query(query_tensor) 2025-08-26T20:37:15.1845555Z 2025-08-26T20:37:15.1845665Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1845874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1845951Z return mod(**inputs) 2025-08-26T20:37:15.1846245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1846327Z outputs = self.mobilebert( 2025-08-26T20:37:15.1846625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1846710Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1847008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1847086Z layer_outputs = layer_module( 2025-08-26T20:37:15.1847387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1847493Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1847817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1847894Z self_outputs = self.self( 2025-08-26T20:37:15.1848193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1848273Z self.key(key_tensor) 2025-08-26T20:37:15.1848277Z 2025-08-26T20:37:15.1848365Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1848460Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1848570Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1848782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1848862Z return mod(**inputs) 2025-08-26T20:37:15.1849162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1849249Z outputs = self.mobilebert( 2025-08-26T20:37:15.1849548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1849634Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1849932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1850008Z layer_outputs = layer_module( 2025-08-26T20:37:15.1850313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1850422Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1850746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1850879Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1851177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1851278Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1851282Z 2025-08-26T20:37:15.1851392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1851609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1851680Z return mod(**inputs) 2025-08-26T20:37:15.1851984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1852059Z outputs = self.mobilebert( 2025-08-26T20:37:15.1852359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1852446Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1852746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1852829Z layer_outputs = layer_module( 2025-08-26T20:37:15.1853122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1853211Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1853516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1853648Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1853952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1854088Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1854409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1854527Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1854532Z 2025-08-26T20:37:15.1854642Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1854860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1854933Z return mod(**inputs) 2025-08-26T20:37:15.1855239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1855318Z outputs = self.mobilebert( 2025-08-26T20:37:15.1855615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1855697Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1855984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1856070Z layer_outputs = layer_module( 2025-08-26T20:37:15.1856355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1856460Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1856748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1856890Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1857195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1857337Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1857340Z 2025-08-26T20:37:15.1857457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1857665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1857737Z return mod(**inputs) 2025-08-26T20:37:15.1858041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1858118Z outputs = self.mobilebert( 2025-08-26T20:37:15.1858421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1858498Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1858801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1858873Z layer_outputs = layer_module( 2025-08-26T20:37:15.1859156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1859259Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1859540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1859659Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1859938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1860060Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1860073Z 2025-08-26T20:37:15.1860182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1860393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1860473Z return mod(**inputs) 2025-08-26T20:37:15.1860797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1860885Z outputs = self.mobilebert( 2025-08-26T20:37:15.1861218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1861300Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1861616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1861693Z layer_outputs = layer_module( 2025-08-26T20:37:15.1862014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1862117Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1862419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1862562Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1862865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1862965Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1862969Z 2025-08-26T20:37:15.1863076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1863292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1863363Z return mod(**inputs) 2025-08-26T20:37:15.1863665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1863769Z outputs = self.mobilebert( 2025-08-26T20:37:15.1864067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1864172Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1864471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1864549Z layer_outputs = layer_module( 2025-08-26T20:37:15.1864862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1864968Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1865276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1865416Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1865736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1865866Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1866163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1866273Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1866276Z 2025-08-26T20:37:15.1866384Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1866602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1866672Z return mod(**inputs) 2025-08-26T20:37:15.1866966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1867051Z outputs = self.mobilebert( 2025-08-26T20:37:15.1867344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1867431Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1867746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1867847Z layer_outputs = layer_module( 2025-08-26T20:37:15.1868148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1868246Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1868552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1868672Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1868977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1869070Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1869074Z 2025-08-26T20:37:15.1869192Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1869402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1869473Z return mod(**inputs) 2025-08-26T20:37:15.1869773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1869849Z outputs = self.mobilebert( 2025-08-26T20:37:15.1870146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1870223Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1870543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1870629Z layer_outputs = layer_module( 2025-08-26T20:37:15.1870952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1871073Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1871368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1871484Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1871788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1871906Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1871911Z 2025-08-26T20:37:15.1872028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1872238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1872317Z return mod(**inputs) 2025-08-26T20:37:15.1872612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1872689Z outputs = self.mobilebert( 2025-08-26T20:37:15.1872992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1873069Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1873370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1873446Z layer_outputs = layer_module( 2025-08-26T20:37:15.1873740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1873849Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1874147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1874310Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1874625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1874725Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1874728Z 2025-08-26T20:37:15.1874838Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1875050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1875129Z return mod(**inputs) 2025-08-26T20:37:15.1875429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1875517Z outputs = self.mobilebert( 2025-08-26T20:37:15.1875825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1875905Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1876218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1876297Z layer_outputs = layer_module( 2025-08-26T20:37:15.1876614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1876715Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1877029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1877187Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1877496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1877659Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1877965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1878077Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1878081Z 2025-08-26T20:37:15.1878192Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1878414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1878488Z return mod(**inputs) 2025-08-26T20:37:15.1878789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1878874Z outputs = self.mobilebert( 2025-08-26T20:37:15.1879175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1879262Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1879655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1879740Z layer_outputs = layer_module( 2025-08-26T20:37:15.1880060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1880164Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1880477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1880601Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1880916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1881013Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1881017Z 2025-08-26T20:37:15.1881127Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1881368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1881460Z return mod(**inputs) 2025-08-26T20:37:15.1881767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1881844Z outputs = self.mobilebert( 2025-08-26T20:37:15.1882140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1882226Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1882525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1882608Z layer_outputs = layer_module( 2025-08-26T20:37:15.1882907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1883009Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1883316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1883436Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1883739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1883856Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1883860Z 2025-08-26T20:37:15.1883996Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1884207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1884300Z return mod(**inputs) 2025-08-26T20:37:15.1884607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1884685Z outputs = self.mobilebert( 2025-08-26T20:37:15.1884991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1885068Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1885365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1885449Z layer_outputs = layer_module( 2025-08-26T20:37:15.1885744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1885853Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1886150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1886292Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1886590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1886681Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1886684Z 2025-08-26T20:37:15.1886802Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1887013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1887092Z return mod(**inputs) 2025-08-26T20:37:15.1887388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1887466Z outputs = self.mobilebert( 2025-08-26T20:37:15.1887773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1887852Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1888184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1888281Z layer_outputs = layer_module( 2025-08-26T20:37:15.1888586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1888686Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1888980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1889121Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1889417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1889553Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1889849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1889949Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1889961Z 2025-08-26T20:37:15.1890072Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1890280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1890357Z return mod(**inputs) 2025-08-26T20:37:15.1890653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1890758Z outputs = self.mobilebert( 2025-08-26T20:37:15.1891052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1891149Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1891455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1891531Z layer_outputs = layer_module( 2025-08-26T20:37:15.1891834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1891962Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1892255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1892351Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1892357Z 2025-08-26T20:37:15.1892465Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1892680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1892754Z return mod(**inputs) 2025-08-26T20:37:15.1893060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1893136Z outputs = self.mobilebert( 2025-08-26T20:37:15.1893433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1893519Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1893812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1893894Z layer_outputs = layer_module( 2025-08-26T20:37:15.1894190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1894314Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1894622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1894759Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1894764Z 2025-08-26T20:37:15.1894897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1895109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1895186Z return mod(**inputs) 2025-08-26T20:37:15.1895484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1895561Z outputs = self.mobilebert( 2025-08-26T20:37:15.1895869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1895946Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1896573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1896659Z layer_outputs = layer_module( 2025-08-26T20:37:15.1896957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1897139Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1897437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1897550Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1897554Z 2025-08-26T20:37:15.1897720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1897940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1898012Z return mod(**inputs) 2025-08-26T20:37:15.1898336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1898421Z outputs = self.mobilebert( 2025-08-26T20:37:15.1898723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1898806Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1899101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1899176Z layer_outputs = layer_module( 2025-08-26T20:37:15.1899480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1899642Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1899937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1900072Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1900377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1900476Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1900479Z 2025-08-26T20:37:15.1900588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1900806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1900878Z return mod(**inputs) 2025-08-26T20:37:15.1901181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1901259Z outputs = self.mobilebert( 2025-08-26T20:37:15.1901564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1901687Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1902014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1902102Z layer_outputs = layer_module( 2025-08-26T20:37:15.1902400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1902586Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1902875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1903003Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1903296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1903384Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1903389Z 2025-08-26T20:37:15.1903503Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1903709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1903789Z return mod(**inputs) 2025-08-26T20:37:15.1904088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1904164Z outputs = self.mobilebert( 2025-08-26T20:37:15.1904474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1904571Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1904878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1904973Z layer_outputs = layer_module( 2025-08-26T20:37:15.1905279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1905457Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1905753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1905892Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1906190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1906328Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1906626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1906725Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1906729Z 2025-08-26T20:37:15.1906847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1907057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1907134Z return mod(**inputs) 2025-08-26T20:37:15.1907427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1907505Z outputs = self.mobilebert( 2025-08-26T20:37:15.1907806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1907884Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1908185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1908262Z layer_outputs = layer_module( 2025-08-26T20:37:15.1908580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1908771Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1909070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1909194Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1909491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1909588Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1909592Z 2025-08-26T20:37:15.1909702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1909911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1909991Z return mod(**inputs) 2025-08-26T20:37:15.1910288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1910375Z outputs = self.mobilebert( 2025-08-26T20:37:15.1910672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1910756Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1911050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1911125Z layer_outputs = layer_module( 2025-08-26T20:37:15.1911449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1911542Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1911872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1911949Z self_outputs = self.self( 2025-08-26T20:37:15.1912246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:37:15.1912330Z self.value(value_tensor) 2025-08-26T20:37:15.1912334Z 2025-08-26T20:37:15.1912444Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1912659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1912729Z return mod(**inputs) 2025-08-26T20:37:15.1913032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1913107Z outputs = self.mobilebert( 2025-08-26T20:37:15.1913405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1913495Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1913792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1913874Z layer_outputs = layer_module( 2025-08-26T20:37:15.1914169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1914347Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1914635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:37:15.1914747Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:37:15.1915042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:37:15.1915148Z layer_input = self.dense(hidden_states) 2025-08-26T20:37:15.1915153Z 2025-08-26T20:37:15.1915269Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1915497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1915568Z return mod(**inputs) 2025-08-26T20:37:15.1915876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1915950Z outputs = self.mobilebert( 2025-08-26T20:37:15.1916253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1916332Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1916628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1916712Z layer_outputs = layer_module( 2025-08-26T20:37:15.1917010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:37:15.1917191Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:37:15.1917491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:37:15.1917615Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:37:15.1917913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:37:15.1918026Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:37:15.1918332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1918451Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1918454Z 2025-08-26T20:37:15.1918573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1918784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1918856Z return mod(**inputs) 2025-08-26T20:37:15.1919160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1919237Z outputs = self.mobilebert( 2025-08-26T20:37:15.1919609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1919698Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1920010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1920090Z layer_outputs = layer_module( 2025-08-26T20:37:15.1920404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1920503Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1920798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1920884Z self_outputs = self.self( 2025-08-26T20:37:15.1921191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:37:15.1921262Z self.query(query_tensor) 2025-08-26T20:37:15.1921268Z 2025-08-26T20:37:15.1921380Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1921576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1921653Z return mod(**inputs) 2025-08-26T20:37:15.1921945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1922027Z outputs = self.mobilebert( 2025-08-26T20:37:15.1922322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1922396Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1922680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1922751Z layer_outputs = layer_module( 2025-08-26T20:37:15.1923036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1923125Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1923406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:37:15.1923495Z self_outputs = self.self( 2025-08-26T20:37:15.1923769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:37:15.1923843Z self.key(key_tensor) 2025-08-26T20:37:15.1923846Z 2025-08-26T20:37:15.1923927Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1924007Z cudagraph partition due to non gpu ops 2025-08-26T20:37:15.1924119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1924311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1924383Z return mod(**inputs) 2025-08-26T20:37:15.1924676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1924754Z outputs = self.mobilebert( 2025-08-26T20:37:15.1925056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1925129Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1925418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1925490Z layer_outputs = layer_module( 2025-08-26T20:37:15.1925775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1925858Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1926142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1926276Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1926555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:37:15.1926671Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1926674Z 2025-08-26T20:37:15.1926778Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1926989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1927055Z return mod(**inputs) 2025-08-26T20:37:15.1927335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1927414Z outputs = self.mobilebert( 2025-08-26T20:37:15.1927691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1927772Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1928054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1928132Z layer_outputs = layer_module( 2025-08-26T20:37:15.1928453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:37:15.1928562Z self_attention_outputs = self.attention( 2025-08-26T20:37:15.1928870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:37:15.1928999Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:37:15.1929301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:37:15.1929437Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1929732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1929842Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1929846Z 2025-08-26T20:37:15.1929956Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1930178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1930244Z return mod(**inputs) 2025-08-26T20:37:15.1930521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1930601Z outputs = self.mobilebert( 2025-08-26T20:37:15.1930885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1930985Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1931272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1931372Z layer_outputs = layer_module( 2025-08-26T20:37:15.1931679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1931783Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1932096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1932218Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1932527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1932617Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1932623Z 2025-08-26T20:37:15.1932731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1932951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1933021Z return mod(**inputs) 2025-08-26T20:37:15.1933332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1933408Z outputs = self.mobilebert( 2025-08-26T20:37:15.1933720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1933797Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1934103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1934187Z layer_outputs = layer_module( 2025-08-26T20:37:15.1934492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1934604Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1934911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1935055Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1935380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1935502Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1935506Z 2025-08-26T20:37:15.1935621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1935834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1935911Z return mod(**inputs) 2025-08-26T20:37:15.1936210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1936289Z outputs = self.mobilebert( 2025-08-26T20:37:15.1936596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1936674Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1936977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1937053Z layer_outputs = layer_module( 2025-08-26T20:37:15.1937351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1937461Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1937761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1937934Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1938231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1938346Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1938351Z 2025-08-26T20:37:15.1938460Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1938675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1938753Z return mod(**inputs) 2025-08-26T20:37:15.1939050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1939131Z outputs = self.mobilebert( 2025-08-26T20:37:15.1939428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1939506Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1939813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1939889Z layer_outputs = layer_module( 2025-08-26T20:37:15.1940194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1940295Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1940595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1940727Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1941022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1941162Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1941457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1941564Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1941567Z 2025-08-26T20:37:15.1941697Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1941908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1942012Z return mod(**inputs) 2025-08-26T20:37:15.1942291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1942371Z outputs = self.mobilebert( 2025-08-26T20:37:15.1942648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1942729Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1943009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1943084Z layer_outputs = layer_module( 2025-08-26T20:37:15.1943370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1943464Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1943752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1943865Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1944144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1944234Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1944254Z 2025-08-26T20:37:15.1944358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1944568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1944654Z return mod(**inputs) 2025-08-26T20:37:15.1944960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1945036Z outputs = self.mobilebert( 2025-08-26T20:37:15.1945332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1945416Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1945716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1945799Z layer_outputs = layer_module( 2025-08-26T20:37:15.1946105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1946208Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1946527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1946647Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1946954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1947074Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1947077Z 2025-08-26T20:37:15.1947192Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1947404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1947474Z return mod(**inputs) 2025-08-26T20:37:15.1947774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1947853Z outputs = self.mobilebert( 2025-08-26T20:37:15.1948152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1948247Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1948561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1948646Z layer_outputs = layer_module( 2025-08-26T20:37:15.1948955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1949055Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1949344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1949482Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1949762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1949849Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1949854Z 2025-08-26T20:37:15.1949964Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1950162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1950234Z return mod(**inputs) 2025-08-26T20:37:15.1950515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1950587Z outputs = self.mobilebert( 2025-08-26T20:37:15.1950878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1950975Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1951282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1951377Z layer_outputs = layer_module( 2025-08-26T20:37:15.1951690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1951792Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1952094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1952234Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1952535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1952674Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1952978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1953078Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1953089Z 2025-08-26T20:37:15.1953202Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1953415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1953494Z return mod(**inputs) 2025-08-26T20:37:15.1953797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1953882Z outputs = self.mobilebert( 2025-08-26T20:37:15.1954185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1954264Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1954574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1954652Z layer_outputs = layer_module( 2025-08-26T20:37:15.1954983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1955085Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1955397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1955528Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1955820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1955917Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1955922Z 2025-08-26T20:37:15.1956030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1956247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1956322Z return mod(**inputs) 2025-08-26T20:37:15.1956625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1956712Z outputs = self.mobilebert( 2025-08-26T20:37:15.1957021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1957106Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1957410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1957487Z layer_outputs = layer_module( 2025-08-26T20:37:15.1957802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1957925Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1958244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:37:15.1958384Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:37:15.1958703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1958825Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1958829Z 2025-08-26T20:37:15.1958942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1959165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1959236Z return mod(**inputs) 2025-08-26T20:37:15.1959622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1959710Z outputs = self.mobilebert( 2025-08-26T20:37:15.1960014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1960106Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1960414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1960501Z layer_outputs = layer_module( 2025-08-26T20:37:15.1960806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1960919Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1961225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1961363Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1961674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:37:15.1961770Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1961774Z 2025-08-26T20:37:15.1961916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1962166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1962241Z return mod(**inputs) 2025-08-26T20:37:15.1962554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1962632Z outputs = self.mobilebert( 2025-08-26T20:37:15.1962948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1963029Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1963340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1963420Z layer_outputs = layer_module( 2025-08-26T20:37:15.1963725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:37:15.1963833Z attention_output = ffn_module(attention_output) 2025-08-26T20:37:15.1964138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:37:15.1964278Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:37:15.1964582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:37:15.1964713Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1965089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1965208Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1965211Z 2025-08-26T20:37:15.1965331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1965548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1965630Z return mod(**inputs) 2025-08-26T20:37:15.1965933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1966012Z outputs = self.mobilebert( 2025-08-26T20:37:15.1966323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1966402Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1966722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1966802Z layer_outputs = layer_module( 2025-08-26T20:37:15.1967107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1967246Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1967553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:37:15.1967653Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1967657Z 2025-08-26T20:37:15.1967768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1967992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1968066Z return mod(**inputs) 2025-08-26T20:37:15.1968370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1968458Z outputs = self.mobilebert( 2025-08-26T20:37:15.1968789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1968877Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1969197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1969274Z layer_outputs = layer_module( 2025-08-26T20:37:15.1969579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:37:15.1969706Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:37:15.1970008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:37:15.1970128Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:37:15.1970132Z 2025-08-26T20:37:15.1970251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1970461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1970532Z return mod(**inputs) 2025-08-26T20:37:15.1970836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1970912Z outputs = self.mobilebert( 2025-08-26T20:37:15.1971212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1971289Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1971583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1971685Z layer_outputs = layer_module( 2025-08-26T20:37:15.1971983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1972177Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1972478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:37:15.1972587Z layer_output = self.dense(intermediate_states) 2025-08-26T20:37:15.1972592Z 2025-08-26T20:37:15.1972701Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1972922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1973002Z return mod(**inputs) 2025-08-26T20:37:15.1973299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1973386Z outputs = self.mobilebert( 2025-08-26T20:37:15.1973684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1973763Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1974077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1974151Z layer_outputs = layer_module( 2025-08-26T20:37:15.1974437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1974595Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1974882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:37:15.1975008Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:37:15.1975297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1975405Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1975425Z 2025-08-26T20:37:15.1975536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1975770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1975843Z return mod(**inputs) 2025-08-26T20:37:15.1976139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1976223Z outputs = self.mobilebert( 2025-08-26T20:37:15.1976517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1976604Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1976900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1976984Z layer_outputs = layer_module( 2025-08-26T20:37:15.1977282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1977449Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1977763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1977886Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1978171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:37:15.1978277Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:37:15.1978281Z 2025-08-26T20:37:15.1978392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1978607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1978673Z return mod(**inputs) 2025-08-26T20:37:15.1978960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-26T20:37:15.1979032Z outputs = self.mobilebert( 2025-08-26T20:37:15.1979319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:37:15.1979394Z encoder_outputs = self.encoder( 2025-08-26T20:37:15.1979671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:37:15.1979751Z layer_outputs = layer_module( 2025-08-26T20:37:15.1980029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:37:15.1980193Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:37:15.1980473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:37:15.1980599Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:37:15.1980894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:37:15.1981022Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:37:15.1981326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:37:15.1981427Z return input_tensor * self.weight + self.bias 2025-08-26T20:37:15.1981431Z 2025-08-26T20:37:15.1981545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1981756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1981828Z return mod(**inputs) 2025-08-26T20:37:15.1982149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-26T20:37:15.1982268Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:37:15.1982574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-26T20:37:15.1982693Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:37:15.1982997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-08-26T20:37:15.1983094Z hidden_states = self.transform(hidden_states) 2025-08-26T20:37:15.1983390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-08-26T20:37:15.1983491Z hidden_states = self.dense(hidden_states) 2025-08-26T20:37:15.1983495Z 2025-08-26T20:37:15.1983606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1983822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1983894Z return mod(**inputs) 2025-08-26T20:37:15.1984190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-26T20:37:15.1984292Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:37:15.1984588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-26T20:37:15.1984733Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:37:15.1985032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-08-26T20:37:15.1985289Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-08-26T20:37:15.1985295Z 2025-08-26T20:37:15.1985405Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1985616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1985697Z return mod(**inputs) 2025-08-26T20:37:15.1985995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-26T20:37:15.1986100Z prediction_scores = self.cls(sequence_output) 2025-08-26T20:37:15.1986397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-26T20:37:15.1986517Z prediction_scores = self.predictions(sequence_output) 2025-08-26T20:37:15.1986823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-08-26T20:37:15.1986911Z hidden_states += self.decoder.bias 2025-08-26T20:37:15.1986917Z 2025-08-26T20:37:15.1987035Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:37:15.1987246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:37:15.1987323Z return mod(**inputs) 2025-08-26T20:37:15.1987619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-08-26T20:37:15.1987826Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:37:15.1987830Z 2025-08-26T20:37:28.7071113Z Compilation time (from dynamo_timed): 40.042420595 2025-08-26T20:37:28.7075984Z pass 2025-08-26T20:37:28.7076416Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:37:28.7077641Z TIMING: _recursive_pre_grad_passes:0.02347 _recursive_joint_graph_passes:1.88391 _recursive_post_grad_passes:0.22153 async_compile.wait:0.79393 code_gen:10.29403 inductor_compile:14.68657 backend_compile:28.11056 gc:0.00079 entire_frame_compile:40.04242 total_wall_time:40.04242 2025-08-26T20:37:28.7078848Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:56770 | FakeTensor.__torch_dispatch__:15340 | ProxyTorchDispatchMode.__torch_dispatch__:21632 2025-08-26T20:37:28.7079440Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-08-26T20:37:35.1528558Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:37:35.1531335Z from pkg_resources import resource_filename 2025-08-26T20:37:35.7723838Z 2025-08-26T20:37:36.3884323Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:37:36.3884820Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:37:36.3957844Z cpu eval MobileBertForQuestionAnswering 2025-08-26T20:37:36.6068600Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:37:36.7451204Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:37:36.8765931Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:05.0611865Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0612576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0618839Z return mod(**inputs) 2025-08-26T20:38:05.0619472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0625480Z outputs = self.mobilebert( 2025-08-26T20:38:05.0626239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-26T20:38:05.0626873Z embedding_output = self.embeddings( 2025-08-26T20:38:05.0627816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-08-26T20:38:05.0628496Z inputs_embeds = torch.cat( 2025-08-26T20:38:05.0628637Z 2025-08-26T20:38:05.0628772Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0629173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0629546Z return mod(**inputs) 2025-08-26T20:38:05.0630003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0630476Z outputs = self.mobilebert( 2025-08-26T20:38:05.0630934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-26T20:38:05.0631397Z embedding_output = self.embeddings( 2025-08-26T20:38:05.0631862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-08-26T20:38:05.0632369Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-08-26T20:38:05.0632561Z 2025-08-26T20:38:05.0632686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0633091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0633453Z return mod(**inputs) 2025-08-26T20:38:05.0633901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0634374Z outputs = self.mobilebert( 2025-08-26T20:38:05.0634827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-26T20:38:05.0636432Z embedding_output = self.embeddings( 2025-08-26T20:38:05.0636985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-08-26T20:38:05.0637461Z embeddings = self.LayerNorm(embeddings) 2025-08-26T20:38:05.0637932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0638400Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0638565Z 2025-08-26T20:38:05.0638682Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0639106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0639692Z return mod(**inputs) 2025-08-26T20:38:05.0640163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0640630Z outputs = self.mobilebert( 2025-08-26T20:38:05.0641068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0641541Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0642005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0642461Z layer_outputs = layer_module( 2025-08-26T20:38:05.0642899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.0643479Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.0644031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.0644540Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.0645029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.0645490Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.0645665Z 2025-08-26T20:38:05.0645782Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0646177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0646529Z return mod(**inputs) 2025-08-26T20:38:05.0646954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0647395Z outputs = self.mobilebert( 2025-08-26T20:38:05.0647853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0648297Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0648738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0649187Z layer_outputs = layer_module( 2025-08-26T20:38:05.0649590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.0650088Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.0650598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.0651058Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.0651539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.0651978Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.0652464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0652944Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0653105Z 2025-08-26T20:38:05.0653221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0653600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0653936Z return mod(**inputs) 2025-08-26T20:38:05.0654361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0654807Z outputs = self.mobilebert( 2025-08-26T20:38:05.0655231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0655671Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0656115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0656537Z layer_outputs = layer_module( 2025-08-26T20:38:05.0656977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0657417Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0657849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.0658288Z self_outputs = self.self( 2025-08-26T20:38:05.0658739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.0659190Z self.query(query_tensor) 2025-08-26T20:38:05.0659346Z 2025-08-26T20:38:05.0659463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0659837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0660191Z return mod(**inputs) 2025-08-26T20:38:05.0660597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0661022Z outputs = self.mobilebert( 2025-08-26T20:38:05.0661426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0661852Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0662275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0662707Z layer_outputs = layer_module( 2025-08-26T20:38:05.0663131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0663579Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0664007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.0664438Z self_outputs = self.self( 2025-08-26T20:38:05.0664883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.0665320Z self.key(key_tensor) 2025-08-26T20:38:05.0665434Z 2025-08-26T20:38:05.0665546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0665935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0666349Z return mod(**inputs) 2025-08-26T20:38:05.0666743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0667156Z outputs = self.mobilebert( 2025-08-26T20:38:05.0667584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0668044Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0668480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0668917Z layer_outputs = layer_module( 2025-08-26T20:38:05.0669343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0669774Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0670208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.0670627Z self_outputs = self.self( 2025-08-26T20:38:05.0671035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.0671473Z self.value(value_tensor) 2025-08-26T20:38:05.0671605Z 2025-08-26T20:38:05.0671697Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.0671929Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.0672184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0672559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0672887Z return mod(**inputs) 2025-08-26T20:38:05.0673308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0673785Z outputs = self.mobilebert( 2025-08-26T20:38:05.0674215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0674680Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0675134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0675595Z layer_outputs = layer_module( 2025-08-26T20:38:05.0676042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0676514Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0676970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.0677479Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.0677993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.0678464Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0678619Z 2025-08-26T20:38:05.0678742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0679130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0679569Z return mod(**inputs) 2025-08-26T20:38:05.0680008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0680475Z outputs = self.mobilebert( 2025-08-26T20:38:05.0680913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0681353Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0681795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0682247Z layer_outputs = layer_module( 2025-08-26T20:38:05.0682700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.0683279Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.0683865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.0684370Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.0684871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.0685335Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.0685490Z 2025-08-26T20:38:05.0685604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0685995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0686353Z return mod(**inputs) 2025-08-26T20:38:05.0686787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0687245Z outputs = self.mobilebert( 2025-08-26T20:38:05.0687679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0688135Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0688586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0689037Z layer_outputs = layer_module( 2025-08-26T20:38:05.0689473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0689986Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0690453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.0690980Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.0691484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.0691986Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0692506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0692985Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0693147Z 2025-08-26T20:38:05.0693270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0693669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0694037Z return mod(**inputs) 2025-08-26T20:38:05.0694466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0694917Z outputs = self.mobilebert( 2025-08-26T20:38:05.0695349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0695793Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0696403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0696858Z layer_outputs = layer_module( 2025-08-26T20:38:05.0697306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0697781Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0698250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0698737Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0699284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0699783Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0699938Z 2025-08-26T20:38:05.0700059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0700436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0700786Z return mod(**inputs) 2025-08-26T20:38:05.0701220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0701665Z outputs = self.mobilebert( 2025-08-26T20:38:05.0702168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0702611Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0703050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0703490Z layer_outputs = layer_module( 2025-08-26T20:38:05.0703929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0704393Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0705755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0706350Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0707276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0707935Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0708149Z 2025-08-26T20:38:05.0708289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0708773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0709143Z return mod(**inputs) 2025-08-26T20:38:05.0709675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0710140Z outputs = self.mobilebert( 2025-08-26T20:38:05.0710580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0711041Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0711592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0712050Z layer_outputs = layer_module( 2025-08-26T20:38:05.0712692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0713279Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0713855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0714382Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0714982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.0715555Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0715718Z 2025-08-26T20:38:05.0715846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0716321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0716688Z return mod(**inputs) 2025-08-26T20:38:05.0717333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0717807Z outputs = self.mobilebert( 2025-08-26T20:38:05.0718387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0718874Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0719397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0720090Z layer_outputs = layer_module( 2025-08-26T20:38:05.0720537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0721133Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0721737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0722257Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0722764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.0723260Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0723765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0724276Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0724475Z 2025-08-26T20:38:05.0724598Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0725038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0725473Z return mod(**inputs) 2025-08-26T20:38:05.0725971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0726455Z outputs = self.mobilebert( 2025-08-26T20:38:05.0726887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0727340Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0727774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0728235Z layer_outputs = layer_module( 2025-08-26T20:38:05.0728680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0729155Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0729640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0730191Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0730736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0731212Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0731369Z 2025-08-26T20:38:05.0731492Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0731877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0732231Z return mod(**inputs) 2025-08-26T20:38:05.0732660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0733107Z outputs = self.mobilebert( 2025-08-26T20:38:05.0733532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0733973Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0734539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0734983Z layer_outputs = layer_module( 2025-08-26T20:38:05.0735519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0735985Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0736537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0737065Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0737555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0738112Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0738293Z 2025-08-26T20:38:05.0738415Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0738883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0739242Z return mod(**inputs) 2025-08-26T20:38:05.0739749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0740207Z outputs = self.mobilebert( 2025-08-26T20:38:05.0740630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0741073Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0741617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0742059Z layer_outputs = layer_module( 2025-08-26T20:38:05.0742604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0743126Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0743613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0744210Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0744802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.0745268Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0745475Z 2025-08-26T20:38:05.0745617Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0746005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0746450Z return mod(**inputs) 2025-08-26T20:38:05.0746878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0747419Z outputs = self.mobilebert( 2025-08-26T20:38:05.0747841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0748288Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0748787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0749326Z layer_outputs = layer_module( 2025-08-26T20:38:05.0749753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0750314Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0750875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0751480Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0752109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.0752608Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0753195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0753753Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0753916Z 2025-08-26T20:38:05.0754044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0754530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0754881Z return mod(**inputs) 2025-08-26T20:38:05.0755488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0755950Z outputs = self.mobilebert( 2025-08-26T20:38:05.0756488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0757036Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0757502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0758038Z layer_outputs = layer_module( 2025-08-26T20:38:05.0758555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0759067Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0760020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0760672Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0761277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0761794Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0761992Z 2025-08-26T20:38:05.0762116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0762519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0762957Z return mod(**inputs) 2025-08-26T20:38:05.0763476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0763944Z outputs = self.mobilebert( 2025-08-26T20:38:05.0764535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0764997Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0765596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0766060Z layer_outputs = layer_module( 2025-08-26T20:38:05.0766618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0767174Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0767653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0768247Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0768857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0769371Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0769637Z 2025-08-26T20:38:05.0769759Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0770200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0770620Z return mod(**inputs) 2025-08-26T20:38:05.0771181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0771649Z outputs = self.mobilebert( 2025-08-26T20:38:05.0772216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0772709Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0773212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0773771Z layer_outputs = layer_module( 2025-08-26T20:38:05.0774273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0774847Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0775426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0775990Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0776504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.0777138Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0777290Z 2025-08-26T20:38:05.0777430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0777903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0778330Z return mod(**inputs) 2025-08-26T20:38:05.0778778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0779323Z outputs = self.mobilebert( 2025-08-26T20:38:05.0779845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0780361Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0780839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0781387Z layer_outputs = layer_module( 2025-08-26T20:38:05.0781829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0782390Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0782956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0783503Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0784058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.0784723Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0785312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0785796Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0785986Z 2025-08-26T20:38:05.0786150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0786558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0787007Z return mod(**inputs) 2025-08-26T20:38:05.0787431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0788011Z outputs = self.mobilebert( 2025-08-26T20:38:05.0788573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0789063Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0789587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0790032Z layer_outputs = layer_module( 2025-08-26T20:38:05.0790472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.0790972Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.0791473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0791940Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0792116Z 2025-08-26T20:38:05.0792282Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0792686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0793139Z return mod(**inputs) 2025-08-26T20:38:05.0793572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0794131Z outputs = self.mobilebert( 2025-08-26T20:38:05.0794664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0795145Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0795671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0796373Z layer_outputs = layer_module( 2025-08-26T20:38:05.0796842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.0797377Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.0797942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0798560Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0798742Z 2025-08-26T20:38:05.0798873Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0799414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0799981Z return mod(**inputs) 2025-08-26T20:38:05.0800502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0801048Z outputs = self.mobilebert( 2025-08-26T20:38:05.0801527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0802031Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0802584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0803132Z layer_outputs = layer_module( 2025-08-26T20:38:05.0803674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.0804321Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.0804979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.0805455Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.0805626Z 2025-08-26T20:38:05.0805748Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0806212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0806654Z return mod(**inputs) 2025-08-26T20:38:05.0807123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0807695Z outputs = self.mobilebert( 2025-08-26T20:38:05.0808136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0808582Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0809035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0809510Z layer_outputs = layer_module( 2025-08-26T20:38:05.0809964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.0810592Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.0811206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.0811746Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.0812328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0812790Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0813012Z 2025-08-26T20:38:05.0813133Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0813514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0813905Z return mod(**inputs) 2025-08-26T20:38:05.0814328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0814835Z outputs = self.mobilebert( 2025-08-26T20:38:05.0815275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0815821Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0816264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0816713Z layer_outputs = layer_module( 2025-08-26T20:38:05.0817150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.0817676Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.0818219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.0818719Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.0819226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.0819720Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0819905Z 2025-08-26T20:38:05.0820019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0820418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0820846Z return mod(**inputs) 2025-08-26T20:38:05.0821276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0821736Z outputs = self.mobilebert( 2025-08-26T20:38:05.0822187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0822719Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0823278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0823719Z layer_outputs = layer_module( 2025-08-26T20:38:05.0824165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.0824779Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.0825416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.0825912Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.0826398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.0826895Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0827402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0827875Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0828040Z 2025-08-26T20:38:05.0828152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0828615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0828969Z return mod(**inputs) 2025-08-26T20:38:05.0829469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0830035Z outputs = self.mobilebert( 2025-08-26T20:38:05.0830507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0831048Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0831569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0832129Z layer_outputs = layer_module( 2025-08-26T20:38:05.0832566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.0833110Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.0833664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.0834158Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.0834651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.0835194Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.0835352Z 2025-08-26T20:38:05.0835475Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0835973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0836338Z return mod(**inputs) 2025-08-26T20:38:05.0836869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0837336Z outputs = self.mobilebert( 2025-08-26T20:38:05.0837774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0838229Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0838675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0839228Z layer_outputs = layer_module( 2025-08-26T20:38:05.0839939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.0840621Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.0841289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.0841780Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.0842315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.0842825Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.0843378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0843921Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0844080Z 2025-08-26T20:38:05.0844196Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0844671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0845105Z return mod(**inputs) 2025-08-26T20:38:05.0845528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0846104Z outputs = self.mobilebert( 2025-08-26T20:38:05.0846585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0847158Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0847650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0848227Z layer_outputs = layer_module( 2025-08-26T20:38:05.0848733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0849320Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0849854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.0850338Z self_outputs = self.self( 2025-08-26T20:38:05.0850841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.0851321Z self.query(query_tensor) 2025-08-26T20:38:05.0851481Z 2025-08-26T20:38:05.0851594Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0851998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0852419Z return mod(**inputs) 2025-08-26T20:38:05.0852902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0853357Z outputs = self.mobilebert( 2025-08-26T20:38:05.0853902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0854397Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0854876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0855415Z layer_outputs = layer_module( 2025-08-26T20:38:05.0855845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0856438Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0856929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.0857491Z self_outputs = self.self( 2025-08-26T20:38:05.0857951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.0858501Z self.key(key_tensor) 2025-08-26T20:38:05.0858627Z 2025-08-26T20:38:05.0858756Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0859220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0859609Z return mod(**inputs) 2025-08-26T20:38:05.0860069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0860619Z outputs = self.mobilebert( 2025-08-26T20:38:05.0861053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0861598Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0862124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0862581Z layer_outputs = layer_module( 2025-08-26T20:38:05.0863101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0863619Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0864082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.0864622Z self_outputs = self.self( 2025-08-26T20:38:05.0865134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.0865613Z self.value(value_tensor) 2025-08-26T20:38:05.0865758Z 2025-08-26T20:38:05.0865858Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.0866088Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.0866338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0866738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0867108Z return mod(**inputs) 2025-08-26T20:38:05.0867525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0867970Z outputs = self.mobilebert( 2025-08-26T20:38:05.0868406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0868941Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0869475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0869945Z layer_outputs = layer_module( 2025-08-26T20:38:05.0870380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0870858Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0871332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.0871849Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.0872362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.0872837Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0873001Z 2025-08-26T20:38:05.0873117Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0873521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0873896Z return mod(**inputs) 2025-08-26T20:38:05.0874345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0874816Z outputs = self.mobilebert( 2025-08-26T20:38:05.0875279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0875754Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0876208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0876665Z layer_outputs = layer_module( 2025-08-26T20:38:05.0877117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.0877671Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.0878228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.0878737Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.0879240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.0879795Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.0879965Z 2025-08-26T20:38:05.0880081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0880482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0880861Z return mod(**inputs) 2025-08-26T20:38:05.0881318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0881773Z outputs = self.mobilebert( 2025-08-26T20:38:05.0882268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0882723Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0883177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0883630Z layer_outputs = layer_module( 2025-08-26T20:38:05.0884080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.0884545Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.0885009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.0885514Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.0886027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.0886546Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0887046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0887505Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0887667Z 2025-08-26T20:38:05.0887779Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0888171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0888522Z return mod(**inputs) 2025-08-26T20:38:05.0888941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0889391Z outputs = self.mobilebert( 2025-08-26T20:38:05.0889811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0890248Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0890711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0891171Z layer_outputs = layer_module( 2025-08-26T20:38:05.0891596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0892062Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0892527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0893011Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0893492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0893943Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0894102Z 2025-08-26T20:38:05.0894215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0894598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0894954Z return mod(**inputs) 2025-08-26T20:38:05.0895371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0895809Z outputs = self.mobilebert( 2025-08-26T20:38:05.0896396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0896924Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0897372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0897860Z layer_outputs = layer_module( 2025-08-26T20:38:05.0898303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0898781Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0899253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0899690Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0900130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0900654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0900844Z 2025-08-26T20:38:05.0900957Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0901342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0901698Z return mod(**inputs) 2025-08-26T20:38:05.0902114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0902562Z outputs = self.mobilebert( 2025-08-26T20:38:05.0902967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0903385Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0903806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0904215Z layer_outputs = layer_module( 2025-08-26T20:38:05.0904627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0905077Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0905507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0906003Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0906489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.0906919Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0907065Z 2025-08-26T20:38:05.0907169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0907525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0907836Z return mod(**inputs) 2025-08-26T20:38:05.0908235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0908656Z outputs = self.mobilebert( 2025-08-26T20:38:05.0909062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0909474Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0909877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0910286Z layer_outputs = layer_module( 2025-08-26T20:38:05.0910690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0911126Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0911559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0912041Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0912506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.0912991Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0913453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0913908Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0914066Z 2025-08-26T20:38:05.0914177Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0914559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0914905Z return mod(**inputs) 2025-08-26T20:38:05.0916826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0917271Z outputs = self.mobilebert( 2025-08-26T20:38:05.0917689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0918130Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0918565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0919006Z layer_outputs = layer_module( 2025-08-26T20:38:05.0919428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0920067Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0920545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0921013Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0921475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0921908Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0922067Z 2025-08-26T20:38:05.0922211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0922583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0922938Z return mod(**inputs) 2025-08-26T20:38:05.0923339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0923751Z outputs = self.mobilebert( 2025-08-26T20:38:05.0924155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0924572Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0924984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0925391Z layer_outputs = layer_module( 2025-08-26T20:38:05.0925811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0926250Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0926695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0927152Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0927600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0928063Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0928265Z 2025-08-26T20:38:05.0928371Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0928740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0929093Z return mod(**inputs) 2025-08-26T20:38:05.0929477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0929896Z outputs = self.mobilebert( 2025-08-26T20:38:05.0930299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0930712Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0931120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0931524Z layer_outputs = layer_module( 2025-08-26T20:38:05.0931927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0932361Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0932811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0933305Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0933793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.0934246Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0934400Z 2025-08-26T20:38:05.0934512Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0934890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0935229Z return mod(**inputs) 2025-08-26T20:38:05.0935648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0936091Z outputs = self.mobilebert( 2025-08-26T20:38:05.0936519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0936981Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0937430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0937873Z layer_outputs = layer_module( 2025-08-26T20:38:05.0938324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0938802Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0939283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0939788Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0940333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.0940841Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0941354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0941828Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0941993Z 2025-08-26T20:38:05.0942106Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0942506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0942880Z return mod(**inputs) 2025-08-26T20:38:05.0943312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0943793Z outputs = self.mobilebert( 2025-08-26T20:38:05.0944229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0944688Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0945113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0945540Z layer_outputs = layer_module( 2025-08-26T20:38:05.0945946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0946397Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0946845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0947308Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0947769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0948200Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0948352Z 2025-08-26T20:38:05.0948463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0948831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0949165Z return mod(**inputs) 2025-08-26T20:38:05.0949558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0949988Z outputs = self.mobilebert( 2025-08-26T20:38:05.0950399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0950836Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0951276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0951719Z layer_outputs = layer_module( 2025-08-26T20:38:05.0952182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0952654Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0953138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.0953624Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.0954092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0954572Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0954758Z 2025-08-26T20:38:05.0954874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0955262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0955613Z return mod(**inputs) 2025-08-26T20:38:05.0956027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0956473Z outputs = self.mobilebert( 2025-08-26T20:38:05.0956905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0957344Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0957770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0958213Z layer_outputs = layer_module( 2025-08-26T20:38:05.0958647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0959136Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0959729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0960247Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0960766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.0961236Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0961380Z 2025-08-26T20:38:05.0961500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0961891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0962245Z return mod(**inputs) 2025-08-26T20:38:05.0962678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0963139Z outputs = self.mobilebert( 2025-08-26T20:38:05.0963581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0964041Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0964481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0964944Z layer_outputs = layer_module( 2025-08-26T20:38:05.0965397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.0965886Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.0966359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.0966866Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.0967381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.0967883Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.0968424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0968925Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0969101Z 2025-08-26T20:38:05.0969217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0969623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0969990Z return mod(**inputs) 2025-08-26T20:38:05.0970437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0970891Z outputs = self.mobilebert( 2025-08-26T20:38:05.0971331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0971791Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0972212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0972642Z layer_outputs = layer_module( 2025-08-26T20:38:05.0973055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.0973531Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.0974007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.0974464Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.0974610Z 2025-08-26T20:38:05.0974723Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0975102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0975480Z return mod(**inputs) 2025-08-26T20:38:05.0975904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0976356Z outputs = self.mobilebert( 2025-08-26T20:38:05.0976751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0977170Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0977580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0978000Z layer_outputs = layer_module( 2025-08-26T20:38:05.0978412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.0978866Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.0979331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.0979791Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.0979969Z 2025-08-26T20:38:05.0980091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0980483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0980804Z return mod(**inputs) 2025-08-26T20:38:05.0981211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0981661Z outputs = self.mobilebert( 2025-08-26T20:38:05.0982078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0982489Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0982923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0983339Z layer_outputs = layer_module( 2025-08-26T20:38:05.0983760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.0984281Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.0984822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.0985295Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.0985473Z 2025-08-26T20:38:05.0985578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0985941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0986269Z return mod(**inputs) 2025-08-26T20:38:05.0986662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0987082Z outputs = self.mobilebert( 2025-08-26T20:38:05.0987495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0987916Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0988323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0988744Z layer_outputs = layer_module( 2025-08-26T20:38:05.0989158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.0989688Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.0990215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.0990681Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.0991147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.0991580Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.0991739Z 2025-08-26T20:38:05.0991846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0992207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0992529Z return mod(**inputs) 2025-08-26T20:38:05.0992927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.0993346Z outputs = self.mobilebert( 2025-08-26T20:38:05.0993778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.0994229Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.0994666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.0995117Z layer_outputs = layer_module( 2025-08-26T20:38:05.0995557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.0996099Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.0996797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.0997302Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.0997819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.0998342Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.0998501Z 2025-08-26T20:38:05.0998620Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.0999055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.0999415Z return mod(**inputs) 2025-08-26T20:38:05.0999972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1000433Z outputs = self.mobilebert( 2025-08-26T20:38:05.1000874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1001324Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1001744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1002166Z layer_outputs = layer_module( 2025-08-26T20:38:05.1002581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1003083Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1003587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1004058Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1004528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1005041Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1005507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1005965Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1006124Z 2025-08-26T20:38:05.1006233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1006601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1006930Z return mod(**inputs) 2025-08-26T20:38:05.1007340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1007781Z outputs = self.mobilebert( 2025-08-26T20:38:05.1008218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1008662Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1009102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1009546Z layer_outputs = layer_module( 2025-08-26T20:38:05.1009991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1010522Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1011072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1011565Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1012040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1012496Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1012651Z 2025-08-26T20:38:05.1012763Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1013148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1013555Z return mod(**inputs) 2025-08-26T20:38:05.1013994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1014485Z outputs = self.mobilebert( 2025-08-26T20:38:05.1014912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1015357Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1015794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1016231Z layer_outputs = layer_module( 2025-08-26T20:38:05.1016667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1017203Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1017742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1018223Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1018697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1019161Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1019612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1020118Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1020279Z 2025-08-26T20:38:05.1020399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1020778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1021156Z return mod(**inputs) 2025-08-26T20:38:05.1021578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1022035Z outputs = self.mobilebert( 2025-08-26T20:38:05.1022457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1022899Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1023333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1023767Z layer_outputs = layer_module( 2025-08-26T20:38:05.1024200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1024647Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1025100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1025544Z self_outputs = self.self( 2025-08-26T20:38:05.1025948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1026370Z self.query(query_tensor) 2025-08-26T20:38:05.1026483Z 2025-08-26T20:38:05.1026585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1026939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1027256Z return mod(**inputs) 2025-08-26T20:38:05.1027639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1028032Z outputs = self.mobilebert( 2025-08-26T20:38:05.1028420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1028874Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1029330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1029783Z layer_outputs = layer_module( 2025-08-26T20:38:05.1030186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1030618Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1031045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1031462Z self_outputs = self.self( 2025-08-26T20:38:05.1031859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1032266Z self.key(key_tensor) 2025-08-26T20:38:05.1032378Z 2025-08-26T20:38:05.1032484Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1032849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1033178Z return mod(**inputs) 2025-08-26T20:38:05.1033573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1034014Z outputs = self.mobilebert( 2025-08-26T20:38:05.1034441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1034910Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1035346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1035801Z layer_outputs = layer_module( 2025-08-26T20:38:05.1036234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1036687Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1037143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1037585Z self_outputs = self.self( 2025-08-26T20:38:05.1038004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1038443Z self.value(value_tensor) 2025-08-26T20:38:05.1038566Z 2025-08-26T20:38:05.1038663Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1038901Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1039151Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1039625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1040003Z return mod(**inputs) 2025-08-26T20:38:05.1040443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1040913Z outputs = self.mobilebert( 2025-08-26T20:38:05.1041353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1041808Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1042258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1042719Z layer_outputs = layer_module( 2025-08-26T20:38:05.1043167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1043641Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1044149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1044660Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1045197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1045662Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1045826Z 2025-08-26T20:38:05.1045939Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1046334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1046692Z return mod(**inputs) 2025-08-26T20:38:05.1047123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1047572Z outputs = self.mobilebert( 2025-08-26T20:38:05.1048014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1048470Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1048928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1049376Z layer_outputs = layer_module( 2025-08-26T20:38:05.1049836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1050398Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1050966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1051494Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1052016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1052489Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1052637Z 2025-08-26T20:38:05.1052743Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1053109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1053442Z return mod(**inputs) 2025-08-26T20:38:05.1053847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1054299Z outputs = self.mobilebert( 2025-08-26T20:38:05.1054740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1055187Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1055624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1056061Z layer_outputs = layer_module( 2025-08-26T20:38:05.1056496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1056952Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1057408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1057888Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1058343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1058817Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1059286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1059730Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1059932Z 2025-08-26T20:38:05.1060047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1060428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1060764Z return mod(**inputs) 2025-08-26T20:38:05.1061173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1061621Z outputs = self.mobilebert( 2025-08-26T20:38:05.1062042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1062489Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1062926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1063364Z layer_outputs = layer_module( 2025-08-26T20:38:05.1063774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1064207Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1064672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1065149Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1065634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1066119Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1066270Z 2025-08-26T20:38:05.1066381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1066763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1067136Z return mod(**inputs) 2025-08-26T20:38:05.1067555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1068000Z outputs = self.mobilebert( 2025-08-26T20:38:05.1068429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1068868Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1069303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1069741Z layer_outputs = layer_module( 2025-08-26T20:38:05.1070166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1070634Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1071105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1071583Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1072061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1072541Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1072725Z 2025-08-26T20:38:05.1072837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1073226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1073576Z return mod(**inputs) 2025-08-26T20:38:05.1073987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1074433Z outputs = self.mobilebert( 2025-08-26T20:38:05.1074887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1075332Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1075791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1076227Z layer_outputs = layer_module( 2025-08-26T20:38:05.1076657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1077120Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1077587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1078097Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1078598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1079068Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1079231Z 2025-08-26T20:38:05.1079344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1079836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1080204Z return mod(**inputs) 2025-08-26T20:38:05.1080635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1081104Z outputs = self.mobilebert( 2025-08-26T20:38:05.1081533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1082000Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1082424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1082891Z layer_outputs = layer_module( 2025-08-26T20:38:05.1083331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1083812Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1084286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1084789Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1085286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1085784Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1086283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1086742Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1086901Z 2025-08-26T20:38:05.1087014Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1087403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1087753Z return mod(**inputs) 2025-08-26T20:38:05.1088172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1088614Z outputs = self.mobilebert( 2025-08-26T20:38:05.1089041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1089482Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1089922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1090361Z layer_outputs = layer_module( 2025-08-26T20:38:05.1090825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1091310Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1091784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1091914Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1092220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1092313Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1092317Z 2025-08-26T20:38:05.1092433Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1092645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1092724Z return mod(**inputs) 2025-08-26T20:38:05.1093029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1093109Z outputs = self.mobilebert( 2025-08-26T20:38:05.1093410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1093489Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1093798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1093874Z layer_outputs = layer_module( 2025-08-26T20:38:05.1094198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1094307Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1094632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1094758Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1095057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1095188Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1095193Z 2025-08-26T20:38:05.1095304Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1095514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1095595Z return mod(**inputs) 2025-08-26T20:38:05.1095898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1095981Z outputs = self.mobilebert( 2025-08-26T20:38:05.1096459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1096561Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1096865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1096946Z layer_outputs = layer_module( 2025-08-26T20:38:05.1097253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1097364Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1097678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1097810Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1098093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1098248Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1098253Z 2025-08-26T20:38:05.1098359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1098597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1098668Z return mod(**inputs) 2025-08-26T20:38:05.1098964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1099039Z outputs = self.mobilebert( 2025-08-26T20:38:05.1099323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1099408Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1099691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1099776Z layer_outputs = layer_module( 2025-08-26T20:38:05.1100058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1100158Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1100450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1100576Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1100875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1101033Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1101338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1101484Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1101491Z 2025-08-26T20:38:05.1101602Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1101825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1101896Z return mod(**inputs) 2025-08-26T20:38:05.1102208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1102286Z outputs = self.mobilebert( 2025-08-26T20:38:05.1102587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1102687Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1102968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1103050Z layer_outputs = layer_module( 2025-08-26T20:38:05.1103344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1103451Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1103750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1103869Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1104177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1104268Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1104272Z 2025-08-26T20:38:05.1104389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1104597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1104670Z return mod(**inputs) 2025-08-26T20:38:05.1105001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1105081Z outputs = self.mobilebert( 2025-08-26T20:38:05.1105408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1105487Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1105790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1105866Z layer_outputs = layer_module( 2025-08-26T20:38:05.1106163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1106267Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1106573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1106699Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1106995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1107116Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1107120Z 2025-08-26T20:38:05.1107234Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1107444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1107520Z return mod(**inputs) 2025-08-26T20:38:05.1107841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1107924Z outputs = self.mobilebert( 2025-08-26T20:38:05.1108248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1108326Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1108632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1108706Z layer_outputs = layer_module( 2025-08-26T20:38:05.1109007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1109106Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1109415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1109558Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1109854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1109955Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1109959Z 2025-08-26T20:38:05.1110068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1110287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1110357Z return mod(**inputs) 2025-08-26T20:38:05.1110654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1110739Z outputs = self.mobilebert( 2025-08-26T20:38:05.1111047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1111135Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1111431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1111509Z layer_outputs = layer_module( 2025-08-26T20:38:05.1111842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1111962Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1112268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1112399Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1112713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1112844Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1113137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1113245Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1113249Z 2025-08-26T20:38:05.1113360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1113580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1113650Z return mod(**inputs) 2025-08-26T20:38:05.1113950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1114035Z outputs = self.mobilebert( 2025-08-26T20:38:05.1114340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1114444Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1114745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1114846Z layer_outputs = layer_module( 2025-08-26T20:38:05.1115143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1115276Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1115584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1115675Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1115679Z 2025-08-26T20:38:05.1115793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1116004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1116085Z return mod(**inputs) 2025-08-26T20:38:05.1116377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1116452Z outputs = self.mobilebert( 2025-08-26T20:38:05.1116743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1116822Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1117126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1117201Z layer_outputs = layer_module( 2025-08-26T20:38:05.1117503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1117641Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1117946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1118076Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1118081Z 2025-08-26T20:38:05.1118193Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1118430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1118512Z return mod(**inputs) 2025-08-26T20:38:05.1118841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1118928Z outputs = self.mobilebert( 2025-08-26T20:38:05.1119235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1119321Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1119692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1119775Z layer_outputs = layer_module( 2025-08-26T20:38:05.1120094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1120271Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1120596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1120692Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1120696Z 2025-08-26T20:38:05.1120800Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1121005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1121072Z return mod(**inputs) 2025-08-26T20:38:05.1121396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1121469Z outputs = self.mobilebert( 2025-08-26T20:38:05.1121785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1121860Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1122141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1122222Z layer_outputs = layer_module( 2025-08-26T20:38:05.1122501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1122665Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1122943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1123068Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1123355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1123450Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1123454Z 2025-08-26T20:38:05.1123563Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1123762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1123835Z return mod(**inputs) 2025-08-26T20:38:05.1124115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1124185Z outputs = self.mobilebert( 2025-08-26T20:38:05.1124468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1124543Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1124832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1124906Z layer_outputs = layer_module( 2025-08-26T20:38:05.1125202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1125386Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1125671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1125804Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1126085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1126180Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1126184Z 2025-08-26T20:38:05.1126287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1126485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1126562Z return mod(**inputs) 2025-08-26T20:38:05.1126847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1126927Z outputs = self.mobilebert( 2025-08-26T20:38:05.1127209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1127283Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1127574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1127683Z layer_outputs = layer_module( 2025-08-26T20:38:05.1127972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1128152Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1128472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1128607Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1128905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1129036Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1129311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1129414Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1129418Z 2025-08-26T20:38:05.1129521Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1129731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1129800Z return mod(**inputs) 2025-08-26T20:38:05.1130093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1130174Z outputs = self.mobilebert( 2025-08-26T20:38:05.1130443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1130522Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1130800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1130870Z layer_outputs = layer_module( 2025-08-26T20:38:05.1131151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1131312Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1131609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1131738Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1132024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1132111Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1132115Z 2025-08-26T20:38:05.1132218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1132423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1132491Z return mod(**inputs) 2025-08-26T20:38:05.1132780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1132855Z outputs = self.mobilebert( 2025-08-26T20:38:05.1133138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1133226Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1133523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1133606Z layer_outputs = layer_module( 2025-08-26T20:38:05.1133915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1134092Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1134412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1134559Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1134841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1134928Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1135212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1135300Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1135304Z 2025-08-26T20:38:05.1135407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1135607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1135674Z return mod(**inputs) 2025-08-26T20:38:05.1135957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1136030Z outputs = self.mobilebert( 2025-08-26T20:38:05.1136312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1136384Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1136657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1136735Z layer_outputs = layer_module( 2025-08-26T20:38:05.1137006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1137100Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1137376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1137448Z self_outputs = self.self( 2025-08-26T20:38:05.1137734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1137826Z self.query(query_tensor) 2025-08-26T20:38:05.1137831Z 2025-08-26T20:38:05.1137943Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1138162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1138235Z return mod(**inputs) 2025-08-26T20:38:05.1138546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1138621Z outputs = self.mobilebert( 2025-08-26T20:38:05.1138931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1139005Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1139291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1139364Z layer_outputs = layer_module( 2025-08-26T20:38:05.1139652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1139748Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1140022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1140100Z self_outputs = self.self( 2025-08-26T20:38:05.1140373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1140459Z self.key(key_tensor) 2025-08-26T20:38:05.1140469Z 2025-08-26T20:38:05.1140571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1140762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1140853Z return mod(**inputs) 2025-08-26T20:38:05.1141130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1141207Z outputs = self.mobilebert( 2025-08-26T20:38:05.1141481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1141552Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1141837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1141906Z layer_outputs = layer_module( 2025-08-26T20:38:05.1142194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1142279Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1142560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1142640Z self_outputs = self.self( 2025-08-26T20:38:05.1142923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1143007Z self.value(value_tensor) 2025-08-26T20:38:05.1143011Z 2025-08-26T20:38:05.1143096Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1143185Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1143289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1143487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1143563Z return mod(**inputs) 2025-08-26T20:38:05.1143848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1143929Z outputs = self.mobilebert( 2025-08-26T20:38:05.1144230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1144306Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1144609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1144683Z layer_outputs = layer_module( 2025-08-26T20:38:05.1144971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1145055Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1145337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1145470Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1145752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1145847Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1145851Z 2025-08-26T20:38:05.1145955Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1146159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1146226Z return mod(**inputs) 2025-08-26T20:38:05.1146509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1146589Z outputs = self.mobilebert( 2025-08-26T20:38:05.1146892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1146976Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1147293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1147370Z layer_outputs = layer_module( 2025-08-26T20:38:05.1147675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1147842Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1148129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1148242Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1148528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1148614Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1148617Z 2025-08-26T20:38:05.1148721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1148927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1148993Z return mod(**inputs) 2025-08-26T20:38:05.1149284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1149356Z outputs = self.mobilebert( 2025-08-26T20:38:05.1149637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1149717Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1149996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1150077Z layer_outputs = layer_module( 2025-08-26T20:38:05.1150374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1150471Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1150795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1150945Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1151235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1151360Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1151650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1151746Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1151750Z 2025-08-26T20:38:05.1151855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1152062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1152131Z return mod(**inputs) 2025-08-26T20:38:05.1152426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1152500Z outputs = self.mobilebert( 2025-08-26T20:38:05.1152783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1152855Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1153136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1153239Z layer_outputs = layer_module( 2025-08-26T20:38:05.1153520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1153644Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1153925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1154039Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1154327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1154414Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1154417Z 2025-08-26T20:38:05.1154536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1154746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1154824Z return mod(**inputs) 2025-08-26T20:38:05.1155240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1155325Z outputs = self.mobilebert( 2025-08-26T20:38:05.1155631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1155708Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1156016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1156092Z layer_outputs = layer_module( 2025-08-26T20:38:05.1156388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1156499Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1156806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1156932Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1157240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1157395Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1157399Z 2025-08-26T20:38:05.1157530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1157749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1157830Z return mod(**inputs) 2025-08-26T20:38:05.1158140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1158226Z outputs = self.mobilebert( 2025-08-26T20:38:05.1158534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1158614Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1158924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1159004Z layer_outputs = layer_module( 2025-08-26T20:38:05.1159318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1159422Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1159837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1159989Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1160297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1160439Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1160444Z 2025-08-26T20:38:05.1161351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1161569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1161641Z return mod(**inputs) 2025-08-26T20:38:05.1161942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1162028Z outputs = self.mobilebert( 2025-08-26T20:38:05.1162332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1162412Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1162692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1162773Z layer_outputs = layer_module( 2025-08-26T20:38:05.1163053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1163150Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1163448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1163585Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1163890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1164022Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1164320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1164426Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1164431Z 2025-08-26T20:38:05.1164540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1164759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1164844Z return mod(**inputs) 2025-08-26T20:38:05.1165208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1165316Z outputs = self.mobilebert( 2025-08-26T20:38:05.1165612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1165700Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1165995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1166080Z layer_outputs = layer_module( 2025-08-26T20:38:05.1166378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1166480Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1166788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1166908Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1167212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1167302Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1167305Z 2025-08-26T20:38:05.1167422Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1167634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1167732Z return mod(**inputs) 2025-08-26T20:38:05.1168037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1168134Z outputs = self.mobilebert( 2025-08-26T20:38:05.1168447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1168526Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1168831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1168915Z layer_outputs = layer_module( 2025-08-26T20:38:05.1169220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1169328Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1169634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1169760Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1170065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1170186Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1170190Z 2025-08-26T20:38:05.1170308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1170522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1170601Z return mod(**inputs) 2025-08-26T20:38:05.1170914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1170991Z outputs = self.mobilebert( 2025-08-26T20:38:05.1171300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1171378Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1171695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1171796Z layer_outputs = layer_module( 2025-08-26T20:38:05.1172118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1172221Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1172515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1172656Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1172963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1173058Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1173062Z 2025-08-26T20:38:05.1173170Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1173382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1173461Z return mod(**inputs) 2025-08-26T20:38:05.1173764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1211583Z outputs = self.mobilebert( 2025-08-26T20:38:05.1212138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1212228Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1212547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1212816Z layer_outputs = layer_module( 2025-08-26T20:38:05.1213137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1213299Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1213632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1213776Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1214065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1214204Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1214488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1214597Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1214605Z 2025-08-26T20:38:05.1214722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1214943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1215028Z return mod(**inputs) 2025-08-26T20:38:05.1215324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1215417Z outputs = self.mobilebert( 2025-08-26T20:38:05.1215701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1215789Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1216075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1216152Z layer_outputs = layer_module( 2025-08-26T20:38:05.1216448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1216551Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1216880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1217001Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1217322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1217422Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1217426Z 2025-08-26T20:38:05.1217538Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1217752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1217823Z return mod(**inputs) 2025-08-26T20:38:05.1218115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1218195Z outputs = self.mobilebert( 2025-08-26T20:38:05.1218476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1218560Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1218844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1218924Z layer_outputs = layer_module( 2025-08-26T20:38:05.1219210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1219309Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1219623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1219738Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1220048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1220166Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1220170Z 2025-08-26T20:38:05.1220287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1220493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1220561Z return mod(**inputs) 2025-08-26T20:38:05.1220854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1220930Z outputs = self.mobilebert( 2025-08-26T20:38:05.1221247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1221325Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1221623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1221713Z layer_outputs = layer_module( 2025-08-26T20:38:05.1221995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1222098Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1222378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1222521Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1222827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1222919Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1222923Z 2025-08-26T20:38:05.1223042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1223263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1223365Z return mod(**inputs) 2025-08-26T20:38:05.1223692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1223774Z outputs = self.mobilebert( 2025-08-26T20:38:05.1224082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1224169Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1224467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1224548Z layer_outputs = layer_module( 2025-08-26T20:38:05.1224829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1224934Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1225216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1225344Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1225644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1225774Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1226086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1226206Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1226210Z 2025-08-26T20:38:05.1226322Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1226561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1226634Z return mod(**inputs) 2025-08-26T20:38:05.1226948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1227029Z outputs = self.mobilebert( 2025-08-26T20:38:05.1227343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1227421Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1227725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1227812Z layer_outputs = layer_module( 2025-08-26T20:38:05.1228141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1228283Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1228593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1228684Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1228698Z 2025-08-26T20:38:05.1228806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1229030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1229108Z return mod(**inputs) 2025-08-26T20:38:05.1229421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1229506Z outputs = self.mobilebert( 2025-08-26T20:38:05.1229803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1229881Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1230206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1230282Z layer_outputs = layer_module( 2025-08-26T20:38:05.1230605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1230738Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1231036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1231165Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1231171Z 2025-08-26T20:38:05.1231280Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1231495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1231567Z return mod(**inputs) 2025-08-26T20:38:05.1231877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1231953Z outputs = self.mobilebert( 2025-08-26T20:38:05.1232257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1232344Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1232641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1232723Z layer_outputs = layer_module( 2025-08-26T20:38:05.1233029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1233221Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1233545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1233649Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1233653Z 2025-08-26T20:38:05.1233770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1233979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1234057Z return mod(**inputs) 2025-08-26T20:38:05.1234364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1234440Z outputs = self.mobilebert( 2025-08-26T20:38:05.1234740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1234818Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1235121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1235199Z layer_outputs = layer_module( 2025-08-26T20:38:05.1235495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1235673Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1235976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1236118Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1236426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1236536Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1236540Z 2025-08-26T20:38:05.1236654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1236889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1236971Z return mod(**inputs) 2025-08-26T20:38:05.1237304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1237394Z outputs = self.mobilebert( 2025-08-26T20:38:05.1237701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1237780Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1238101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1238179Z layer_outputs = layer_module( 2025-08-26T20:38:05.1238491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1238666Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1238981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1239117Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1239419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1239612Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1239618Z 2025-08-26T20:38:05.1239734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1239981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1240056Z return mod(**inputs) 2025-08-26T20:38:05.1240366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1240475Z outputs = self.mobilebert( 2025-08-26T20:38:05.1240783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1240875Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1241188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1241272Z layer_outputs = layer_module( 2025-08-26T20:38:05.1241568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1241737Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1242043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1242177Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1242484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1242615Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1242919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1243024Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1243028Z 2025-08-26T20:38:05.1243130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1243336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1243402Z return mod(**inputs) 2025-08-26T20:38:05.1243691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1243764Z outputs = self.mobilebert( 2025-08-26T20:38:05.1244058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1244158Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1244439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1244518Z layer_outputs = layer_module( 2025-08-26T20:38:05.1244795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1244968Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1245249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1245363Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1245654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1245740Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1245744Z 2025-08-26T20:38:05.1245854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1246057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1246128Z return mod(**inputs) 2025-08-26T20:38:05.1246437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1246573Z outputs = self.mobilebert( 2025-08-26T20:38:05.1246874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1246980Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1247287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1247363Z layer_outputs = layer_module( 2025-08-26T20:38:05.1247659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1247837Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1248134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1248250Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1248522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1248608Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1248890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1248980Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1248984Z 2025-08-26T20:38:05.1249094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1249288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1249361Z return mod(**inputs) 2025-08-26T20:38:05.1249636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1249708Z outputs = self.mobilebert( 2025-08-26T20:38:05.1249995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1250071Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1250374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1250447Z layer_outputs = layer_module( 2025-08-26T20:38:05.1250747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1250844Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1251121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1251200Z self_outputs = self.self( 2025-08-26T20:38:05.1251478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1251561Z self.query(query_tensor) 2025-08-26T20:38:05.1251564Z 2025-08-26T20:38:05.1251668Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1251870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1251947Z return mod(**inputs) 2025-08-26T20:38:05.1252232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1252312Z outputs = self.mobilebert( 2025-08-26T20:38:05.1252593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1252667Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1252958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1253049Z layer_outputs = layer_module( 2025-08-26T20:38:05.1253344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1253449Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1253736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1253823Z self_outputs = self.self( 2025-08-26T20:38:05.1254120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1254199Z self.key(key_tensor) 2025-08-26T20:38:05.1254203Z 2025-08-26T20:38:05.1254322Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1254525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1254594Z return mod(**inputs) 2025-08-26T20:38:05.1254873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1254955Z outputs = self.mobilebert( 2025-08-26T20:38:05.1255252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1255335Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1255642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1255714Z layer_outputs = layer_module( 2025-08-26T20:38:05.1256001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1256085Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1256373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1256444Z self_outputs = self.self( 2025-08-26T20:38:05.1256727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1256810Z self.value(value_tensor) 2025-08-26T20:38:05.1256827Z 2025-08-26T20:38:05.1256911Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1257013Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1257117Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1257317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1257383Z return mod(**inputs) 2025-08-26T20:38:05.1257660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1257741Z outputs = self.mobilebert( 2025-08-26T20:38:05.1258008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1258087Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1258360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1258430Z layer_outputs = layer_module( 2025-08-26T20:38:05.1258708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1258792Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1259075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1259203Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1259491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1259596Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1259616Z 2025-08-26T20:38:05.1259720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1259929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1259997Z return mod(**inputs) 2025-08-26T20:38:05.1260286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1260359Z outputs = self.mobilebert( 2025-08-26T20:38:05.1260636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1260718Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1260998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1261078Z layer_outputs = layer_module( 2025-08-26T20:38:05.1261354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1261527Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1261808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1261921Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1262207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1262293Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1262296Z 2025-08-26T20:38:05.1262408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1262617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1262688Z return mod(**inputs) 2025-08-26T20:38:05.1262995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1263088Z outputs = self.mobilebert( 2025-08-26T20:38:05.1263408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1263488Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1263791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1263867Z layer_outputs = layer_module( 2025-08-26T20:38:05.1264165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1264263Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1264558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1264694Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1264992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1265129Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1265433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1265531Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1265535Z 2025-08-26T20:38:05.1265649Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1265860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1265960Z return mod(**inputs) 2025-08-26T20:38:05.1266261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1266354Z outputs = self.mobilebert( 2025-08-26T20:38:05.1266661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1266741Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1267044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1267120Z layer_outputs = layer_module( 2025-08-26T20:38:05.1267416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1267526Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1267823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1267953Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1268249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1268345Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1268349Z 2025-08-26T20:38:05.1268458Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1268666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1268744Z return mod(**inputs) 2025-08-26T20:38:05.1269041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1269126Z outputs = self.mobilebert( 2025-08-26T20:38:05.1269421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1269500Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1269819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1269898Z layer_outputs = layer_module( 2025-08-26T20:38:05.1270220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1270326Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1270626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1270746Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1271042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1271171Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1271176Z 2025-08-26T20:38:05.1271285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1271504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1271574Z return mod(**inputs) 2025-08-26T20:38:05.1271876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1271959Z outputs = self.mobilebert( 2025-08-26T20:38:05.1272255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1272341Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1272648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1272759Z layer_outputs = layer_module( 2025-08-26T20:38:05.1273057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1273178Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1273480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1273615Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1273915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1274005Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1274009Z 2025-08-26T20:38:05.1274122Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1274335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1274405Z return mod(**inputs) 2025-08-26T20:38:05.1274707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1274787Z outputs = self.mobilebert( 2025-08-26T20:38:05.1275090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1275170Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1275467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1275552Z layer_outputs = layer_module( 2025-08-26T20:38:05.1275855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1275967Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1276264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1276402Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1276719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1276867Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1277174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1277274Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1277278Z 2025-08-26T20:38:05.1277396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1277605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1277679Z return mod(**inputs) 2025-08-26T20:38:05.1277985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1278062Z outputs = self.mobilebert( 2025-08-26T20:38:05.1278370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1278450Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1278748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1278832Z layer_outputs = layer_module( 2025-08-26T20:38:05.1279136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1279243Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1279845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1280015Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1280337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1280431Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1280435Z 2025-08-26T20:38:05.1280562Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1280778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1280861Z return mod(**inputs) 2025-08-26T20:38:05.1281182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1281258Z outputs = self.mobilebert( 2025-08-26T20:38:05.1281565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1281643Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1281950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1282027Z layer_outputs = layer_module( 2025-08-26T20:38:05.1282333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1282433Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1282728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1282853Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1283149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1283277Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1283282Z 2025-08-26T20:38:05.1283394Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1283620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1283700Z return mod(**inputs) 2025-08-26T20:38:05.1284016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1284104Z outputs = self.mobilebert( 2025-08-26T20:38:05.1284402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1284488Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1284783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1284859Z layer_outputs = layer_module( 2025-08-26T20:38:05.1285163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1285267Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1285570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1285705Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1286001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1286129Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1286132Z 2025-08-26T20:38:05.1286240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1286475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1286546Z return mod(**inputs) 2025-08-26T20:38:05.1286849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1286951Z outputs = self.mobilebert( 2025-08-26T20:38:05.1287246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1287332Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1287625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1287706Z layer_outputs = layer_module( 2025-08-26T20:38:05.1287997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1288099Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1288399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1288530Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1288834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1288965Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1289269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1289370Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1289374Z 2025-08-26T20:38:05.1289483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1289702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1289773Z return mod(**inputs) 2025-08-26T20:38:05.1290078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1290155Z outputs = self.mobilebert( 2025-08-26T20:38:05.1290469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1290571Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1290869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1290952Z layer_outputs = layer_module( 2025-08-26T20:38:05.1291249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1291355Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1291652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1291773Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1292079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1292168Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1292172Z 2025-08-26T20:38:05.1292289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1292498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1292571Z return mod(**inputs) 2025-08-26T20:38:05.1292881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1292954Z outputs = self.mobilebert( 2025-08-26T20:38:05.1293261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1293337Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1293641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1293713Z layer_outputs = layer_module( 2025-08-26T20:38:05.1293993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1294094Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1294373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1294492Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1294777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1294889Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1294902Z 2025-08-26T20:38:05.1295006Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1295205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1295278Z return mod(**inputs) 2025-08-26T20:38:05.1295561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1295640Z outputs = self.mobilebert( 2025-08-26T20:38:05.1295916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1295989Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1296443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1296526Z layer_outputs = layer_module( 2025-08-26T20:38:05.1296817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1296916Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1297246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1297418Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1297700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1297792Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1297796Z 2025-08-26T20:38:05.1297900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1298107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1298174Z return mod(**inputs) 2025-08-26T20:38:05.1298458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1298541Z outputs = self.mobilebert( 2025-08-26T20:38:05.1298825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1298909Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1299200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1299276Z layer_outputs = layer_module( 2025-08-26T20:38:05.1299579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1299712Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1299996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1300143Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1300432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1300554Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1300832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1300934Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1300938Z 2025-08-26T20:38:05.1301047Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1301263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1301335Z return mod(**inputs) 2025-08-26T20:38:05.1301635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1301722Z outputs = self.mobilebert( 2025-08-26T20:38:05.1302020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1302105Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1302398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1302481Z layer_outputs = layer_module( 2025-08-26T20:38:05.1302784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1302912Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1303219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1303305Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1303308Z 2025-08-26T20:38:05.1303414Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1303628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1303697Z return mod(**inputs) 2025-08-26T20:38:05.1304009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1304082Z outputs = self.mobilebert( 2025-08-26T20:38:05.1304369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1304441Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1304727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1304801Z layer_outputs = layer_module( 2025-08-26T20:38:05.1305097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1305231Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1305526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1305650Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1305654Z 2025-08-26T20:38:05.1305762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1305970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1306046Z return mod(**inputs) 2025-08-26T20:38:05.1306363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1306447Z outputs = self.mobilebert( 2025-08-26T20:38:05.1306766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1306848Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1307130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1307201Z layer_outputs = layer_module( 2025-08-26T20:38:05.1307492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1307653Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1307940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1308036Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1308040Z 2025-08-26T20:38:05.1308144Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1308351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1308418Z return mod(**inputs) 2025-08-26T20:38:05.1308711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1308784Z outputs = self.mobilebert( 2025-08-26T20:38:05.1309076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1309150Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1309430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1309511Z layer_outputs = layer_module( 2025-08-26T20:38:05.1309791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1309978Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1310276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1310404Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1310691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1310785Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1310788Z 2025-08-26T20:38:05.1310900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1311113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1311190Z return mod(**inputs) 2025-08-26T20:38:05.1311491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1311568Z outputs = self.mobilebert( 2025-08-26T20:38:05.1311873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1311950Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1312254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1312330Z layer_outputs = layer_module( 2025-08-26T20:38:05.1312624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1312820Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1313117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1313278Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1313573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1313673Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1313676Z 2025-08-26T20:38:05.1313785Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1313996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1314075Z return mod(**inputs) 2025-08-26T20:38:05.1314374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1314459Z outputs = self.mobilebert( 2025-08-26T20:38:05.1314764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1314845Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1315163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1315243Z layer_outputs = layer_module( 2025-08-26T20:38:05.1315556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1315728Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1316044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1316179Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1316472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1316611Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1316925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1317049Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1317054Z 2025-08-26T20:38:05.1317164Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1317379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1317451Z return mod(**inputs) 2025-08-26T20:38:05.1317752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1317839Z outputs = self.mobilebert( 2025-08-26T20:38:05.1318143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1318231Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1318541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1318618Z layer_outputs = layer_module( 2025-08-26T20:38:05.1318935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1319115Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1319428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1319641Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1319953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1320070Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1320074Z 2025-08-26T20:38:05.1320189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1320415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1320489Z return mod(**inputs) 2025-08-26T20:38:05.1320805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1320884Z outputs = self.mobilebert( 2025-08-26T20:38:05.1321195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1321282Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1321575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1321659Z layer_outputs = layer_module( 2025-08-26T20:38:05.1321953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1322130Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1322425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1322542Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1322842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1322935Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1323239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1323335Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1323341Z 2025-08-26T20:38:05.1323449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1323691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1323764Z return mod(**inputs) 2025-08-26T20:38:05.1324085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1324165Z outputs = self.mobilebert( 2025-08-26T20:38:05.1324466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1324543Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1324844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1324930Z layer_outputs = layer_module( 2025-08-26T20:38:05.1325236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1325340Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1325652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1325728Z self_outputs = self.self( 2025-08-26T20:38:05.1326034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1326112Z self.query(query_tensor) 2025-08-26T20:38:05.1326115Z 2025-08-26T20:38:05.1326231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1326459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1326538Z return mod(**inputs) 2025-08-26T20:38:05.1326835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1326932Z outputs = self.mobilebert( 2025-08-26T20:38:05.1327236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1327317Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1327622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1327698Z layer_outputs = layer_module( 2025-08-26T20:38:05.1327991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1328091Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1328386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1328468Z self_outputs = self.self( 2025-08-26T20:38:05.1328764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1328835Z self.key(key_tensor) 2025-08-26T20:38:05.1328846Z 2025-08-26T20:38:05.1328955Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1329163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1329241Z return mod(**inputs) 2025-08-26T20:38:05.1329542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1329624Z outputs = self.mobilebert( 2025-08-26T20:38:05.1329922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1330000Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1330326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1330403Z layer_outputs = layer_module( 2025-08-26T20:38:05.1330727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1330820Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1331117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1331200Z self_outputs = self.self( 2025-08-26T20:38:05.1331494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1331580Z self.value(value_tensor) 2025-08-26T20:38:05.1331584Z 2025-08-26T20:38:05.1331672Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1331766Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1331877Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1332089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1332170Z return mod(**inputs) 2025-08-26T20:38:05.1332473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1332555Z outputs = self.mobilebert( 2025-08-26T20:38:05.1332852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1332940Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1333246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1333319Z layer_outputs = layer_module( 2025-08-26T20:38:05.1333625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1333715Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1334014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1334153Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1334451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1334552Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1334556Z 2025-08-26T20:38:05.1334666Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1334885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1334955Z return mod(**inputs) 2025-08-26T20:38:05.1335256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1335342Z outputs = self.mobilebert( 2025-08-26T20:38:05.1335641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1335723Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1336017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1336093Z layer_outputs = layer_module( 2025-08-26T20:38:05.1336396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1336569Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1336876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1337015Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1337335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1337424Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1337428Z 2025-08-26T20:38:05.1337537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1337751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1337820Z return mod(**inputs) 2025-08-26T20:38:05.1338128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1338205Z outputs = self.mobilebert( 2025-08-26T20:38:05.1338502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1338587Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1338885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1338970Z layer_outputs = layer_module( 2025-08-26T20:38:05.1339266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1339360Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1339670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1339819Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1340124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1340282Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1340653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1340758Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1340762Z 2025-08-26T20:38:05.1340876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1341087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1341157Z return mod(**inputs) 2025-08-26T20:38:05.1341461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1341539Z outputs = self.mobilebert( 2025-08-26T20:38:05.1341839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1341919Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1342217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1342301Z layer_outputs = layer_module( 2025-08-26T20:38:05.1342601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1342711Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1343009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1343127Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1343431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1343521Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1343527Z 2025-08-26T20:38:05.1343644Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1343879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1343961Z return mod(**inputs) 2025-08-26T20:38:05.1344284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1344361Z outputs = self.mobilebert( 2025-08-26T20:38:05.1344668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1344747Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1345060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1345138Z layer_outputs = layer_module( 2025-08-26T20:38:05.1345446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1345570Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1345871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1345999Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1346297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1346424Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1346428Z 2025-08-26T20:38:05.1346558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1346770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1346850Z return mod(**inputs) 2025-08-26T20:38:05.1347177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1347264Z outputs = self.mobilebert( 2025-08-26T20:38:05.1347570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1347650Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1347953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1348031Z layer_outputs = layer_module( 2025-08-26T20:38:05.1348343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1348450Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1348765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1348908Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1349217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1349318Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1349322Z 2025-08-26T20:38:05.1349435Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1349657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1349731Z return mod(**inputs) 2025-08-26T20:38:05.1350043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1350130Z outputs = self.mobilebert( 2025-08-26T20:38:05.1350426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1350514Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1350833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1350972Z layer_outputs = layer_module( 2025-08-26T20:38:05.1351279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1351382Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1351693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1351829Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1352142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1352279Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1352593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1352695Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1352699Z 2025-08-26T20:38:05.1352812Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1353033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1353105Z return mod(**inputs) 2025-08-26T20:38:05.1353421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1353519Z outputs = self.mobilebert( 2025-08-26T20:38:05.1353825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1353934Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1354246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1354336Z layer_outputs = layer_module( 2025-08-26T20:38:05.1354649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1354753Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1355069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1355193Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1355512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1355605Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1355610Z 2025-08-26T20:38:05.1355730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1355949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1356023Z return mod(**inputs) 2025-08-26T20:38:05.1356345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1356425Z outputs = self.mobilebert( 2025-08-26T20:38:05.1356738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1356819Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1357129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1357215Z layer_outputs = layer_module( 2025-08-26T20:38:05.1357522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1357650Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1357975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1358109Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1358424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1358550Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1358554Z 2025-08-26T20:38:05.1358678Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1358904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1358989Z return mod(**inputs) 2025-08-26T20:38:05.1359308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1359392Z outputs = self.mobilebert( 2025-08-26T20:38:05.1359792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1359878Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1360191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1360270Z layer_outputs = layer_module( 2025-08-26T20:38:05.1360581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1360711Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1361023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1361188Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1361514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1361618Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1361623Z 2025-08-26T20:38:05.1361736Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1361963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1362038Z return mod(**inputs) 2025-08-26T20:38:05.1362350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1362443Z outputs = self.mobilebert( 2025-08-26T20:38:05.1362749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1362839Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1363144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1363226Z layer_outputs = layer_module( 2025-08-26T20:38:05.1363538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1363640Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1363949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1364083Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1364388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1364529Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1364851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1364977Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1364982Z 2025-08-26T20:38:05.1365096Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1365321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1365394Z return mod(**inputs) 2025-08-26T20:38:05.1365712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1365798Z outputs = self.mobilebert( 2025-08-26T20:38:05.1366094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1366181Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1366491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1366567Z layer_outputs = layer_module( 2025-08-26T20:38:05.1366879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1366979Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1367290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1367410Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1367744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1367835Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1367858Z 2025-08-26T20:38:05.1367970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1368190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1368262Z return mod(**inputs) 2025-08-26T20:38:05.1368572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1368653Z outputs = self.mobilebert( 2025-08-26T20:38:05.1368961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1369047Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1369358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1369442Z layer_outputs = layer_module( 2025-08-26T20:38:05.1369748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1369857Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1370162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1370280Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1370596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1370715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1370719Z 2025-08-26T20:38:05.1370833Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1371045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1371122Z return mod(**inputs) 2025-08-26T20:38:05.1371423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1371520Z outputs = self.mobilebert( 2025-08-26T20:38:05.1371853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1371932Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1372255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1372331Z layer_outputs = layer_module( 2025-08-26T20:38:05.1372642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1372751Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1373061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1373200Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1373512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1373610Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1373614Z 2025-08-26T20:38:05.1373724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1373932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1374008Z return mod(**inputs) 2025-08-26T20:38:05.1374311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1374427Z outputs = self.mobilebert( 2025-08-26T20:38:05.1374730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1374824Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1375139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1375216Z layer_outputs = layer_module( 2025-08-26T20:38:05.1375527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1375627Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1375930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1376068Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1376369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1376508Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1376812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1376918Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1376922Z 2025-08-26T20:38:05.1377030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1377241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1377318Z return mod(**inputs) 2025-08-26T20:38:05.1377619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1377705Z outputs = self.mobilebert( 2025-08-26T20:38:05.1378001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1378089Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1378408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1378485Z layer_outputs = layer_module( 2025-08-26T20:38:05.1378806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1378940Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1379245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1379336Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1379343Z 2025-08-26T20:38:05.1379453Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1379671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1379745Z return mod(**inputs) 2025-08-26T20:38:05.1380057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1380133Z outputs = self.mobilebert( 2025-08-26T20:38:05.1380437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1380515Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1380812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1380899Z layer_outputs = layer_module( 2025-08-26T20:38:05.1381203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1381362Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1381688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1381813Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1381817Z 2025-08-26T20:38:05.1381939Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1382152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1382233Z return mod(**inputs) 2025-08-26T20:38:05.1382541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1382626Z outputs = self.mobilebert( 2025-08-26T20:38:05.1382931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1383009Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1383312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1383393Z layer_outputs = layer_module( 2025-08-26T20:38:05.1383708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1383881Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1384190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1384303Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1384307Z 2025-08-26T20:38:05.1384420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1384648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1384722Z return mod(**inputs) 2025-08-26T20:38:05.1385045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1385148Z outputs = self.mobilebert( 2025-08-26T20:38:05.1385473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1385563Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1385874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1385961Z layer_outputs = layer_module( 2025-08-26T20:38:05.1386274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1386452Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1386773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1386916Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1387236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1387339Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1387343Z 2025-08-26T20:38:05.1387467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1387688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1387762Z return mod(**inputs) 2025-08-26T20:38:05.1388087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1388183Z outputs = self.mobilebert( 2025-08-26T20:38:05.1388500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1388605Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1388915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1389004Z layer_outputs = layer_module( 2025-08-26T20:38:05.1389313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1389493Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1389800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1389946Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1390258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1390354Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1390359Z 2025-08-26T20:38:05.1390483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1390704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1390785Z return mod(**inputs) 2025-08-26T20:38:05.1391101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1391179Z outputs = self.mobilebert( 2025-08-26T20:38:05.1391497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1391578Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1391896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1391977Z layer_outputs = layer_module( 2025-08-26T20:38:05.1392319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1392513Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1392819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1392960Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1393268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1393409Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1393715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1393818Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1393828Z 2025-08-26T20:38:05.1393942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1394159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1394239Z return mod(**inputs) 2025-08-26T20:38:05.1394558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1394644Z outputs = self.mobilebert( 2025-08-26T20:38:05.1394941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1395039Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1395353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1395452Z layer_outputs = layer_module( 2025-08-26T20:38:05.1395766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1395946Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1396403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1396539Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1396850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1396956Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1396961Z 2025-08-26T20:38:05.1397075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1397300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1397377Z return mod(**inputs) 2025-08-26T20:38:05.1397688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1397777Z outputs = self.mobilebert( 2025-08-26T20:38:05.1398085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1398177Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1398486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1398567Z layer_outputs = layer_module( 2025-08-26T20:38:05.1398884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1399060Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1399435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1399612Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1399974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1400074Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1400386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1400497Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1400503Z 2025-08-26T20:38:05.1400617Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1400846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1400923Z return mod(**inputs) 2025-08-26T20:38:05.1401248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1401326Z outputs = self.mobilebert( 2025-08-26T20:38:05.1401624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1401713Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1402009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1402094Z layer_outputs = layer_module( 2025-08-26T20:38:05.1402391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1402514Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1402824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1402928Z self_outputs = self.self( 2025-08-26T20:38:05.1403234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1403313Z self.query(query_tensor) 2025-08-26T20:38:05.1403316Z 2025-08-26T20:38:05.1403434Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1403642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1403713Z return mod(**inputs) 2025-08-26T20:38:05.1404039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1404117Z outputs = self.mobilebert( 2025-08-26T20:38:05.1404423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1404504Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1404810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1404898Z layer_outputs = layer_module( 2025-08-26T20:38:05.1405204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1405305Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1405612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1405698Z self_outputs = self.self( 2025-08-26T20:38:05.1406006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1406078Z self.key(key_tensor) 2025-08-26T20:38:05.1406083Z 2025-08-26T20:38:05.1406198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1406426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1406504Z return mod(**inputs) 2025-08-26T20:38:05.1406823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1406902Z outputs = self.mobilebert( 2025-08-26T20:38:05.1407206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1407284Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1407588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1407667Z layer_outputs = layer_module( 2025-08-26T20:38:05.1407969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1408068Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1408390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1408472Z self_outputs = self.self( 2025-08-26T20:38:05.1408777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1408859Z self.value(value_tensor) 2025-08-26T20:38:05.1408862Z 2025-08-26T20:38:05.1408950Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1409034Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1409169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1409374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1409473Z return mod(**inputs) 2025-08-26T20:38:05.1409783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1409860Z outputs = self.mobilebert( 2025-08-26T20:38:05.1410165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1410240Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1410552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1410627Z layer_outputs = layer_module( 2025-08-26T20:38:05.1410929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1411027Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1411323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1411466Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1411771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1411870Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1411874Z 2025-08-26T20:38:05.1411983Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1412193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1412271Z return mod(**inputs) 2025-08-26T20:38:05.1412579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1412665Z outputs = self.mobilebert( 2025-08-26T20:38:05.1412974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1413073Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1413396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1413474Z layer_outputs = layer_module( 2025-08-26T20:38:05.1413784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1413954Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1414271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1414393Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1414700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1414798Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1414804Z 2025-08-26T20:38:05.1414912Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1415133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1415203Z return mod(**inputs) 2025-08-26T20:38:05.1415517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1415601Z outputs = self.mobilebert( 2025-08-26T20:38:05.1415909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1416011Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1416325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1416427Z layer_outputs = layer_module( 2025-08-26T20:38:05.1416735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1416825Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1417129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1417259Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1417568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1417714Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1418072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1418172Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1418176Z 2025-08-26T20:38:05.1418279Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1418478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1418543Z return mod(**inputs) 2025-08-26T20:38:05.1418828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1418901Z outputs = self.mobilebert( 2025-08-26T20:38:05.1419182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1419265Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1419542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1419624Z layer_outputs = layer_module( 2025-08-26T20:38:05.1419924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1420024Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1420325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1420441Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1420731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1420815Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1420820Z 2025-08-26T20:38:05.1420927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1421120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1421187Z return mod(**inputs) 2025-08-26T20:38:05.1421478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1421551Z outputs = self.mobilebert( 2025-08-26T20:38:05.1421841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1421915Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1422191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1422271Z layer_outputs = layer_module( 2025-08-26T20:38:05.1422549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1422679Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1422962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1423103Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1423387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1423499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1423503Z 2025-08-26T20:38:05.1423616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1423815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1423887Z return mod(**inputs) 2025-08-26T20:38:05.1424173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1424247Z outputs = self.mobilebert( 2025-08-26T20:38:05.1424535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1424610Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1424902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1424975Z layer_outputs = layer_module( 2025-08-26T20:38:05.1425262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1425357Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1425640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1425774Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1426054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1426148Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1426151Z 2025-08-26T20:38:05.1426295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1426506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1426579Z return mod(**inputs) 2025-08-26T20:38:05.1426856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1426934Z outputs = self.mobilebert( 2025-08-26T20:38:05.1427216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1427297Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1427585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1427662Z layer_outputs = layer_module( 2025-08-26T20:38:05.1427968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1428070Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1428371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1428502Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1428798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1428957Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1429249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1429373Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1429376Z 2025-08-26T20:38:05.1429492Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1429700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1429766Z return mod(**inputs) 2025-08-26T20:38:05.1430055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1430138Z outputs = self.mobilebert( 2025-08-26T20:38:05.1430425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1430507Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1430792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1430866Z layer_outputs = layer_module( 2025-08-26T20:38:05.1431159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1431253Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1431549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1431664Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1431956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1432044Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1432049Z 2025-08-26T20:38:05.1432151Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1432359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1432428Z return mod(**inputs) 2025-08-26T20:38:05.1432743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1432816Z outputs = self.mobilebert( 2025-08-26T20:38:05.1433112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1433195Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1433472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1433551Z layer_outputs = layer_module( 2025-08-26T20:38:05.1433841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1433947Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1434245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1434368Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1434674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1434793Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1434797Z 2025-08-26T20:38:05.1434914Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1435125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1435196Z return mod(**inputs) 2025-08-26T20:38:05.1435502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1435598Z outputs = self.mobilebert( 2025-08-26T20:38:05.1435900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1436001Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1436306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1436384Z layer_outputs = layer_module( 2025-08-26T20:38:05.1436679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1436787Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1437090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1437235Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1437535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1437633Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1437645Z 2025-08-26T20:38:05.1437759Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1437975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1438057Z return mod(**inputs) 2025-08-26T20:38:05.1438361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1438449Z outputs = self.mobilebert( 2025-08-26T20:38:05.1438749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1438833Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1439143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1439224Z layer_outputs = layer_module( 2025-08-26T20:38:05.1439631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1439737Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1440052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1440193Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1440491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1440628Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1440925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1441032Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1441036Z 2025-08-26T20:38:05.1441148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1441360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1441442Z return mod(**inputs) 2025-08-26T20:38:05.1441744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1441830Z outputs = self.mobilebert( 2025-08-26T20:38:05.1442124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1442246Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1442550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1442648Z layer_outputs = layer_module( 2025-08-26T20:38:05.1442953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1443053Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1443359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1443478Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1443776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1443874Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1443879Z 2025-08-26T20:38:05.1443988Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1444205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1444276Z return mod(**inputs) 2025-08-26T20:38:05.1444576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1444659Z outputs = self.mobilebert( 2025-08-26T20:38:05.1444957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1445043Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1445342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1445425Z layer_outputs = layer_module( 2025-08-26T20:38:05.1445731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1445833Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1446136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1446276Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1446598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1446720Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1446724Z 2025-08-26T20:38:05.1446839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1447049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1447118Z return mod(**inputs) 2025-08-26T20:38:05.1447422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1447501Z outputs = self.mobilebert( 2025-08-26T20:38:05.1447805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1447887Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1448184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1448269Z layer_outputs = layer_module( 2025-08-26T20:38:05.1448564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1448670Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1448963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1449109Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1449398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1449511Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1449515Z 2025-08-26T20:38:05.1449628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1449827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1449902Z return mod(**inputs) 2025-08-26T20:38:05.1450190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1450262Z outputs = self.mobilebert( 2025-08-26T20:38:05.1450550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1450624Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1450914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1450989Z layer_outputs = layer_module( 2025-08-26T20:38:05.1451295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1451402Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1451708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1451845Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1452152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1452287Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1452593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1452692Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1452696Z 2025-08-26T20:38:05.1452835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1453047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1453143Z return mod(**inputs) 2025-08-26T20:38:05.1453458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1453534Z outputs = self.mobilebert( 2025-08-26T20:38:05.1453850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1453927Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1454241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1454329Z layer_outputs = layer_module( 2025-08-26T20:38:05.1454621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1454743Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1455033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1455124Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1455127Z 2025-08-26T20:38:05.1455228Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1455429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1455493Z return mod(**inputs) 2025-08-26T20:38:05.1455791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1455870Z outputs = self.mobilebert( 2025-08-26T20:38:05.1456164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1456250Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1456557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1456639Z layer_outputs = layer_module( 2025-08-26T20:38:05.1456946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1457075Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1457387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1457509Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1457513Z 2025-08-26T20:38:05.1457631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1457852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1457922Z return mod(**inputs) 2025-08-26T20:38:05.1458245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1458322Z outputs = self.mobilebert( 2025-08-26T20:38:05.1458640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1458719Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1459040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1459117Z layer_outputs = layer_module( 2025-08-26T20:38:05.1459419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1459654Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1459971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1460081Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1460085Z 2025-08-26T20:38:05.1460203Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1460409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1460475Z return mod(**inputs) 2025-08-26T20:38:05.1460763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1460843Z outputs = self.mobilebert( 2025-08-26T20:38:05.1461125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1461208Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1461515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1461596Z layer_outputs = layer_module( 2025-08-26T20:38:05.1461913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1462095Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1462397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1462876Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1463183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1463304Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1463309Z 2025-08-26T20:38:05.1463413Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1463622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1463692Z return mod(**inputs) 2025-08-26T20:38:05.1464004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1464082Z outputs = self.mobilebert( 2025-08-26T20:38:05.1464388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1464477Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1464771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1464858Z layer_outputs = layer_module( 2025-08-26T20:38:05.1465158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1465327Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1465635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1465765Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1466071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1466163Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1466167Z 2025-08-26T20:38:05.1466283Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1466496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1466566Z return mod(**inputs) 2025-08-26T20:38:05.1466904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1467000Z outputs = self.mobilebert( 2025-08-26T20:38:05.1467306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1467384Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1467690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1467768Z layer_outputs = layer_module( 2025-08-26T20:38:05.1468064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1468240Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1468551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1468691Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1468986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1469114Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1469426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1469547Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1469551Z 2025-08-26T20:38:05.1469670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1469879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1469977Z return mod(**inputs) 2025-08-26T20:38:05.1470279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1470357Z outputs = self.mobilebert( 2025-08-26T20:38:05.1470660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1470737Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1471039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1471114Z layer_outputs = layer_module( 2025-08-26T20:38:05.1471413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1471593Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1471894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1472019Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1472316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1472414Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1472418Z 2025-08-26T20:38:05.1472526Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1472738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1472820Z return mod(**inputs) 2025-08-26T20:38:05.1473117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1473203Z outputs = self.mobilebert( 2025-08-26T20:38:05.1473517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1473597Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1473922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1474003Z layer_outputs = layer_module( 2025-08-26T20:38:05.1474314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1474488Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1474804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1474925Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1475233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1475336Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1475645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1475753Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1475756Z 2025-08-26T20:38:05.1475870Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1476093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1476164Z return mod(**inputs) 2025-08-26T20:38:05.1476503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1476596Z outputs = self.mobilebert( 2025-08-26T20:38:05.1476924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1477013Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1477320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1477398Z layer_outputs = layer_module( 2025-08-26T20:38:05.1477709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1477802Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1478113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1478195Z self_outputs = self.self( 2025-08-26T20:38:05.1478499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1478587Z self.query(query_tensor) 2025-08-26T20:38:05.1478591Z 2025-08-26T20:38:05.1478707Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1478931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1479003Z return mod(**inputs) 2025-08-26T20:38:05.1479316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1479394Z outputs = self.mobilebert( 2025-08-26T20:38:05.1479785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1479880Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1480186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1480275Z layer_outputs = layer_module( 2025-08-26T20:38:05.1480606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1480705Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1481038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1481120Z self_outputs = self.self( 2025-08-26T20:38:05.1481437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1481511Z self.key(key_tensor) 2025-08-26T20:38:05.1481517Z 2025-08-26T20:38:05.1481640Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1481858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1481932Z return mod(**inputs) 2025-08-26T20:38:05.1482260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1482338Z outputs = self.mobilebert( 2025-08-26T20:38:05.1482656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1482737Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1483042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1483130Z layer_outputs = layer_module( 2025-08-26T20:38:05.1483435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1483557Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1483865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1483960Z self_outputs = self.self( 2025-08-26T20:38:05.1484276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1484359Z self.value(value_tensor) 2025-08-26T20:38:05.1484363Z 2025-08-26T20:38:05.1484463Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1484549Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1484671Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1484890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1484962Z return mod(**inputs) 2025-08-26T20:38:05.1485283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1485362Z outputs = self.mobilebert( 2025-08-26T20:38:05.1485681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1485766Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1486075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1486163Z layer_outputs = layer_module( 2025-08-26T20:38:05.1486473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1486572Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1486879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1487015Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1487327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1487443Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1487448Z 2025-08-26T20:38:05.1487570Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1487808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1487889Z return mod(**inputs) 2025-08-26T20:38:05.1488204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1488284Z outputs = self.mobilebert( 2025-08-26T20:38:05.1488609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1488692Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1489008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1489087Z layer_outputs = layer_module( 2025-08-26T20:38:05.1489393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1489580Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1489889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1490021Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1490397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1490525Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1490530Z 2025-08-26T20:38:05.1490639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1490869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1490948Z return mod(**inputs) 2025-08-26T20:38:05.1491250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1491335Z outputs = self.mobilebert( 2025-08-26T20:38:05.1491630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1491706Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1492009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1492086Z layer_outputs = layer_module( 2025-08-26T20:38:05.1492390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1492481Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1492784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1492916Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1493215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1493355Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1493650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1493757Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1493761Z 2025-08-26T20:38:05.1493869Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1494082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1494154Z return mod(**inputs) 2025-08-26T20:38:05.1494471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1494558Z outputs = self.mobilebert( 2025-08-26T20:38:05.1494874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1494961Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1495257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1495333Z layer_outputs = layer_module( 2025-08-26T20:38:05.1495639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1495742Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1496047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1496363Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1496681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1496774Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1496778Z 2025-08-26T20:38:05.1496890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1497109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1497181Z return mod(**inputs) 2025-08-26T20:38:05.1497547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1497624Z outputs = self.mobilebert( 2025-08-26T20:38:05.1497944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1498031Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1498331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1498415Z layer_outputs = layer_module( 2025-08-26T20:38:05.1498711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1498812Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1499119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1499241Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1499549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1499673Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1499677Z 2025-08-26T20:38:05.1499795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1500010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1500081Z return mod(**inputs) 2025-08-26T20:38:05.1500394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1500470Z outputs = self.mobilebert( 2025-08-26T20:38:05.1500774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1500855Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1501151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1501239Z layer_outputs = layer_module( 2025-08-26T20:38:05.1501567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1501710Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1501997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1502132Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1502414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1502501Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1502504Z 2025-08-26T20:38:05.1502616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1502820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1502898Z return mod(**inputs) 2025-08-26T20:38:05.1503206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1503296Z outputs = self.mobilebert( 2025-08-26T20:38:05.1503592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1503669Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1503974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1504069Z layer_outputs = layer_module( 2025-08-26T20:38:05.1504376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1504497Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1504797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1504936Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1505235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1505373Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1505673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1505777Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1505783Z 2025-08-26T20:38:05.1505893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1506103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1506180Z return mod(**inputs) 2025-08-26T20:38:05.1506464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1506543Z outputs = self.mobilebert( 2025-08-26T20:38:05.1506824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1506898Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1507191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1507263Z layer_outputs = layer_module( 2025-08-26T20:38:05.1507550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1507646Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1507954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1508071Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1508365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1508459Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1508463Z 2025-08-26T20:38:05.1508565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1508773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1508840Z return mod(**inputs) 2025-08-26T20:38:05.1509134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1509217Z outputs = self.mobilebert( 2025-08-26T20:38:05.1509522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1509607Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1509911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1509992Z layer_outputs = layer_module( 2025-08-26T20:38:05.1510291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1510393Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1510705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1510841Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1511144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1511276Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1511280Z 2025-08-26T20:38:05.1511383Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1511592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1511657Z return mod(**inputs) 2025-08-26T20:38:05.1511951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1512023Z outputs = self.mobilebert( 2025-08-26T20:38:05.1512334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1512413Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1512712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1512797Z layer_outputs = layer_module( 2025-08-26T20:38:05.1513093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1513201Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1513499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1513631Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1513936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1514027Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1514030Z 2025-08-26T20:38:05.1514144Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1514355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1514433Z return mod(**inputs) 2025-08-26T20:38:05.1514751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1514846Z outputs = self.mobilebert( 2025-08-26T20:38:05.1515153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1515230Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1515530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1515609Z layer_outputs = layer_module( 2025-08-26T20:38:05.1515909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1516020Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1516319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1516459Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1516758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1516895Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1517191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1517290Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1517313Z 2025-08-26T20:38:05.1517434Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1517644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1517742Z return mod(**inputs) 2025-08-26T20:38:05.1518059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1518136Z outputs = self.mobilebert( 2025-08-26T20:38:05.1518464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1518542Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1518848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1518923Z layer_outputs = layer_module( 2025-08-26T20:38:05.1519230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1519329Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1519691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1519829Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1520140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1520238Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1520242Z 2025-08-26T20:38:05.1520356Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1520574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1520654Z return mod(**inputs) 2025-08-26T20:38:05.1520967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1521053Z outputs = self.mobilebert( 2025-08-26T20:38:05.1521370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1521482Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1521799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1521878Z layer_outputs = layer_module( 2025-08-26T20:38:05.1522182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1522283Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1522584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1522705Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1523001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1523133Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1523137Z 2025-08-26T20:38:05.1523246Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1523466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1523537Z return mod(**inputs) 2025-08-26T20:38:05.1523844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1523920Z outputs = self.mobilebert( 2025-08-26T20:38:05.1524222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1524328Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1524625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1524729Z layer_outputs = layer_module( 2025-08-26T20:38:05.1525024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1525125Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1525427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1525562Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1525865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1525957Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1525961Z 2025-08-26T20:38:05.1526076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1526286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1526356Z return mod(**inputs) 2025-08-26T20:38:05.1526664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1526742Z outputs = self.mobilebert( 2025-08-26T20:38:05.1527047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1527124Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1527429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1527514Z layer_outputs = layer_module( 2025-08-26T20:38:05.1527818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1527926Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1528249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1528390Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1528710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1528831Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1529115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1529203Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1529208Z 2025-08-26T20:38:05.1529314Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1529507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1529572Z return mod(**inputs) 2025-08-26T20:38:05.1529855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1529925Z outputs = self.mobilebert( 2025-08-26T20:38:05.1530205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1530276Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1530552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1530622Z layer_outputs = layer_module( 2025-08-26T20:38:05.1530916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1531041Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1531332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1531422Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1531425Z 2025-08-26T20:38:05.1531526Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1531721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1531792Z return mod(**inputs) 2025-08-26T20:38:05.1532072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1532148Z outputs = self.mobilebert( 2025-08-26T20:38:05.1532422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1532499Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1532772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1532843Z layer_outputs = layer_module( 2025-08-26T20:38:05.1533125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1533243Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1533522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1533630Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1533633Z 2025-08-26T20:38:05.1533734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1533939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1534003Z return mod(**inputs) 2025-08-26T20:38:05.1534290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1534380Z outputs = self.mobilebert( 2025-08-26T20:38:05.1534679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1534753Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1535054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1535138Z layer_outputs = layer_module( 2025-08-26T20:38:05.1535495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1535663Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1535944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1536041Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1536056Z 2025-08-26T20:38:05.1536158Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1536359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1536433Z return mod(**inputs) 2025-08-26T20:38:05.1536718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1536797Z outputs = self.mobilebert( 2025-08-26T20:38:05.1537078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1537167Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1537454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1537542Z layer_outputs = layer_module( 2025-08-26T20:38:05.1537830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1537993Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1538277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1538409Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1538692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1538794Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1538798Z 2025-08-26T20:38:05.1538901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1539107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1539172Z return mod(**inputs) 2025-08-26T20:38:05.1539459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1539542Z outputs = self.mobilebert( 2025-08-26T20:38:05.1539823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1539903Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1540189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1540263Z layer_outputs = layer_module( 2025-08-26T20:38:05.1540561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1540719Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1541018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1541158Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1541441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1541525Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1541529Z 2025-08-26T20:38:05.1541631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1541837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1541905Z return mod(**inputs) 2025-08-26T20:38:05.1542200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1542273Z outputs = self.mobilebert( 2025-08-26T20:38:05.1542555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1542636Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1542971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1543054Z layer_outputs = layer_module( 2025-08-26T20:38:05.1543363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1543535Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1543850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1544001Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1544308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1544431Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1544713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1544804Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1544807Z 2025-08-26T20:38:05.1544915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1545112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1545179Z return mod(**inputs) 2025-08-26T20:38:05.1545466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1545538Z outputs = self.mobilebert( 2025-08-26T20:38:05.1545826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1545899Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1546184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1546259Z layer_outputs = layer_module( 2025-08-26T20:38:05.1546532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1546703Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1546986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1547106Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1547407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1547494Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1547498Z 2025-08-26T20:38:05.1547624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1547827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1547903Z return mod(**inputs) 2025-08-26T20:38:05.1548214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1548289Z outputs = self.mobilebert( 2025-08-26T20:38:05.1548597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1548674Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1548981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1549067Z layer_outputs = layer_module( 2025-08-26T20:38:05.1549351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1549514Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1549799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1549918Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1550219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1550313Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1550619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1550713Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1550723Z 2025-08-26T20:38:05.1550830Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1551039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1551117Z return mod(**inputs) 2025-08-26T20:38:05.1551418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1551505Z outputs = self.mobilebert( 2025-08-26T20:38:05.1551812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1551891Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1552198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1552278Z layer_outputs = layer_module( 2025-08-26T20:38:05.1552584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1552677Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1552974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1553059Z self_outputs = self.self( 2025-08-26T20:38:05.1553355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1553449Z self.query(query_tensor) 2025-08-26T20:38:05.1553453Z 2025-08-26T20:38:05.1553564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1553769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1553836Z return mod(**inputs) 2025-08-26T20:38:05.1554137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1554236Z outputs = self.mobilebert( 2025-08-26T20:38:05.1554520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1554599Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1554878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1554953Z layer_outputs = layer_module( 2025-08-26T20:38:05.1555247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1555339Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1555650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1555725Z self_outputs = self.self( 2025-08-26T20:38:05.1556026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1556107Z self.key(key_tensor) 2025-08-26T20:38:05.1556111Z 2025-08-26T20:38:05.1556221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1556440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1556509Z return mod(**inputs) 2025-08-26T20:38:05.1556834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1556911Z outputs = self.mobilebert( 2025-08-26T20:38:05.1557230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1557316Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1557616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1557699Z layer_outputs = layer_module( 2025-08-26T20:38:05.1557997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1558087Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1558391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1558469Z self_outputs = self.self( 2025-08-26T20:38:05.1558770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1558850Z self.value(value_tensor) 2025-08-26T20:38:05.1558853Z 2025-08-26T20:38:05.1558949Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1559035Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1559149Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1559370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1559508Z return mod(**inputs) 2025-08-26T20:38:05.1559838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1559916Z outputs = self.mobilebert( 2025-08-26T20:38:05.1560230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1560319Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1560616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1560722Z layer_outputs = layer_module( 2025-08-26T20:38:05.1561031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1561119Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1561410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1561535Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1561822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1561911Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1561915Z 2025-08-26T20:38:05.1562027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1562227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1562294Z return mod(**inputs) 2025-08-26T20:38:05.1562591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1562662Z outputs = self.mobilebert( 2025-08-26T20:38:05.1562950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1563023Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1563305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1563400Z layer_outputs = layer_module( 2025-08-26T20:38:05.1563703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1563940Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1564244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1564365Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1564761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1564867Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1564872Z 2025-08-26T20:38:05.1565016Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1565302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1565414Z return mod(**inputs) 2025-08-26T20:38:05.1565846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1565952Z outputs = self.mobilebert( 2025-08-26T20:38:05.1566293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1566374Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1566657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1566736Z layer_outputs = layer_module( 2025-08-26T20:38:05.1567049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1567139Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1567439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1567582Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1567915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1568062Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1568384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1568486Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1568498Z 2025-08-26T20:38:05.1568606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1568817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1568896Z return mod(**inputs) 2025-08-26T20:38:05.1569191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1569275Z outputs = self.mobilebert( 2025-08-26T20:38:05.1569573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1569651Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1569954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1570031Z layer_outputs = layer_module( 2025-08-26T20:38:05.1570332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1570435Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1570765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1570893Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1571214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1571312Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1571316Z 2025-08-26T20:38:05.1571426Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1571644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1571714Z return mod(**inputs) 2025-08-26T20:38:05.1572014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1572097Z outputs = self.mobilebert( 2025-08-26T20:38:05.1572401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1572487Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1572783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1572862Z layer_outputs = layer_module( 2025-08-26T20:38:05.1573173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1573277Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1573588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1573723Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1574027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1574150Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1574154Z 2025-08-26T20:38:05.1574261Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1574483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1574571Z return mod(**inputs) 2025-08-26T20:38:05.1574898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1574978Z outputs = self.mobilebert( 2025-08-26T20:38:05.1575276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1575361Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1575664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1575744Z layer_outputs = layer_module( 2025-08-26T20:38:05.1576025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1576129Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1576410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1576538Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1576833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1576922Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1576926Z 2025-08-26T20:38:05.1577045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1577341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1577475Z return mod(**inputs) 2025-08-26T20:38:05.1577937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1578058Z outputs = self.mobilebert( 2025-08-26T20:38:05.1578494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1578596Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1578908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1578986Z layer_outputs = layer_module( 2025-08-26T20:38:05.1579352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1579461Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1579777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1579920Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1580236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1580370Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1580697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1580795Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1580799Z 2025-08-26T20:38:05.1580917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1581128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1581208Z return mod(**inputs) 2025-08-26T20:38:05.1581513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1581591Z outputs = self.mobilebert( 2025-08-26T20:38:05.1581918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1581997Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1582314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1582392Z layer_outputs = layer_module( 2025-08-26T20:38:05.1582686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1582794Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1583088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1583222Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1583528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1583631Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1583634Z 2025-08-26T20:38:05.1583749Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1583964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1584043Z return mod(**inputs) 2025-08-26T20:38:05.1584350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1584438Z outputs = self.mobilebert( 2025-08-26T20:38:05.1584739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1584839Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1585160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1585258Z layer_outputs = layer_module( 2025-08-26T20:38:05.1585568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1585671Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1585986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1586108Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1586423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1586553Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1586557Z 2025-08-26T20:38:05.1586667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1586894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1586969Z return mod(**inputs) 2025-08-26T20:38:05.1587291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1587378Z outputs = self.mobilebert( 2025-08-26T20:38:05.1587692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1587779Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1588096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1588185Z layer_outputs = layer_module( 2025-08-26T20:38:05.1588502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1588606Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1588943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1589097Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1589411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1589514Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1589518Z 2025-08-26T20:38:05.1589634Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1589844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1589918Z return mod(**inputs) 2025-08-26T20:38:05.1590225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1590303Z outputs = self.mobilebert( 2025-08-26T20:38:05.1590613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1590690Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1590998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1591087Z layer_outputs = layer_module( 2025-08-26T20:38:05.1591393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1591504Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1591834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1591968Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1592300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1592432Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1592747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1592848Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1592852Z 2025-08-26T20:38:05.1592970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1593188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1593261Z return mod(**inputs) 2025-08-26T20:38:05.1593591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1593671Z outputs = self.mobilebert( 2025-08-26T20:38:05.1593989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1594066Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1594366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1594451Z layer_outputs = layer_module( 2025-08-26T20:38:05.1594754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1594864Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1595176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1595309Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1595620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1595728Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1595732Z 2025-08-26T20:38:05.1595854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1596088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1596358Z return mod(**inputs) 2025-08-26T20:38:05.1596802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1596890Z outputs = self.mobilebert( 2025-08-26T20:38:05.1597206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1597290Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1597603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1597685Z layer_outputs = layer_module( 2025-08-26T20:38:05.1597999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1598103Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1598409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1598540Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1598845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1599034Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1599038Z 2025-08-26T20:38:05.1599150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1599408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1599537Z return mod(**inputs) 2025-08-26T20:38:05.1599855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1599944Z outputs = self.mobilebert( 2025-08-26T20:38:05.1600252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1600341Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1600648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1600727Z layer_outputs = layer_module( 2025-08-26T20:38:05.1601042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1601146Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1601461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1601598Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1601906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1602006Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1602010Z 2025-08-26T20:38:05.1602122Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1602347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1602421Z return mod(**inputs) 2025-08-26T20:38:05.1602737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1602817Z outputs = self.mobilebert( 2025-08-26T20:38:05.1603155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1603246Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1603598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1603686Z layer_outputs = layer_module( 2025-08-26T20:38:05.1603992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1604097Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1604413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1604549Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1604867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1604998Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1605310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1605412Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1605416Z 2025-08-26T20:38:05.1605529Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1605753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1605846Z return mod(**inputs) 2025-08-26T20:38:05.1606162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1606257Z outputs = self.mobilebert( 2025-08-26T20:38:05.1606564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1606654Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1606962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1607049Z layer_outputs = layer_module( 2025-08-26T20:38:05.1607355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1607483Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1607766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1607852Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1607857Z 2025-08-26T20:38:05.1607969Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1608178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1608255Z return mod(**inputs) 2025-08-26T20:38:05.1608556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1608632Z outputs = self.mobilebert( 2025-08-26T20:38:05.1608934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1609007Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1609295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1609369Z layer_outputs = layer_module( 2025-08-26T20:38:05.1609658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1609802Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1610106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1610227Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1610230Z 2025-08-26T20:38:05.1610330Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1610530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1610593Z return mod(**inputs) 2025-08-26T20:38:05.1610874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1610955Z outputs = self.mobilebert( 2025-08-26T20:38:05.1611225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1611304Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1611574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1611654Z layer_outputs = layer_module( 2025-08-26T20:38:05.1611932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1612091Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1612379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1612491Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1612495Z 2025-08-26T20:38:05.1612605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1612822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1612890Z return mod(**inputs) 2025-08-26T20:38:05.1613183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1613254Z outputs = self.mobilebert( 2025-08-26T20:38:05.1613539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1613611Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1613899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1613972Z layer_outputs = layer_module( 2025-08-26T20:38:05.1614263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1614428Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1614700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1614827Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1615103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1615201Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1615205Z 2025-08-26T20:38:05.1615307Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1615505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1615580Z return mod(**inputs) 2025-08-26T20:38:05.1615878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1615962Z outputs = self.mobilebert( 2025-08-26T20:38:05.1616277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1616371Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1616678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1616755Z layer_outputs = layer_module( 2025-08-26T20:38:05.1617058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1617215Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1617503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1617629Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1617911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1618006Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1618010Z 2025-08-26T20:38:05.1618115Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1618338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1618411Z return mod(**inputs) 2025-08-26T20:38:05.1618718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1618834Z outputs = self.mobilebert( 2025-08-26T20:38:05.1619129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1619236Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1619537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1619620Z layer_outputs = layer_module( 2025-08-26T20:38:05.1619917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1620085Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1620376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1620497Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1620779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1620900Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1621173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1621270Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1621275Z 2025-08-26T20:38:05.1621378Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1621586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1621651Z return mod(**inputs) 2025-08-26T20:38:05.1621942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1622016Z outputs = self.mobilebert( 2025-08-26T20:38:05.1622294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1622377Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1622674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1622754Z layer_outputs = layer_module( 2025-08-26T20:38:05.1623067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1623242Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1623521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1623629Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1623913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1623995Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1624000Z 2025-08-26T20:38:05.1624106Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1624302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1624368Z return mod(**inputs) 2025-08-26T20:38:05.1624661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1624735Z outputs = self.mobilebert( 2025-08-26T20:38:05.1625023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1625095Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1625390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1625470Z layer_outputs = layer_module( 2025-08-26T20:38:05.1625770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1625940Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1626223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1626340Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1626621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1626708Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1627000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1627094Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1627099Z 2025-08-26T20:38:05.1627209Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1627408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1627483Z return mod(**inputs) 2025-08-26T20:38:05.1627764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1627837Z outputs = self.mobilebert( 2025-08-26T20:38:05.1628122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1628194Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1628482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1628556Z layer_outputs = layer_module( 2025-08-26T20:38:05.1628832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1628929Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1629233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1629332Z self_outputs = self.self( 2025-08-26T20:38:05.1629614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1629688Z self.query(query_tensor) 2025-08-26T20:38:05.1629698Z 2025-08-26T20:38:05.1629803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1630001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1630078Z return mod(**inputs) 2025-08-26T20:38:05.1630360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1630439Z outputs = self.mobilebert( 2025-08-26T20:38:05.1630721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1630793Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1631084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1631155Z layer_outputs = layer_module( 2025-08-26T20:38:05.1631440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1631525Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1631821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1631900Z self_outputs = self.self( 2025-08-26T20:38:05.1632193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1632272Z self.key(key_tensor) 2025-08-26T20:38:05.1632276Z 2025-08-26T20:38:05.1632380Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1632587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1632653Z return mod(**inputs) 2025-08-26T20:38:05.1632936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1633016Z outputs = self.mobilebert( 2025-08-26T20:38:05.1633299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1633381Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1633663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1633740Z layer_outputs = layer_module( 2025-08-26T20:38:05.1634046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1634138Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1634446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1634523Z self_outputs = self.self( 2025-08-26T20:38:05.1634817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1634901Z self.value(value_tensor) 2025-08-26T20:38:05.1634904Z 2025-08-26T20:38:05.1634992Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1635086Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1635199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1635456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1635531Z return mod(**inputs) 2025-08-26T20:38:05.1635858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1635946Z outputs = self.mobilebert( 2025-08-26T20:38:05.1636245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1636330Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1636635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1636713Z layer_outputs = layer_module( 2025-08-26T20:38:05.1637016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1637109Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1637420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1637554Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1637869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1637960Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1637964Z 2025-08-26T20:38:05.1638073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1638311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1638381Z return mod(**inputs) 2025-08-26T20:38:05.1638709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1638809Z outputs = self.mobilebert( 2025-08-26T20:38:05.1639124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1639212Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1639598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1639691Z layer_outputs = layer_module( 2025-08-26T20:38:05.1640008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1640198Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1640520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1640646Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1640961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1641053Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1641058Z 2025-08-26T20:38:05.1641180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1641411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1641486Z return mod(**inputs) 2025-08-26T20:38:05.1641816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1641896Z outputs = self.mobilebert( 2025-08-26T20:38:05.1642214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1642295Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1642648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1642729Z layer_outputs = layer_module( 2025-08-26T20:38:05.1643064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1643170Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1643489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1643634Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1643952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1644091Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1644424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1644524Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1644528Z 2025-08-26T20:38:05.1644653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1644868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1644948Z return mod(**inputs) 2025-08-26T20:38:05.1645270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1645350Z outputs = self.mobilebert( 2025-08-26T20:38:05.1645686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1645767Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1646098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1646175Z layer_outputs = layer_module( 2025-08-26T20:38:05.1646491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1646606Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1646930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1647062Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1647368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1647459Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1647464Z 2025-08-26T20:38:05.1647564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1647762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1647838Z return mod(**inputs) 2025-08-26T20:38:05.1648128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1648208Z outputs = self.mobilebert( 2025-08-26T20:38:05.1648490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1648562Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1648856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1648929Z layer_outputs = layer_module( 2025-08-26T20:38:05.1649224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1649321Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1649628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1649758Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1650042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1650163Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1650167Z 2025-08-26T20:38:05.1650271Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1650484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1650550Z return mod(**inputs) 2025-08-26T20:38:05.1650827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1650907Z outputs = self.mobilebert( 2025-08-26T20:38:05.1651178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1651259Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1651531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1651610Z layer_outputs = layer_module( 2025-08-26T20:38:05.1651886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1651997Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1652277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1652433Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1652718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1652806Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1652810Z 2025-08-26T20:38:05.1652914Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1653120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1653187Z return mod(**inputs) 2025-08-26T20:38:05.1653478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1653557Z outputs = self.mobilebert( 2025-08-26T20:38:05.1653861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1653941Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1654243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1654323Z layer_outputs = layer_module( 2025-08-26T20:38:05.1654602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1654707Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1654986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1655111Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1655410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1655526Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1655821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1655912Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1655915Z 2025-08-26T20:38:05.1656037Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1656229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1656294Z return mod(**inputs) 2025-08-26T20:38:05.1656579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1656651Z outputs = self.mobilebert( 2025-08-26T20:38:05.1656928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1656999Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1657275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1657353Z layer_outputs = layer_module( 2025-08-26T20:38:05.1657625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1657727Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1657997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1658114Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1658385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1658489Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1658493Z 2025-08-26T20:38:05.1658623Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1658823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1658896Z return mod(**inputs) 2025-08-26T20:38:05.1659181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1659255Z outputs = self.mobilebert( 2025-08-26T20:38:05.1659545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1659618Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1659909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1659981Z layer_outputs = layer_module( 2025-08-26T20:38:05.1660262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1660356Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1660628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1660747Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1661027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1661148Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1661152Z 2025-08-26T20:38:05.1661253Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1661452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1661525Z return mod(**inputs) 2025-08-26T20:38:05.1661809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1661892Z outputs = self.mobilebert( 2025-08-26T20:38:05.1662190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1662290Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1662573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1662646Z layer_outputs = layer_module( 2025-08-26T20:38:05.1662932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1663029Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1663315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1663440Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1663718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1663811Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1663815Z 2025-08-26T20:38:05.1663916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1664120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1664185Z return mod(**inputs) 2025-08-26T20:38:05.1664475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1664569Z outputs = self.mobilebert( 2025-08-26T20:38:05.1664848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1664949Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1665231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1665313Z layer_outputs = layer_module( 2025-08-26T20:38:05.1665599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1665697Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1665987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1666115Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1666407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1666532Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1666827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1666924Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1666927Z 2025-08-26T20:38:05.1667035Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1667245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1667314Z return mod(**inputs) 2025-08-26T20:38:05.1667611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1667692Z outputs = self.mobilebert( 2025-08-26T20:38:05.1667994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1668084Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1668387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1668488Z layer_outputs = layer_module( 2025-08-26T20:38:05.1668789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1668893Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1669173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1669287Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1669584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1669677Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1669681Z 2025-08-26T20:38:05.1669803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1670016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1670088Z return mod(**inputs) 2025-08-26T20:38:05.1670407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1670486Z outputs = self.mobilebert( 2025-08-26T20:38:05.1670797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1670875Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1671194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1671325Z layer_outputs = layer_module( 2025-08-26T20:38:05.1671636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1671763Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1672059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1672192Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1672474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1672585Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1672598Z 2025-08-26T20:38:05.1672702Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1672903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1672978Z return mod(**inputs) 2025-08-26T20:38:05.1673263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1673345Z outputs = self.mobilebert( 2025-08-26T20:38:05.1673645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1673724Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1674032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1674108Z layer_outputs = layer_module( 2025-08-26T20:38:05.1674423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1674526Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1674825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1674965Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1675276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1675376Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1675397Z 2025-08-26T20:38:05.1675510Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1675727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1675797Z return mod(**inputs) 2025-08-26T20:38:05.1676108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1676194Z outputs = self.mobilebert( 2025-08-26T20:38:05.1676490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1676579Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1676881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1676958Z layer_outputs = layer_module( 2025-08-26T20:38:05.1677265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1677365Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1677668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1677799Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1678128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1678256Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1678570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1678677Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1678681Z 2025-08-26T20:38:05.1678790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1679007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1679076Z return mod(**inputs) 2025-08-26T20:38:05.1679374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1679546Z outputs = self.mobilebert( 2025-08-26T20:38:05.1679853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1679939Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1680246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1680333Z layer_outputs = layer_module( 2025-08-26T20:38:05.1680638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1680774Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1681081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1681174Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1681178Z 2025-08-26T20:38:05.1681296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1681510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1681583Z return mod(**inputs) 2025-08-26T20:38:05.1681908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1682006Z outputs = self.mobilebert( 2025-08-26T20:38:05.1682332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1682414Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1682718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1682795Z layer_outputs = layer_module( 2025-08-26T20:38:05.1683092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1683228Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1683520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1683648Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1683653Z 2025-08-26T20:38:05.1683763Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1683974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1684052Z return mod(**inputs) 2025-08-26T20:38:05.1684349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1684433Z outputs = self.mobilebert( 2025-08-26T20:38:05.1684730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1684833Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1685128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1685222Z layer_outputs = layer_module( 2025-08-26T20:38:05.1685549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1685721Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1686024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1686125Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1686129Z 2025-08-26T20:38:05.1686240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1686457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1686527Z return mod(**inputs) 2025-08-26T20:38:05.1686832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1686911Z outputs = self.mobilebert( 2025-08-26T20:38:05.1687217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1687297Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1687598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1687682Z layer_outputs = layer_module( 2025-08-26T20:38:05.1687977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1688153Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1688452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1688586Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1688907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1689023Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1689027Z 2025-08-26T20:38:05.1689147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1689355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1689433Z return mod(**inputs) 2025-08-26T20:38:05.1689755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1689833Z outputs = self.mobilebert( 2025-08-26T20:38:05.1690131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1690210Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1690514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1690590Z layer_outputs = layer_module( 2025-08-26T20:38:05.1690888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1691061Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1691370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1691507Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1691828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1691945Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1691948Z 2025-08-26T20:38:05.1692058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1692267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1692346Z return mod(**inputs) 2025-08-26T20:38:05.1692648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1692732Z outputs = self.mobilebert( 2025-08-26T20:38:05.1693039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1693116Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1693420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1693497Z layer_outputs = layer_module( 2025-08-26T20:38:05.1693804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1693970Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1694275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1694406Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1694700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1694839Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1695138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1695246Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1695249Z 2025-08-26T20:38:05.1695358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1695593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1695666Z return mod(**inputs) 2025-08-26T20:38:05.1695982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1696067Z outputs = self.mobilebert( 2025-08-26T20:38:05.1696733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1696846Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1697151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1697228Z layer_outputs = layer_module( 2025-08-26T20:38:05.1697532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1697709Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1698014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1698134Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1698438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1698529Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1698593Z 2025-08-26T20:38:05.1698705Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1698922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1699020Z return mod(**inputs) 2025-08-26T20:38:05.1699329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1699406Z outputs = self.mobilebert( 2025-08-26T20:38:05.1699706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1699793Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1700093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1700177Z layer_outputs = layer_module( 2025-08-26T20:38:05.1700477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1700657Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1700959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1701076Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1701383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1701477Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1701789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1701888Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1701891Z 2025-08-26T20:38:05.1702002Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1702223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1702295Z return mod(**inputs) 2025-08-26T20:38:05.1702604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1702706Z outputs = self.mobilebert( 2025-08-26T20:38:05.1703051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1703133Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1703430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1703515Z layer_outputs = layer_module( 2025-08-26T20:38:05.1703809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1703909Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1704208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1704288Z self_outputs = self.self( 2025-08-26T20:38:05.1704601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1704681Z self.query(query_tensor) 2025-08-26T20:38:05.1704687Z 2025-08-26T20:38:05.1704809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1705035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1705115Z return mod(**inputs) 2025-08-26T20:38:05.1705416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1705512Z outputs = self.mobilebert( 2025-08-26T20:38:05.1705816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1705914Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1706221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1706296Z layer_outputs = layer_module( 2025-08-26T20:38:05.1706589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1706684Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1706962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1707040Z self_outputs = self.self( 2025-08-26T20:38:05.1707318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1707389Z self.key(key_tensor) 2025-08-26T20:38:05.1707399Z 2025-08-26T20:38:05.1707505Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1707704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1707777Z return mod(**inputs) 2025-08-26T20:38:05.1708060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1708144Z outputs = self.mobilebert( 2025-08-26T20:38:05.1708440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1708516Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1708819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1708897Z layer_outputs = layer_module( 2025-08-26T20:38:05.1709199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1709291Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1709606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1709709Z self_outputs = self.self( 2025-08-26T20:38:05.1710006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1710090Z self.value(value_tensor) 2025-08-26T20:38:05.1710094Z 2025-08-26T20:38:05.1710182Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1710274Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1710387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1710599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1710677Z return mod(**inputs) 2025-08-26T20:38:05.1710983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1711067Z outputs = self.mobilebert( 2025-08-26T20:38:05.1711373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1711447Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1711733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1711804Z layer_outputs = layer_module( 2025-08-26T20:38:05.1712091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1712193Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1712473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1712628Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1712924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1713026Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1713029Z 2025-08-26T20:38:05.1713136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1713351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1713421Z return mod(**inputs) 2025-08-26T20:38:05.1713718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1713804Z outputs = self.mobilebert( 2025-08-26T20:38:05.1714096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1714182Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1714479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1714557Z layer_outputs = layer_module( 2025-08-26T20:38:05.1714861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1715033Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1715338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1715459Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1715758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1715846Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1715850Z 2025-08-26T20:38:05.1715975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1716194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1716281Z return mod(**inputs) 2025-08-26T20:38:05.1716593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1716672Z outputs = self.mobilebert( 2025-08-26T20:38:05.1716970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1717059Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1717362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1717451Z layer_outputs = layer_module( 2025-08-26T20:38:05.1717757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1717855Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1718164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1718301Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1718616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1718753Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1719087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1719189Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1719211Z 2025-08-26T20:38:05.1719331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1719613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1719691Z return mod(**inputs) 2025-08-26T20:38:05.1720009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1720091Z outputs = self.mobilebert( 2025-08-26T20:38:05.1720416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1720495Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1720812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1720899Z layer_outputs = layer_module( 2025-08-26T20:38:05.1721214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1721328Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1721644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1721770Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1722095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1722188Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1722192Z 2025-08-26T20:38:05.1722313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1722531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1722613Z return mod(**inputs) 2025-08-26T20:38:05.1722938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1723046Z outputs = self.mobilebert( 2025-08-26T20:38:05.1724245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1724339Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1724653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1724730Z layer_outputs = layer_module( 2025-08-26T20:38:05.1725046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1725161Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1725473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1725606Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1725923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1726055Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1726059Z 2025-08-26T20:38:05.1726172Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1726404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1726484Z return mod(**inputs) 2025-08-26T20:38:05.1726802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1726913Z outputs = self.mobilebert( 2025-08-26T20:38:05.1727228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1727326Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1727682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1727754Z layer_outputs = layer_module( 2025-08-26T20:38:05.1728041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1728134Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1728456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1728584Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1728871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1728965Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1728969Z 2025-08-26T20:38:05.1729085Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1729288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1729355Z return mod(**inputs) 2025-08-26T20:38:05.1729635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1729713Z outputs = self.mobilebert( 2025-08-26T20:38:05.1729990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1730070Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1730349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1730433Z layer_outputs = layer_module( 2025-08-26T20:38:05.1730779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1730881Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1731195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1731323Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1731610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1731731Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1732016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1732108Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1732113Z 2025-08-26T20:38:05.1732217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1732423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1732488Z return mod(**inputs) 2025-08-26T20:38:05.1732775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1732847Z outputs = self.mobilebert( 2025-08-26T20:38:05.1733127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1733207Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1733505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1733583Z layer_outputs = layer_module( 2025-08-26T20:38:05.1733882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1733979Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1734268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1734382Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1734670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1734771Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1734774Z 2025-08-26T20:38:05.1734880Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1735076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1735141Z return mod(**inputs) 2025-08-26T20:38:05.1735426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1735497Z outputs = self.mobilebert( 2025-08-26T20:38:05.1735775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1735848Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1736126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1736206Z layer_outputs = layer_module( 2025-08-26T20:38:05.1736485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1736588Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1736867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1736989Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1737291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1737426Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1737430Z 2025-08-26T20:38:05.1737543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1737743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1737817Z return mod(**inputs) 2025-08-26T20:38:05.1738121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1738201Z outputs = self.mobilebert( 2025-08-26T20:38:05.1738503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1738582Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1738889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1738967Z layer_outputs = layer_module( 2025-08-26T20:38:05.1739271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1739371Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1739666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1739808Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1740125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1740241Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1740245Z 2025-08-26T20:38:05.1740357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1740577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1740650Z return mod(**inputs) 2025-08-26T20:38:05.1740953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1741037Z outputs = self.mobilebert( 2025-08-26T20:38:05.1741333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1741417Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1741718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1741792Z layer_outputs = layer_module( 2025-08-26T20:38:05.1742096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1742195Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1742502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1742636Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1742931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1743068Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1743363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1743465Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1743470Z 2025-08-26T20:38:05.1743571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1743797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1743865Z return mod(**inputs) 2025-08-26T20:38:05.1744167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1744251Z outputs = self.mobilebert( 2025-08-26T20:38:05.1744533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1744612Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1744894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1744967Z layer_outputs = layer_module( 2025-08-26T20:38:05.1745261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1745354Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1745639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1745753Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1746039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1746123Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1746127Z 2025-08-26T20:38:05.1746233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1746475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1746546Z return mod(**inputs) 2025-08-26T20:38:05.1746876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1746956Z outputs = self.mobilebert( 2025-08-26T20:38:05.1747253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1747338Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1747632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1747716Z layer_outputs = layer_module( 2025-08-26T20:38:05.1748008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1748116Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1748410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1748529Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1748840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1748954Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1748957Z 2025-08-26T20:38:05.1749067Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1749262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1749336Z return mod(**inputs) 2025-08-26T20:38:05.1749618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1749692Z outputs = self.mobilebert( 2025-08-26T20:38:05.1749975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1750052Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1750371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1750449Z layer_outputs = layer_module( 2025-08-26T20:38:05.1750770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1750874Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1751154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1751286Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1751568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1751662Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1751665Z 2025-08-26T20:38:05.1751768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1751967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1752044Z return mod(**inputs) 2025-08-26T20:38:05.1752326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1752407Z outputs = self.mobilebert( 2025-08-26T20:38:05.1752689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1752761Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1753063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1753137Z layer_outputs = layer_module( 2025-08-26T20:38:05.1753473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1753574Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1753873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1754015Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1754309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1754446Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1754743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1754848Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1754854Z 2025-08-26T20:38:05.1754964Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1755177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1755256Z return mod(**inputs) 2025-08-26T20:38:05.1755558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1755641Z outputs = self.mobilebert( 2025-08-26T20:38:05.1755936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1756019Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1756313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1756390Z layer_outputs = layer_module( 2025-08-26T20:38:05.1756692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1756838Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1757164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1757256Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1757259Z 2025-08-26T20:38:05.1757370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1757586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1757658Z return mod(**inputs) 2025-08-26T20:38:05.1757961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1758041Z outputs = self.mobilebert( 2025-08-26T20:38:05.1758350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1758451Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1758756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1758842Z layer_outputs = layer_module( 2025-08-26T20:38:05.1759143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1759282Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1759657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1759810Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1759815Z 2025-08-26T20:38:05.1759938Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1760180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1760263Z return mod(**inputs) 2025-08-26T20:38:05.1760574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1760662Z outputs = self.mobilebert( 2025-08-26T20:38:05.1760972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1761052Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1761368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1761448Z layer_outputs = layer_module( 2025-08-26T20:38:05.1761761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1761925Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1762208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1762315Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1762319Z 2025-08-26T20:38:05.1762421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1762629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1762697Z return mod(**inputs) 2025-08-26T20:38:05.1763016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1763097Z outputs = self.mobilebert( 2025-08-26T20:38:05.1763402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1763490Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1763820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1763908Z layer_outputs = layer_module( 2025-08-26T20:38:05.1764233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1764408Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1764723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1764860Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1765174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1765277Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1765280Z 2025-08-26T20:38:05.1765399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1765615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1765688Z return mod(**inputs) 2025-08-26T20:38:05.1766001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1766079Z outputs = self.mobilebert( 2025-08-26T20:38:05.1766390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1766490Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1766791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1766895Z layer_outputs = layer_module( 2025-08-26T20:38:05.1767205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1767383Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1767693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1767838Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1768144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1768240Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1768246Z 2025-08-26T20:38:05.1768367Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1768583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1768667Z return mod(**inputs) 2025-08-26T20:38:05.1768979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1769059Z outputs = self.mobilebert( 2025-08-26T20:38:05.1769377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1769461Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1769778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1769857Z layer_outputs = layer_module( 2025-08-26T20:38:05.1770172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1770332Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1770631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1770764Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1771059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1771189Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1771469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1771562Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1771573Z 2025-08-26T20:38:05.1771676Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1771873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1771950Z return mod(**inputs) 2025-08-26T20:38:05.1772233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1772314Z outputs = self.mobilebert( 2025-08-26T20:38:05.1772595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1772670Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1772958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1773030Z layer_outputs = layer_module( 2025-08-26T20:38:05.1773316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1773497Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1773806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1773932Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1774234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1774331Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1774335Z 2025-08-26T20:38:05.1774451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1774655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1774721Z return mod(**inputs) 2025-08-26T20:38:05.1775007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1775091Z outputs = self.mobilebert( 2025-08-26T20:38:05.1775387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1775473Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1775774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1775851Z layer_outputs = layer_module( 2025-08-26T20:38:05.1776153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1776324Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1776632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1776751Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1777059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1777165Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1777490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1777594Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1777597Z 2025-08-26T20:38:05.1777699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1777904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1777971Z return mod(**inputs) 2025-08-26T20:38:05.1778266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1778341Z outputs = self.mobilebert( 2025-08-26T20:38:05.1778622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1778705Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1778999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1779085Z layer_outputs = layer_module( 2025-08-26T20:38:05.1779384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1779477Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1779783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1779889Z self_outputs = self.self( 2025-08-26T20:38:05.1780194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1780287Z self.query(query_tensor) 2025-08-26T20:38:05.1780291Z 2025-08-26T20:38:05.1780408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1780623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1780703Z return mod(**inputs) 2025-08-26T20:38:05.1780998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1781068Z outputs = self.mobilebert( 2025-08-26T20:38:05.1781368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1781445Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1781744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1781828Z layer_outputs = layer_module( 2025-08-26T20:38:05.1782135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1782228Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1782521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1782597Z self_outputs = self.self( 2025-08-26T20:38:05.1782904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1782975Z self.key(key_tensor) 2025-08-26T20:38:05.1782978Z 2025-08-26T20:38:05.1783093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1783306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1783383Z return mod(**inputs) 2025-08-26T20:38:05.1783688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1783780Z outputs = self.mobilebert( 2025-08-26T20:38:05.1784101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1784182Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1784483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1784557Z layer_outputs = layer_module( 2025-08-26T20:38:05.1784856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1784956Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1785251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1785333Z self_outputs = self.self( 2025-08-26T20:38:05.1785627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1785708Z self.value(value_tensor) 2025-08-26T20:38:05.1785713Z 2025-08-26T20:38:05.1785802Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1785885Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1786001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1786212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1786290Z return mod(**inputs) 2025-08-26T20:38:05.1786589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1786686Z outputs = self.mobilebert( 2025-08-26T20:38:05.1786992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1787087Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1787392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1787470Z layer_outputs = layer_module( 2025-08-26T20:38:05.1787765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1787862Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1788158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1788299Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1788594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1788696Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1788700Z 2025-08-26T20:38:05.1788810Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1789019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1789099Z return mod(**inputs) 2025-08-26T20:38:05.1789398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1789482Z outputs = self.mobilebert( 2025-08-26T20:38:05.1789775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1789853Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1790164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1790241Z layer_outputs = layer_module( 2025-08-26T20:38:05.1790564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1790752Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1791061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1791179Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1791478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1791577Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1791581Z 2025-08-26T20:38:05.1791689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1791905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1791976Z return mod(**inputs) 2025-08-26T20:38:05.1792276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1792361Z outputs = self.mobilebert( 2025-08-26T20:38:05.1792658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1792742Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1793040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1793121Z layer_outputs = layer_module( 2025-08-26T20:38:05.1793460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1793549Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1793871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1794003Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1794308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1794442Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1794736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1794845Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1794851Z 2025-08-26T20:38:05.1794959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1795177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1795249Z return mod(**inputs) 2025-08-26T20:38:05.1795558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1795637Z outputs = self.mobilebert( 2025-08-26T20:38:05.1795934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1796020Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1796658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1796793Z layer_outputs = layer_module( 2025-08-26T20:38:05.1797106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1797218Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1797534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1797714Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1798056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1798151Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1798154Z 2025-08-26T20:38:05.1798275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1798492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1798566Z return mod(**inputs) 2025-08-26T20:38:05.1798887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1798968Z outputs = self.mobilebert( 2025-08-26T20:38:05.1799282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1799364Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1799721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1799815Z layer_outputs = layer_module( 2025-08-26T20:38:05.1800123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1800234Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1800545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1800711Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1801019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1801172Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1801177Z 2025-08-26T20:38:05.1801299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1801520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1801602Z return mod(**inputs) 2025-08-26T20:38:05.1801913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1801991Z outputs = self.mobilebert( 2025-08-26T20:38:05.1802296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1802374Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1802679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1802758Z layer_outputs = layer_module( 2025-08-26T20:38:05.1803068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1803171Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1803476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1803620Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1803935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1804036Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1804041Z 2025-08-26T20:38:05.1804152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1804367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1804446Z return mod(**inputs) 2025-08-26T20:38:05.1804781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1804879Z outputs = self.mobilebert( 2025-08-26T20:38:05.1805189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1805273Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1805553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1805626Z layer_outputs = layer_module( 2025-08-26T20:38:05.1805915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1806009Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1806301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1806428Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1806711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1806841Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1807125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1807226Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1807249Z 2025-08-26T20:38:05.1807354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1807557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1807641Z return mod(**inputs) 2025-08-26T20:38:05.1807929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1808010Z outputs = self.mobilebert( 2025-08-26T20:38:05.1808292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1808372Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1808677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1808753Z layer_outputs = layer_module( 2025-08-26T20:38:05.1809064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1809165Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1809477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1809601Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1809917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1810005Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1810009Z 2025-08-26T20:38:05.1810116Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1810334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1810402Z return mod(**inputs) 2025-08-26T20:38:05.1810721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1810801Z outputs = self.mobilebert( 2025-08-26T20:38:05.1811105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1811191Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1811512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1811614Z layer_outputs = layer_module( 2025-08-26T20:38:05.1811922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1812031Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1812338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1812458Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1812763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1812889Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1812893Z 2025-08-26T20:38:05.1813012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1813231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1813303Z return mod(**inputs) 2025-08-26T20:38:05.1813626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1813701Z outputs = self.mobilebert( 2025-08-26T20:38:05.1814007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1814103Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1814410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1814502Z layer_outputs = layer_module( 2025-08-26T20:38:05.1814796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1814906Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1815203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1815342Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1815644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1815738Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1815752Z 2025-08-26T20:38:05.1815863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1816078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1816159Z return mod(**inputs) 2025-08-26T20:38:05.1816470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1816555Z outputs = self.mobilebert( 2025-08-26T20:38:05.1816870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1816946Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1817248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1817323Z layer_outputs = layer_module( 2025-08-26T20:38:05.1817626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1817725Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1818022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1818177Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1818494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1818633Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1818935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1819038Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1819042Z 2025-08-26T20:38:05.1819152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1819363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1819443Z return mod(**inputs) 2025-08-26T20:38:05.1819747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1819829Z outputs = self.mobilebert( 2025-08-26T20:38:05.1820130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1820206Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1820513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1820589Z layer_outputs = layer_module( 2025-08-26T20:38:05.1820895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1821016Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1821321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1821460Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1821760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1821858Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1821862Z 2025-08-26T20:38:05.1821968Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1822186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1822257Z return mod(**inputs) 2025-08-26T20:38:05.1822559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1822648Z outputs = self.mobilebert( 2025-08-26T20:38:05.1822953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1823041Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1823358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1823443Z layer_outputs = layer_module( 2025-08-26T20:38:05.1823744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1823843Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1824147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1824270Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1824575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1824696Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1824700Z 2025-08-26T20:38:05.1824825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1825062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1825135Z return mod(**inputs) 2025-08-26T20:38:05.1825444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1825524Z outputs = self.mobilebert( 2025-08-26T20:38:05.1825832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1825911Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1826214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1826299Z layer_outputs = layer_module( 2025-08-26T20:38:05.1826598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1826707Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1827006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1827139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1827456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1827546Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1827572Z 2025-08-26T20:38:05.1827689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1827899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1827996Z return mod(**inputs) 2025-08-26T20:38:05.1828302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1828379Z outputs = self.mobilebert( 2025-08-26T20:38:05.1828685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1828762Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1829067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1829142Z layer_outputs = layer_module( 2025-08-26T20:38:05.1829441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1829549Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1829849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1829989Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1830290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1830427Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1830725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1830822Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1830826Z 2025-08-26T20:38:05.1830947Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1831158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1831237Z return mod(**inputs) 2025-08-26T20:38:05.1831567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1831645Z outputs = self.mobilebert( 2025-08-26T20:38:05.1831966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1832047Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1832355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1832431Z layer_outputs = layer_module( 2025-08-26T20:38:05.1832735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1832866Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1833168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1833272Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1833275Z 2025-08-26T20:38:05.1833383Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1833605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1833674Z return mod(**inputs) 2025-08-26T20:38:05.1833985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1834069Z outputs = self.mobilebert( 2025-08-26T20:38:05.1834378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1834489Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1834793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1834898Z layer_outputs = layer_module( 2025-08-26T20:38:05.1835204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1835336Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1835646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1835768Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1835772Z 2025-08-26T20:38:05.1835890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1836114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1836184Z return mod(**inputs) 2025-08-26T20:38:05.1836488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1836565Z outputs = self.mobilebert( 2025-08-26T20:38:05.1836864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1836943Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1837245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1837320Z layer_outputs = layer_module( 2025-08-26T20:38:05.1837620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1837799Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1838102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1838217Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1838222Z 2025-08-26T20:38:05.1838354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1838569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1838666Z return mod(**inputs) 2025-08-26T20:38:05.1838975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1839062Z outputs = self.mobilebert( 2025-08-26T20:38:05.1839367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1839521Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1839836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1839917Z layer_outputs = layer_module( 2025-08-26T20:38:05.1840231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1840405Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1840719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1840857Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1841161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1841271Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1841299Z 2025-08-26T20:38:05.1841416Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1841643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1841737Z return mod(**inputs) 2025-08-26T20:38:05.1842056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1842137Z outputs = self.mobilebert( 2025-08-26T20:38:05.1842445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1842537Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1842843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1842930Z layer_outputs = layer_module( 2025-08-26T20:38:05.1843237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1843409Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1843725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1843862Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1844172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1844264Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1844267Z 2025-08-26T20:38:05.1844386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1844599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1844673Z return mod(**inputs) 2025-08-26T20:38:05.1844985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1845065Z outputs = self.mobilebert( 2025-08-26T20:38:05.1845400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1845482Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1845813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1845902Z layer_outputs = layer_module( 2025-08-26T20:38:05.1846205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1846384Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1846689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1846831Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1847138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1847271Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1847582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1847684Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1847687Z 2025-08-26T20:38:05.1847808Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1848020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1848100Z return mod(**inputs) 2025-08-26T20:38:05.1848429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1848509Z outputs = self.mobilebert( 2025-08-26T20:38:05.1848844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1848924Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1849235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1849314Z layer_outputs = layer_module( 2025-08-26T20:38:05.1849633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1849822Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1850131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1850262Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1850569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1850669Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1850673Z 2025-08-26T20:38:05.1850784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1851000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1851080Z return mod(**inputs) 2025-08-26T20:38:05.1851400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1851484Z outputs = self.mobilebert( 2025-08-26T20:38:05.1851789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1851870Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1852183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1852281Z layer_outputs = layer_module( 2025-08-26T20:38:05.1852614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1852792Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1853104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1853224Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1853528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1853631Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1853947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1854055Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1854060Z 2025-08-26T20:38:05.1854172Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1854387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1854465Z return mod(**inputs) 2025-08-26T20:38:05.1854783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1854869Z outputs = self.mobilebert( 2025-08-26T20:38:05.1855184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1855303Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1855612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1855715Z layer_outputs = layer_module( 2025-08-26T20:38:05.1856022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1856114Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1856424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1856500Z self_outputs = self.self( 2025-08-26T20:38:05.1856795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1856879Z self.query(query_tensor) 2025-08-26T20:38:05.1856885Z 2025-08-26T20:38:05.1856994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1857214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1857286Z return mod(**inputs) 2025-08-26T20:38:05.1857597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1857674Z outputs = self.mobilebert( 2025-08-26T20:38:05.1857971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1858055Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1858350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1858432Z layer_outputs = layer_module( 2025-08-26T20:38:05.1858731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1858821Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1859130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1859226Z self_outputs = self.self( 2025-08-26T20:38:05.1859549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1859624Z self.key(key_tensor) 2025-08-26T20:38:05.1859628Z 2025-08-26T20:38:05.1859745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1859954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1860023Z return mod(**inputs) 2025-08-26T20:38:05.1860340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1860417Z outputs = self.mobilebert( 2025-08-26T20:38:05.1860726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1860805Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1861112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1861198Z layer_outputs = layer_module( 2025-08-26T20:38:05.1861503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1861600Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1861906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1862003Z self_outputs = self.self( 2025-08-26T20:38:05.1862318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1862412Z self.value(value_tensor) 2025-08-26T20:38:05.1862416Z 2025-08-26T20:38:05.1862512Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1862600Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1862715Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1862929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1862999Z return mod(**inputs) 2025-08-26T20:38:05.1863305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1863381Z outputs = self.mobilebert( 2025-08-26T20:38:05.1863686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1863765Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1864060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1864145Z layer_outputs = layer_module( 2025-08-26T20:38:05.1864445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1864541Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1864843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1864975Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1865282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1865374Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1865378Z 2025-08-26T20:38:05.1865495Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1865707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1865787Z return mod(**inputs) 2025-08-26T20:38:05.1866108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1866202Z outputs = self.mobilebert( 2025-08-26T20:38:05.1866508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1866586Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1866888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1866962Z layer_outputs = layer_module( 2025-08-26T20:38:05.1867260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1867439Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1867738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1867863Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1868159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1868255Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1868259Z 2025-08-26T20:38:05.1868365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1868570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1868669Z return mod(**inputs) 2025-08-26T20:38:05.1868971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1869078Z outputs = self.mobilebert( 2025-08-26T20:38:05.1869377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1869453Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1869757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1869833Z layer_outputs = layer_module( 2025-08-26T20:38:05.1870139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1870227Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1870533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1870666Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1870962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1871104Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1871401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1871506Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1871510Z 2025-08-26T20:38:05.1871619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1871835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1871906Z return mod(**inputs) 2025-08-26T20:38:05.1872209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1872292Z outputs = self.mobilebert( 2025-08-26T20:38:05.1872599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1872704Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1873031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1873112Z layer_outputs = layer_module( 2025-08-26T20:38:05.1873429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1873536Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1873852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1873980Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1874296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1874390Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1874396Z 2025-08-26T20:38:05.1874507Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1874733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1874806Z return mod(**inputs) 2025-08-26T20:38:05.1875125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1875204Z outputs = self.mobilebert( 2025-08-26T20:38:05.1875510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1875640Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1875951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1876070Z layer_outputs = layer_module( 2025-08-26T20:38:05.1876380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1876487Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1876805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1876932Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1877247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1877373Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1877377Z 2025-08-26T20:38:05.1877499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1877719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1877790Z return mod(**inputs) 2025-08-26T20:38:05.1878118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1878200Z outputs = self.mobilebert( 2025-08-26T20:38:05.1878517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1878598Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1878909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1879001Z layer_outputs = layer_module( 2025-08-26T20:38:05.1879309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1879425Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1879826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1879980Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1880314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1880412Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1880417Z 2025-08-26T20:38:05.1880545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1880760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1880845Z return mod(**inputs) 2025-08-26T20:38:05.1881157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1881238Z outputs = self.mobilebert( 2025-08-26T20:38:05.1881557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1881636Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1881953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1882032Z layer_outputs = layer_module( 2025-08-26T20:38:05.1882352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1882456Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1882763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1882937Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1883262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1883404Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1883718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1883829Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1883833Z 2025-08-26T20:38:05.1883946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1884164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1884244Z return mod(**inputs) 2025-08-26T20:38:05.1884555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1884641Z outputs = self.mobilebert( 2025-08-26T20:38:05.1884948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1885029Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1885344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1885422Z layer_outputs = layer_module( 2025-08-26T20:38:05.1885737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1885841Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1886156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1886285Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1886591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1886694Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1886748Z 2025-08-26T20:38:05.1886862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1887101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1887176Z return mod(**inputs) 2025-08-26T20:38:05.1887486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1887573Z outputs = self.mobilebert( 2025-08-26T20:38:05.1887874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1887961Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1888266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1888345Z layer_outputs = layer_module( 2025-08-26T20:38:05.1888657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1888759Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1889069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1889191Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1889503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1889647Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1889651Z 2025-08-26T20:38:05.1889762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1889984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1890076Z return mod(**inputs) 2025-08-26T20:38:05.1890394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1890475Z outputs = self.mobilebert( 2025-08-26T20:38:05.1890786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1890865Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1891172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1891258Z layer_outputs = layer_module( 2025-08-26T20:38:05.1891567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1891676Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1891986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1892123Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1892437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1892529Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1892533Z 2025-08-26T20:38:05.1892660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1892868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1892948Z return mod(**inputs) 2025-08-26T20:38:05.1893245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1893322Z outputs = self.mobilebert( 2025-08-26T20:38:05.1893642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1893721Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1894043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1894121Z layer_outputs = layer_module( 2025-08-26T20:38:05.1894419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1894527Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1894824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1894963Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1895263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1895399Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1895694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1895793Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1895796Z 2025-08-26T20:38:05.1895912Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1896120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1896420Z return mod(**inputs) 2025-08-26T20:38:05.1896892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1896973Z outputs = self.mobilebert( 2025-08-26T20:38:05.1897322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1897403Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1897714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1897792Z layer_outputs = layer_module( 2025-08-26T20:38:05.1898098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1898199Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1898498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1898632Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1898934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1899033Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1899039Z 2025-08-26T20:38:05.1899148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1899361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1899442Z return mod(**inputs) 2025-08-26T20:38:05.1899743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1899825Z outputs = self.mobilebert( 2025-08-26T20:38:05.1900120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1900206Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1900503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1900582Z layer_outputs = layer_module( 2025-08-26T20:38:05.1900914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1901042Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1901345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1901464Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1901761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1901888Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1901892Z 2025-08-26T20:38:05.1902000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1902219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1902291Z return mod(**inputs) 2025-08-26T20:38:05.1902600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1902678Z outputs = self.mobilebert( 2025-08-26T20:38:05.1902973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1903059Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1903354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1903434Z layer_outputs = layer_module( 2025-08-26T20:38:05.1903749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1903847Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1904170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1904299Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1904604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1904694Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1904698Z 2025-08-26T20:38:05.1904812Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1905020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1905092Z return mod(**inputs) 2025-08-26T20:38:05.1905398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1905476Z outputs = self.mobilebert( 2025-08-26T20:38:05.1905775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1905853Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1906150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1906232Z layer_outputs = layer_module( 2025-08-26T20:38:05.1906538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1906638Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1906914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1907045Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1907322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1907461Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1907774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1907870Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1907874Z 2025-08-26T20:38:05.1907984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1908183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1908250Z return mod(**inputs) 2025-08-26T20:38:05.1908553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1908629Z outputs = self.mobilebert( 2025-08-26T20:38:05.1908938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1909017Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1909321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1909394Z layer_outputs = layer_module( 2025-08-26T20:38:05.1909673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1909803Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1910084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1910196Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1910200Z 2025-08-26T20:38:05.1910303Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1910515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1910590Z return mod(**inputs) 2025-08-26T20:38:05.1910876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1910956Z outputs = self.mobilebert( 2025-08-26T20:38:05.1911235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1911316Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1911595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1911668Z layer_outputs = layer_module( 2025-08-26T20:38:05.1911960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1912083Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1912370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1912485Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1912489Z 2025-08-26T20:38:05.1912591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1912797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1912864Z return mod(**inputs) 2025-08-26T20:38:05.1913154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1913229Z outputs = self.mobilebert( 2025-08-26T20:38:05.1913518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1913592Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1913884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1913971Z layer_outputs = layer_module( 2025-08-26T20:38:05.1914285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1914463Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1914759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1914857Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1914861Z 2025-08-26T20:38:05.1914971Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1915167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1915241Z return mod(**inputs) 2025-08-26T20:38:05.1915526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1915607Z outputs = self.mobilebert( 2025-08-26T20:38:05.1915901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1915977Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1916280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1916355Z layer_outputs = layer_module( 2025-08-26T20:38:05.1916677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1916843Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1917158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1917297Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1917598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1917703Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1917707Z 2025-08-26T20:38:05.1917820Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1918036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1918107Z return mod(**inputs) 2025-08-26T20:38:05.1918407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1918493Z outputs = self.mobilebert( 2025-08-26T20:38:05.1918792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1918876Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1919176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1919252Z layer_outputs = layer_module( 2025-08-26T20:38:05.1919789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1919964Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1920283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1920418Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1920756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1920856Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1920859Z 2025-08-26T20:38:05.1920978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1921189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1921257Z return mod(**inputs) 2025-08-26T20:38:05.1921550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1921622Z outputs = self.mobilebert( 2025-08-26T20:38:05.1921905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1921986Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1922270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1922357Z layer_outputs = layer_module( 2025-08-26T20:38:05.1922659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1922832Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1923129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1923253Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1923558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.1923680Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1923982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1924075Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1924078Z 2025-08-26T20:38:05.1924189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1924387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1924453Z return mod(**inputs) 2025-08-26T20:38:05.1924742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1924812Z outputs = self.mobilebert( 2025-08-26T20:38:05.1925098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1925171Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1925452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1925532Z layer_outputs = layer_module( 2025-08-26T20:38:05.1925820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1925992Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1926274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1926393Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1926672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1926757Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1926761Z 2025-08-26T20:38:05.1926874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1927089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1927166Z return mod(**inputs) 2025-08-26T20:38:05.1927469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1927543Z outputs = self.mobilebert( 2025-08-26T20:38:05.1927833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1927908Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1928200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1928274Z layer_outputs = layer_module( 2025-08-26T20:38:05.1928556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1928731Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1929017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.1929137Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.1929416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.1929511Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.1929795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1929908Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1929920Z 2025-08-26T20:38:05.1930024Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1930242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1930318Z return mod(**inputs) 2025-08-26T20:38:05.1930603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1930685Z outputs = self.mobilebert( 2025-08-26T20:38:05.1930967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1931040Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1931329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1931403Z layer_outputs = layer_module( 2025-08-26T20:38:05.1931689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1931777Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1932060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1932139Z self_outputs = self.self( 2025-08-26T20:38:05.1932430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.1932515Z self.query(query_tensor) 2025-08-26T20:38:05.1932518Z 2025-08-26T20:38:05.1932628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1932849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1932923Z return mod(**inputs) 2025-08-26T20:38:05.1933230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1933315Z outputs = self.mobilebert( 2025-08-26T20:38:05.1933639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1933726Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1934039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1934118Z layer_outputs = layer_module( 2025-08-26T20:38:05.1934424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1934516Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1934818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1934894Z self_outputs = self.self( 2025-08-26T20:38:05.1935188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.1935265Z self.key(key_tensor) 2025-08-26T20:38:05.1935269Z 2025-08-26T20:38:05.1935372Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1935583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1935653Z return mod(**inputs) 2025-08-26T20:38:05.1935957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1936032Z outputs = self.mobilebert( 2025-08-26T20:38:05.1936325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1936429Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1936726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1936826Z layer_outputs = layer_module( 2025-08-26T20:38:05.1937129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1937220Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1937527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.1937602Z self_outputs = self.self( 2025-08-26T20:38:05.1937908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.1937984Z self.value(value_tensor) 2025-08-26T20:38:05.1937990Z 2025-08-26T20:38:05.1938084Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1938167Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.1938279Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1938498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1938568Z return mod(**inputs) 2025-08-26T20:38:05.1938879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1938955Z outputs = self.mobilebert( 2025-08-26T20:38:05.1939254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1939339Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1939638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1939723Z layer_outputs = layer_module( 2025-08-26T20:38:05.1940019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1940110Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1940435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1940585Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1940890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.1940981Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1940985Z 2025-08-26T20:38:05.1941101Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1941308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1941380Z return mod(**inputs) 2025-08-26T20:38:05.1941685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1941762Z outputs = self.mobilebert( 2025-08-26T20:38:05.1942062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1942139Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1942434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1942515Z layer_outputs = layer_module( 2025-08-26T20:38:05.1942812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.1942993Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.1943315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.1943464Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.1943760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.1943849Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.1943853Z 2025-08-26T20:38:05.1943973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1944183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1944260Z return mod(**inputs) 2025-08-26T20:38:05.1944559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1944636Z outputs = self.mobilebert( 2025-08-26T20:38:05.1944945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1945023Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1945326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1945402Z layer_outputs = layer_module( 2025-08-26T20:38:05.1945707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.1945794Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.1946092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.1946231Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.1946526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.1946669Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1946970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1947084Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1947097Z 2025-08-26T20:38:05.1947221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1947434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1947513Z return mod(**inputs) 2025-08-26T20:38:05.1947812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1947897Z outputs = self.mobilebert( 2025-08-26T20:38:05.1948194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1948275Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1948587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1948669Z layer_outputs = layer_module( 2025-08-26T20:38:05.1948985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1949091Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1949392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1949525Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1949831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1949952Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1949956Z 2025-08-26T20:38:05.1950067Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1950309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1950383Z return mod(**inputs) 2025-08-26T20:38:05.1950694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1950782Z outputs = self.mobilebert( 2025-08-26T20:38:05.1951089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1951175Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1951478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1951559Z layer_outputs = layer_module( 2025-08-26T20:38:05.1951871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1951978Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1952288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1952415Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1952727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1952852Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1952856Z 2025-08-26T20:38:05.1952967Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1953190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1953264Z return mod(**inputs) 2025-08-26T20:38:05.1953578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1953657Z outputs = self.mobilebert( 2025-08-26T20:38:05.1953982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1954071Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1954394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1954483Z layer_outputs = layer_module( 2025-08-26T20:38:05.1954787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1954897Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1955206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1955345Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1955659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1955751Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1955755Z 2025-08-26T20:38:05.1955874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1956090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1956162Z return mod(**inputs) 2025-08-26T20:38:05.1956474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1956552Z outputs = self.mobilebert( 2025-08-26T20:38:05.1956890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1956968Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1957310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1957389Z layer_outputs = layer_module( 2025-08-26T20:38:05.1957696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1957807Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1958112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1958255Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1958561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1958695Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1959012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1959115Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1959119Z 2025-08-26T20:38:05.1959238Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1959533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1959627Z return mod(**inputs) 2025-08-26T20:38:05.1959936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1960016Z outputs = self.mobilebert( 2025-08-26T20:38:05.1960331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1960411Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1960723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1960827Z layer_outputs = layer_module( 2025-08-26T20:38:05.1961159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1961276Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1961580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1961715Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1962031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1962132Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1962136Z 2025-08-26T20:38:05.1962250Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1962468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1962549Z return mod(**inputs) 2025-08-26T20:38:05.1962860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1962948Z outputs = self.mobilebert( 2025-08-26T20:38:05.1963251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1963331Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1963642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1963739Z layer_outputs = layer_module( 2025-08-26T20:38:05.1964057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1964178Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1964489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1964614Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1964922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1965063Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1965067Z 2025-08-26T20:38:05.1965179Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1965403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1965477Z return mod(**inputs) 2025-08-26T20:38:05.1965787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1965873Z outputs = self.mobilebert( 2025-08-26T20:38:05.1966179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1966267Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1966576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1966660Z layer_outputs = layer_module( 2025-08-26T20:38:05.1966966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1967068Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1967384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1967519Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1967850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1967944Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1967948Z 2025-08-26T20:38:05.1968086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1968303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1968384Z return mod(**inputs) 2025-08-26T20:38:05.1968689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1968766Z outputs = self.mobilebert( 2025-08-26T20:38:05.1969074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1969152Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1969448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1969532Z layer_outputs = layer_module( 2025-08-26T20:38:05.1969827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1969934Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1970227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1970353Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1970673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1970825Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1971163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1971263Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1971266Z 2025-08-26T20:38:05.1971386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1971596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1971664Z return mod(**inputs) 2025-08-26T20:38:05.1971971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1972046Z outputs = self.mobilebert( 2025-08-26T20:38:05.1972358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1972435Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1972732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1972816Z layer_outputs = layer_module( 2025-08-26T20:38:05.1973116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1973225Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1973523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1973649Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1973956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1974046Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1974049Z 2025-08-26T20:38:05.1974165Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1974376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1974471Z return mod(**inputs) 2025-08-26T20:38:05.1974796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1974875Z outputs = self.mobilebert( 2025-08-26T20:38:05.1975182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1975259Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1975563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1975641Z layer_outputs = layer_module( 2025-08-26T20:38:05.1975945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1976046Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1976355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.1976481Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.1976789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1976917Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1976920Z 2025-08-26T20:38:05.1977028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1977257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1977347Z return mod(**inputs) 2025-08-26T20:38:05.1977656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1977759Z outputs = self.mobilebert( 2025-08-26T20:38:05.1978056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1978141Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1978443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1978520Z layer_outputs = layer_module( 2025-08-26T20:38:05.1978825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1978925Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1979238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1979370Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1979677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.1979777Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1979781Z 2025-08-26T20:38:05.1979890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1980113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1980184Z return mod(**inputs) 2025-08-26T20:38:05.1980500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1980589Z outputs = self.mobilebert( 2025-08-26T20:38:05.1980903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1980988Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1981290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1981388Z layer_outputs = layer_module( 2025-08-26T20:38:05.1981706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.1981809Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.1982109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.1982239Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.1982543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.1982672Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.1982978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1983081Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1983085Z 2025-08-26T20:38:05.1983198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1983421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1983494Z return mod(**inputs) 2025-08-26T20:38:05.1983809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1983887Z outputs = self.mobilebert( 2025-08-26T20:38:05.1984192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1984306Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1984616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1984718Z layer_outputs = layer_module( 2025-08-26T20:38:05.1985020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1985160Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1985461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.1985552Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.1985556Z 2025-08-26T20:38:05.1985676Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1985896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1985975Z return mod(**inputs) 2025-08-26T20:38:05.1986288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1986369Z outputs = self.mobilebert( 2025-08-26T20:38:05.1986683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1986764Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1987083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1987160Z layer_outputs = layer_module( 2025-08-26T20:38:05.1987481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.1987615Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.1987924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.1988060Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.1988064Z 2025-08-26T20:38:05.1988995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1989233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1989327Z return mod(**inputs) 2025-08-26T20:38:05.1989637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1989724Z outputs = self.mobilebert( 2025-08-26T20:38:05.1990029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1990121Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1990428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1990514Z layer_outputs = layer_module( 2025-08-26T20:38:05.1990895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1991151Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1991552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.1991661Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.1991666Z 2025-08-26T20:38:05.1991788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1992005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1992113Z return mod(**inputs) 2025-08-26T20:38:05.1992427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1992561Z outputs = self.mobilebert( 2025-08-26T20:38:05.1992879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1992958Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1993270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1993349Z layer_outputs = layer_module( 2025-08-26T20:38:05.1993655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1993838Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1994150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.1994292Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.1994601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.1994710Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.1994714Z 2025-08-26T20:38:05.1994828Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1995043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1995125Z return mod(**inputs) 2025-08-26T20:38:05.1995432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1995517Z outputs = self.mobilebert( 2025-08-26T20:38:05.1995824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1995904Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1996484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1996590Z layer_outputs = layer_module( 2025-08-26T20:38:05.1997104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.1997337Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.1997656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.1997793Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.1998101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.1998203Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.1998209Z 2025-08-26T20:38:05.1998320Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.1998550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.1998624Z return mod(**inputs) 2025-08-26T20:38:05.1998935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.1999023Z outputs = self.mobilebert( 2025-08-26T20:38:05.1999326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.1999413Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.1999817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.1999944Z layer_outputs = layer_module( 2025-08-26T20:38:05.2000251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2000454Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2000777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2000912Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2001225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2001359Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2001666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2001780Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2001785Z 2025-08-26T20:38:05.2001900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2002130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2002205Z return mod(**inputs) 2025-08-26T20:38:05.2002528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2002610Z outputs = self.mobilebert( 2025-08-26T20:38:05.2002920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2003008Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2003313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2003400Z layer_outputs = layer_module( 2025-08-26T20:38:05.2003705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2003905Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2004240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2004362Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2004664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2004752Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2004756Z 2025-08-26T20:38:05.2004872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2005085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2005155Z return mod(**inputs) 2025-08-26T20:38:05.2005464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2005544Z outputs = self.mobilebert( 2025-08-26T20:38:05.2005846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2005923Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2006217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2006301Z layer_outputs = layer_module( 2025-08-26T20:38:05.2006595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2006791Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2007090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2007235Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2007532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2007628Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2007932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2008028Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2008033Z 2025-08-26T20:38:05.2008147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2008357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2008434Z return mod(**inputs) 2025-08-26T20:38:05.2008741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2008818Z outputs = self.mobilebert( 2025-08-26T20:38:05.2009122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2009200Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2009502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2009576Z layer_outputs = layer_module( 2025-08-26T20:38:05.2009872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2009974Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2010270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2010354Z self_outputs = self.self( 2025-08-26T20:38:05.2010669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2010749Z self.query(query_tensor) 2025-08-26T20:38:05.2010761Z 2025-08-26T20:38:05.2010903Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2011115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2011193Z return mod(**inputs) 2025-08-26T20:38:05.2011491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2011576Z outputs = self.mobilebert( 2025-08-26T20:38:05.2011872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2011948Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2012257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2012334Z layer_outputs = layer_module( 2025-08-26T20:38:05.2012643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2012735Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2013032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2013113Z self_outputs = self.self( 2025-08-26T20:38:05.2013408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2013504Z self.key(key_tensor) 2025-08-26T20:38:05.2013509Z 2025-08-26T20:38:05.2013618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2013854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2013925Z return mod(**inputs) 2025-08-26T20:38:05.2014223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2014309Z outputs = self.mobilebert( 2025-08-26T20:38:05.2014617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2014700Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2015007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2015085Z layer_outputs = layer_module( 2025-08-26T20:38:05.2015398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2015490Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2015796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2015872Z self_outputs = self.self( 2025-08-26T20:38:05.2016177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2016261Z self.value(value_tensor) 2025-08-26T20:38:05.2016265Z 2025-08-26T20:38:05.2016352Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2016446Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2016556Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2016770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2016840Z return mod(**inputs) 2025-08-26T20:38:05.2017157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2017242Z outputs = self.mobilebert( 2025-08-26T20:38:05.2017563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2017666Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2017964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2018040Z layer_outputs = layer_module( 2025-08-26T20:38:05.2018345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2018439Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2018753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2018889Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2019205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2019295Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2019301Z 2025-08-26T20:38:05.2019409Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2019628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2019698Z return mod(**inputs) 2025-08-26T20:38:05.2020003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2020098Z outputs = self.mobilebert( 2025-08-26T20:38:05.2020394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2020504Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2020788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2020868Z layer_outputs = layer_module( 2025-08-26T20:38:05.2021163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2021342Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2021655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2021774Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2022080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2022168Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2022174Z 2025-08-26T20:38:05.2022288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2022497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2022567Z return mod(**inputs) 2025-08-26T20:38:05.2022882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2022955Z outputs = self.mobilebert( 2025-08-26T20:38:05.2023247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2023320Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2023613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2023685Z layer_outputs = layer_module( 2025-08-26T20:38:05.2023968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2024079Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2024375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2024507Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2024787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2024910Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2025201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2025300Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2025303Z 2025-08-26T20:38:05.2025423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2025634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2025710Z return mod(**inputs) 2025-08-26T20:38:05.2026014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2026086Z outputs = self.mobilebert( 2025-08-26T20:38:05.2026374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2026446Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2026731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2026819Z layer_outputs = layer_module( 2025-08-26T20:38:05.2027095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2027216Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2027497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2027622Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2027899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2027992Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2027995Z 2025-08-26T20:38:05.2028098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2028297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2028375Z return mod(**inputs) 2025-08-26T20:38:05.2028660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2028744Z outputs = self.mobilebert( 2025-08-26T20:38:05.2029025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2029100Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2029391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2029464Z layer_outputs = layer_module( 2025-08-26T20:38:05.2029760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2029865Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2030172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2030294Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2030614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2030744Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2030764Z 2025-08-26T20:38:05.2030876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2031091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2031164Z return mod(**inputs) 2025-08-26T20:38:05.2031470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2031561Z outputs = self.mobilebert( 2025-08-26T20:38:05.2031862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2031953Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2032254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2032340Z layer_outputs = layer_module( 2025-08-26T20:38:05.2032642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2032751Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2033068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2033210Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2033538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2033633Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2033654Z 2025-08-26T20:38:05.2033767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2033991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2034064Z return mod(**inputs) 2025-08-26T20:38:05.2034380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2034459Z outputs = self.mobilebert( 2025-08-26T20:38:05.2034769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2034847Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2035150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2035250Z layer_outputs = layer_module( 2025-08-26T20:38:05.2035551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2035667Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2035973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2036109Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2036421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2036554Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2036861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2036966Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2036969Z 2025-08-26T20:38:05.2037089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2037332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2037407Z return mod(**inputs) 2025-08-26T20:38:05.2037740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2037822Z outputs = self.mobilebert( 2025-08-26T20:38:05.2038140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2038221Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2038524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2038613Z layer_outputs = layer_module( 2025-08-26T20:38:05.2038918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2039032Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2039338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2039548Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2039867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2039961Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2039966Z 2025-08-26T20:38:05.2040086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2040304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2040410Z return mod(**inputs) 2025-08-26T20:38:05.2040722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2040819Z outputs = self.mobilebert( 2025-08-26T20:38:05.2041145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2041228Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2041553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2041631Z layer_outputs = layer_module( 2025-08-26T20:38:05.2041947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2042051Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2042364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2042497Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2042812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2042945Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2042950Z 2025-08-26T20:38:05.2043063Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2043282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2043361Z return mod(**inputs) 2025-08-26T20:38:05.2043677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2043765Z outputs = self.mobilebert( 2025-08-26T20:38:05.2044079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2044166Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2044497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2044576Z layer_outputs = layer_module( 2025-08-26T20:38:05.2044902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2045006Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2045319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2045455Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2045761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2045864Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2045869Z 2025-08-26T20:38:05.2045984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2046210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2046282Z return mod(**inputs) 2025-08-26T20:38:05.2046600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2046678Z outputs = self.mobilebert( 2025-08-26T20:38:05.2046986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2047073Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2047377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2047486Z layer_outputs = layer_module( 2025-08-26T20:38:05.2047791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2047916Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2048236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2048371Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2048689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2048817Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2049121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2049224Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2049227Z 2025-08-26T20:38:05.2049339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2049558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2049629Z return mod(**inputs) 2025-08-26T20:38:05.2049940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2050016Z outputs = self.mobilebert( 2025-08-26T20:38:05.2050315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2050399Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2050693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2050778Z layer_outputs = layer_module( 2025-08-26T20:38:05.2051074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2051182Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2051493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2051632Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2051940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2052029Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2052034Z 2025-08-26T20:38:05.2052152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2052363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2052435Z return mod(**inputs) 2025-08-26T20:38:05.2052741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2052820Z outputs = self.mobilebert( 2025-08-26T20:38:05.2053124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2053201Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2053504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2053581Z layer_outputs = layer_module( 2025-08-26T20:38:05.2053879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2053987Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2054299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2054446Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2054742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2054862Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2054874Z 2025-08-26T20:38:05.2054983Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2055192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2055269Z return mod(**inputs) 2025-08-26T20:38:05.2055580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2055666Z outputs = self.mobilebert( 2025-08-26T20:38:05.2055984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2056063Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2056365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2056441Z layer_outputs = layer_module( 2025-08-26T20:38:05.2056744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2056843Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2057162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2057301Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2057624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2057719Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2057725Z 2025-08-26T20:38:05.2057833Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2058066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2058139Z return mod(**inputs) 2025-08-26T20:38:05.2058468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2058556Z outputs = self.mobilebert( 2025-08-26T20:38:05.2058857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2058943Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2059243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2059322Z layer_outputs = layer_module( 2025-08-26T20:38:05.2059635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2059739Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2060057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2060188Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2060523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2060651Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2060970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2061094Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2061098Z 2025-08-26T20:38:05.2061236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2061460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2061529Z return mod(**inputs) 2025-08-26T20:38:05.2061846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2061931Z outputs = self.mobilebert( 2025-08-26T20:38:05.2062260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2062343Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2062638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2062721Z layer_outputs = layer_module( 2025-08-26T20:38:05.2063018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2063149Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2063477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2063568Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2063572Z 2025-08-26T20:38:05.2063686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2063898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2063968Z return mod(**inputs) 2025-08-26T20:38:05.2064344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2064427Z outputs = self.mobilebert( 2025-08-26T20:38:05.2064735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2064815Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2065143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2065221Z layer_outputs = layer_module( 2025-08-26T20:38:05.2065537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2065676Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2065976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2066105Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2066110Z 2025-08-26T20:38:05.2066221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2066430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2066510Z return mod(**inputs) 2025-08-26T20:38:05.2066812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2066896Z outputs = self.mobilebert( 2025-08-26T20:38:05.2067193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2067278Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2067577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2067652Z layer_outputs = layer_module( 2025-08-26T20:38:05.2067979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2068350Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2068710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2068839Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2068872Z 2025-08-26T20:38:05.2069021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2069259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2069378Z return mod(**inputs) 2025-08-26T20:38:05.2069732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2069900Z outputs = self.mobilebert( 2025-08-26T20:38:05.2070220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2070328Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2070681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2070782Z layer_outputs = layer_module( 2025-08-26T20:38:05.2071170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2071363Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2071716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2071871Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2072193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2072356Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2072362Z 2025-08-26T20:38:05.2072504Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2072834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2072929Z return mod(**inputs) 2025-08-26T20:38:05.2073313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2073401Z outputs = self.mobilebert( 2025-08-26T20:38:05.2073741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2073888Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2074211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2074339Z layer_outputs = layer_module( 2025-08-26T20:38:05.2074661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2074867Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2075265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2075424Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2075777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2075893Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2075897Z 2025-08-26T20:38:05.2076064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2076325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2076450Z return mod(**inputs) 2025-08-26T20:38:05.2076801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2076902Z outputs = self.mobilebert( 2025-08-26T20:38:05.2077252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2077347Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2077731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2077831Z layer_outputs = layer_module( 2025-08-26T20:38:05.2078161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2078393Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2078730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2078952Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2079292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2079557Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2079896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2080056Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2080060Z 2025-08-26T20:38:05.2080224Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2080495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2080633Z return mod(**inputs) 2025-08-26T20:38:05.2080976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2081099Z outputs = self.mobilebert( 2025-08-26T20:38:05.2081471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2081565Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2081964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2082068Z layer_outputs = layer_module( 2025-08-26T20:38:05.2082438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2082644Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2082971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2083158Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2083498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2083652Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2083657Z 2025-08-26T20:38:05.2083798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2084066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2084150Z return mod(**inputs) 2025-08-26T20:38:05.2084501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2084668Z outputs = self.mobilebert( 2025-08-26T20:38:05.2085024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2085174Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2085508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2085651Z layer_outputs = layer_module( 2025-08-26T20:38:05.2085999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2086202Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2086565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2086715Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2087062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2087206Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2087590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2087717Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2087726Z 2025-08-26T20:38:05.2087864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2088142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2088225Z return mod(**inputs) 2025-08-26T20:38:05.2088618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2088725Z outputs = self.mobilebert( 2025-08-26T20:38:05.2089057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2089193Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2089542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2089696Z layer_outputs = layer_module( 2025-08-26T20:38:05.2090080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2090232Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2090564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2090668Z self_outputs = self.self( 2025-08-26T20:38:05.2091032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2091164Z self.query(query_tensor) 2025-08-26T20:38:05.2091171Z 2025-08-26T20:38:05.2091350Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2091591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2091689Z return mod(**inputs) 2025-08-26T20:38:05.2092062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2092154Z outputs = self.mobilebert( 2025-08-26T20:38:05.2092556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2092660Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2093023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2093150Z layer_outputs = layer_module( 2025-08-26T20:38:05.2093473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2093653Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2093989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2094116Z self_outputs = self.self( 2025-08-26T20:38:05.2094445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2094568Z self.key(key_tensor) 2025-08-26T20:38:05.2094572Z 2025-08-26T20:38:05.2094691Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2094973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2095110Z return mod(**inputs) 2025-08-26T20:38:05.2095442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2095574Z outputs = self.mobilebert( 2025-08-26T20:38:05.2095897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2095987Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2096555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2096661Z layer_outputs = layer_module( 2025-08-26T20:38:05.2097024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2097141Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2097494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2097642Z self_outputs = self.self( 2025-08-26T20:38:05.2098049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2098188Z self.value(value_tensor) 2025-08-26T20:38:05.2098193Z 2025-08-26T20:38:05.2098332Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2098473Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2098596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2098853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2098997Z return mod(**inputs) 2025-08-26T20:38:05.2099325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2099453Z outputs = self.mobilebert( 2025-08-26T20:38:05.2099806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2099949Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2100301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2100399Z layer_outputs = layer_module( 2025-08-26T20:38:05.2100757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2100870Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2101266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2101450Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2101863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2102001Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2102006Z 2025-08-26T20:38:05.2102141Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2102417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2102498Z return mod(**inputs) 2025-08-26T20:38:05.2102877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2102978Z outputs = self.mobilebert( 2025-08-26T20:38:05.2103304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2103435Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2103763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2103899Z layer_outputs = layer_module( 2025-08-26T20:38:05.2104235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2104463Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2104790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2104968Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2105305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2105438Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2105444Z 2025-08-26T20:38:05.2105616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2105847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2105980Z return mod(**inputs) 2025-08-26T20:38:05.2106340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2106427Z outputs = self.mobilebert( 2025-08-26T20:38:05.2106844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2106945Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2107313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2107412Z layer_outputs = layer_module( 2025-08-26T20:38:05.2107732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2107884Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2108237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2108432Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2108757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2108938Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2109235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2109367Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2109410Z 2025-08-26T20:38:05.2109541Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2109805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2109925Z return mod(**inputs) 2025-08-26T20:38:05.2110247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2110383Z outputs = self.mobilebert( 2025-08-26T20:38:05.2110703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2110797Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2111134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2111232Z layer_outputs = layer_module( 2025-08-26T20:38:05.2111547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2111697Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2112041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2112182Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2112488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2112628Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2112632Z 2025-08-26T20:38:05.2112767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2113077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2113170Z return mod(**inputs) 2025-08-26T20:38:05.2113521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2113627Z outputs = self.mobilebert( 2025-08-26T20:38:05.2113947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2114101Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2114478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2114608Z layer_outputs = layer_module( 2025-08-26T20:38:05.2114944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2115076Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2115427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2115594Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2115955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2116105Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2116109Z 2025-08-26T20:38:05.2116276Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2116513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2116598Z return mod(**inputs) 2025-08-26T20:38:05.2116992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2117092Z outputs = self.mobilebert( 2025-08-26T20:38:05.2117448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2117557Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2117916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2118064Z layer_outputs = layer_module( 2025-08-26T20:38:05.2118426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2118594Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2118930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2119126Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2119507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2119747Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2119752Z 2025-08-26T20:38:05.2119894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2120146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2120277Z return mod(**inputs) 2025-08-26T20:38:05.2120625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2120767Z outputs = self.mobilebert( 2025-08-26T20:38:05.2121098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2121216Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2121548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2121643Z layer_outputs = layer_module( 2025-08-26T20:38:05.2121968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2122103Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2122457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2122636Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2122962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2123140Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2123431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2136217Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2136240Z 2025-08-26T20:38:05.2136430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2136676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2136762Z return mod(**inputs) 2025-08-26T20:38:05.2137094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2137187Z outputs = self.mobilebert( 2025-08-26T20:38:05.2137487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2137568Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2137864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2137941Z layer_outputs = layer_module( 2025-08-26T20:38:05.2138242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2138422Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2138709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2138860Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2139139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2139239Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2139244Z 2025-08-26T20:38:05.2139354Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2139571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2139642Z return mod(**inputs) 2025-08-26T20:38:05.2139923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2140013Z outputs = self.mobilebert( 2025-08-26T20:38:05.2140296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2140383Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2140667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2140748Z layer_outputs = layer_module( 2025-08-26T20:38:05.2141032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2141131Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2141423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2141541Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2141835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2141957Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2141963Z 2025-08-26T20:38:05.2142084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2142335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2142433Z return mod(**inputs) 2025-08-26T20:38:05.2142743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2142820Z outputs = self.mobilebert( 2025-08-26T20:38:05.2143107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2143186Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2143482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2143569Z layer_outputs = layer_module( 2025-08-26T20:38:05.2143864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2143975Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2144272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2144414Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2144716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2144808Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2144812Z 2025-08-26T20:38:05.2144951Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2145170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2145263Z return mod(**inputs) 2025-08-26T20:38:05.2145546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2145620Z outputs = self.mobilebert( 2025-08-26T20:38:05.2145905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2145977Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2146258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2146329Z layer_outputs = layer_module( 2025-08-26T20:38:05.2146600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2146702Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2146980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2147112Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2147400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2147525Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2147812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2147908Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2147912Z 2025-08-26T20:38:05.2148018Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2148227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2148294Z return mod(**inputs) 2025-08-26T20:38:05.2148586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2148680Z outputs = self.mobilebert( 2025-08-26T20:38:05.2148995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2149084Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2149397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2149479Z layer_outputs = layer_module( 2025-08-26T20:38:05.2149762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2149870Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2150160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2150287Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2150598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2150695Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2150700Z 2025-08-26T20:38:05.2150824Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2151039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2151114Z return mod(**inputs) 2025-08-26T20:38:05.2151430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2151530Z outputs = self.mobilebert( 2025-08-26T20:38:05.2151831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2151928Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2152227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2152298Z layer_outputs = layer_module( 2025-08-26T20:38:05.2152583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2152690Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2153011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2153138Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2153438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2153557Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2153570Z 2025-08-26T20:38:05.2153679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2153895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2153972Z return mod(**inputs) 2025-08-26T20:38:05.2154272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2154358Z outputs = self.mobilebert( 2025-08-26T20:38:05.2154648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2154724Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2155030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2155104Z layer_outputs = layer_module( 2025-08-26T20:38:05.2155408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2155556Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2155874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2156016Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2156312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2156411Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2156415Z 2025-08-26T20:38:05.2156527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2156745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2156816Z return mod(**inputs) 2025-08-26T20:38:05.2157120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2157209Z outputs = self.mobilebert( 2025-08-26T20:38:05.2157508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2157594Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2157890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2157966Z layer_outputs = layer_module( 2025-08-26T20:38:05.2158275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2158396Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2158710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2158879Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2159186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2159322Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2159747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2159861Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2159866Z 2025-08-26T20:38:05.2159978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2160207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2160281Z return mod(**inputs) 2025-08-26T20:38:05.2160589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2160681Z outputs = self.mobilebert( 2025-08-26T20:38:05.2160999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2161087Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2161382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2161465Z layer_outputs = layer_module( 2025-08-26T20:38:05.2161757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2161889Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2162192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2162283Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2162287Z 2025-08-26T20:38:05.2162439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2162671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2162745Z return mod(**inputs) 2025-08-26T20:38:05.2163051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2163129Z outputs = self.mobilebert( 2025-08-26T20:38:05.2163429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2163509Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2163810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2163886Z layer_outputs = layer_module( 2025-08-26T20:38:05.2164183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2164318Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2164613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2164739Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2164743Z 2025-08-26T20:38:05.2164853Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2165067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2165165Z return mod(**inputs) 2025-08-26T20:38:05.2165464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2165567Z outputs = self.mobilebert( 2025-08-26T20:38:05.2165869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2165950Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2166230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2166301Z layer_outputs = layer_module( 2025-08-26T20:38:05.2166586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2166749Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2167042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2167148Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2167153Z 2025-08-26T20:38:05.2167262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2167457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2167522Z return mod(**inputs) 2025-08-26T20:38:05.2167813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2167887Z outputs = self.mobilebert( 2025-08-26T20:38:05.2168175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2168247Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2168530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2168611Z layer_outputs = layer_module( 2025-08-26T20:38:05.2168890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2169078Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2169375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2169503Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2169790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2169885Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2169890Z 2025-08-26T20:38:05.2170003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2170201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2170277Z return mod(**inputs) 2025-08-26T20:38:05.2170752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2170839Z outputs = self.mobilebert( 2025-08-26T20:38:05.2171134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2171210Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2171503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2171576Z layer_outputs = layer_module( 2025-08-26T20:38:05.2171860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2172052Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2172351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2172489Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2172773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2172879Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2172883Z 2025-08-26T20:38:05.2172985Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2173181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2173258Z return mod(**inputs) 2025-08-26T20:38:05.2173538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2173620Z outputs = self.mobilebert( 2025-08-26T20:38:05.2173901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2173974Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2174261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2174333Z layer_outputs = layer_module( 2025-08-26T20:38:05.2174619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2174783Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2175061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2175184Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2175459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2175606Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2175896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2175998Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2176001Z 2025-08-26T20:38:05.2176102Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2176305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2176372Z return mod(**inputs) 2025-08-26T20:38:05.2176660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2176742Z outputs = self.mobilebert( 2025-08-26T20:38:05.2177024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2177107Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2177392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2177464Z layer_outputs = layer_module( 2025-08-26T20:38:05.2177756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2177920Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2178209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2178338Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2178621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2178723Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2178729Z 2025-08-26T20:38:05.2178831Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2179046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2179113Z return mod(**inputs) 2025-08-26T20:38:05.2179407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2179480Z outputs = self.mobilebert( 2025-08-26T20:38:05.2179775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2179859Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2180160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2180250Z layer_outputs = layer_module( 2025-08-26T20:38:05.2180534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2180702Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2180987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2181097Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2181387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2181482Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2181786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2181885Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2181889Z 2025-08-26T20:38:05.2182022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2182253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2182327Z return mod(**inputs) 2025-08-26T20:38:05.2182637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2182714Z outputs = self.mobilebert( 2025-08-26T20:38:05.2183017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2183096Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2183396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2183481Z layer_outputs = layer_module( 2025-08-26T20:38:05.2183778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2183878Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2184178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2184256Z self_outputs = self.self( 2025-08-26T20:38:05.2184559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2184635Z self.query(query_tensor) 2025-08-26T20:38:05.2184639Z 2025-08-26T20:38:05.2184777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2184987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2185065Z return mod(**inputs) 2025-08-26T20:38:05.2185403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2185480Z outputs = self.mobilebert( 2025-08-26T20:38:05.2185787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2185865Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2186172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2186248Z layer_outputs = layer_module( 2025-08-26T20:38:05.2186549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2186648Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2186944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2187029Z self_outputs = self.self( 2025-08-26T20:38:05.2187323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2187405Z self.key(key_tensor) 2025-08-26T20:38:05.2187409Z 2025-08-26T20:38:05.2187518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2187726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2187804Z return mod(**inputs) 2025-08-26T20:38:05.2188100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2188186Z outputs = self.mobilebert( 2025-08-26T20:38:05.2188480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2188559Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2188879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2188956Z layer_outputs = layer_module( 2025-08-26T20:38:05.2189280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2189372Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2189666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2189748Z self_outputs = self.self( 2025-08-26T20:38:05.2190042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2190124Z self.value(value_tensor) 2025-08-26T20:38:05.2190130Z 2025-08-26T20:38:05.2190221Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2190313Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2190425Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2190639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2190935Z return mod(**inputs) 2025-08-26T20:38:05.2191249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2191334Z outputs = self.mobilebert( 2025-08-26T20:38:05.2191633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2191736Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2192042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2192137Z layer_outputs = layer_module( 2025-08-26T20:38:05.2192442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2192535Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2192834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2192976Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2193276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2193379Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2193385Z 2025-08-26T20:38:05.2193499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2193721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2193797Z return mod(**inputs) 2025-08-26T20:38:05.2194109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2194196Z outputs = self.mobilebert( 2025-08-26T20:38:05.2194502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2194589Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2194893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2194972Z layer_outputs = layer_module( 2025-08-26T20:38:05.2195292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2195469Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2195788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2195929Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2196467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2196567Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2196572Z 2025-08-26T20:38:05.2196685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2196910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2196984Z return mod(**inputs) 2025-08-26T20:38:05.2197307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2197387Z outputs = self.mobilebert( 2025-08-26T20:38:05.2197695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2197782Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2198089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2198175Z layer_outputs = layer_module( 2025-08-26T20:38:05.2198480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2198580Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2198896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2199064Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2199391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2199613Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2199940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2200042Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2200046Z 2025-08-26T20:38:05.2200163Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2200379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2200453Z return mod(**inputs) 2025-08-26T20:38:05.2200770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2200853Z outputs = self.mobilebert( 2025-08-26T20:38:05.2201164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2201247Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2201551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2201641Z layer_outputs = layer_module( 2025-08-26T20:38:05.2201947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2202063Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2202369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2202498Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2202811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2202909Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2202913Z 2025-08-26T20:38:05.2203065Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2203302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2203386Z return mod(**inputs) 2025-08-26T20:38:05.2203698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2203778Z outputs = self.mobilebert( 2025-08-26T20:38:05.2204095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2204176Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2204506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2204589Z layer_outputs = layer_module( 2025-08-26T20:38:05.2204896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2205010Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2205319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2205449Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2205722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2205841Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2205863Z 2025-08-26T20:38:05.2205965Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2206157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2206250Z return mod(**inputs) 2025-08-26T20:38:05.2206529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2206608Z outputs = self.mobilebert( 2025-08-26T20:38:05.2206884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2206958Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2207242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2207313Z layer_outputs = layer_module( 2025-08-26T20:38:05.2207594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2207689Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2207973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2208098Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2208375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2208469Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2208472Z 2025-08-26T20:38:05.2208573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2208775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2208840Z return mod(**inputs) 2025-08-26T20:38:05.2209119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2209197Z outputs = self.mobilebert( 2025-08-26T20:38:05.2209474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2209566Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2209853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2209931Z layer_outputs = layer_module( 2025-08-26T20:38:05.2210203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2210295Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2210576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2210698Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2210975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2211098Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2211371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2211468Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2211472Z 2025-08-26T20:38:05.2211570Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2211766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2211831Z return mod(**inputs) 2025-08-26T20:38:05.2212115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2212211Z outputs = self.mobilebert( 2025-08-26T20:38:05.2212486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2212582Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2212865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2212947Z layer_outputs = layer_module( 2025-08-26T20:38:05.2213236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2213328Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2213617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2213738Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2214051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2214143Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2214147Z 2025-08-26T20:38:05.2214264Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2214477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2214550Z return mod(**inputs) 2025-08-26T20:38:05.2214864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2214943Z outputs = self.mobilebert( 2025-08-26T20:38:05.2215257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2215337Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2215634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2215718Z layer_outputs = layer_module( 2025-08-26T20:38:05.2216026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2216134Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2216429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2216551Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2216833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2216947Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2216952Z 2025-08-26T20:38:05.2217068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2217266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2217341Z return mod(**inputs) 2025-08-26T20:38:05.2217625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2217699Z outputs = self.mobilebert( 2025-08-26T20:38:05.2217985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2218060Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2218344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2218416Z layer_outputs = layer_module( 2025-08-26T20:38:05.2218703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2218816Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2219114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2219248Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2219527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2219618Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2219621Z 2025-08-26T20:38:05.2219724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2219920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2219994Z return mod(**inputs) 2025-08-26T20:38:05.2220277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2220356Z outputs = self.mobilebert( 2025-08-26T20:38:05.2220639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2220722Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2221002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2221074Z layer_outputs = layer_module( 2025-08-26T20:38:05.2221357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2221450Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2221739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2221867Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2222146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2222296Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2222596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2222705Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2222710Z 2025-08-26T20:38:05.2222817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2223032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2223103Z return mod(**inputs) 2025-08-26T20:38:05.2223399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2223481Z outputs = self.mobilebert( 2025-08-26T20:38:05.2223761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2223845Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2224125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2224198Z layer_outputs = layer_module( 2025-08-26T20:38:05.2224486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2224580Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2224865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2224995Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2225280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2225382Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2225386Z 2025-08-26T20:38:05.2225490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2225697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2225766Z return mod(**inputs) 2025-08-26T20:38:05.2226059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2226131Z outputs = self.mobilebert( 2025-08-26T20:38:05.2226410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2226491Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2226768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2226848Z layer_outputs = layer_module( 2025-08-26T20:38:05.2227130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2227230Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2227510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2227623Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2227910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2228021Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2228026Z 2025-08-26T20:38:05.2228137Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2228334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2228401Z return mod(**inputs) 2025-08-26T20:38:05.2228705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2228781Z outputs = self.mobilebert( 2025-08-26T20:38:05.2229085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2229159Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2229452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2229523Z layer_outputs = layer_module( 2025-08-26T20:38:05.2229805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2229908Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2230189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2230322Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2230602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2230686Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2230696Z 2025-08-26T20:38:05.2230804Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2231013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2231091Z return mod(**inputs) 2025-08-26T20:38:05.2231418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2231499Z outputs = self.mobilebert( 2025-08-26T20:38:05.2231799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2231874Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2232161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2232236Z layer_outputs = layer_module( 2025-08-26T20:38:05.2232523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2232621Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2232908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2233041Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2233329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2233468Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2233767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2233876Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2233880Z 2025-08-26T20:38:05.2233989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2234200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2234280Z return mod(**inputs) 2025-08-26T20:38:05.2234580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2234665Z outputs = self.mobilebert( 2025-08-26T20:38:05.2234961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2235039Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2235360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2235467Z layer_outputs = layer_module( 2025-08-26T20:38:05.2235774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2235907Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2236223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2236314Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2236318Z 2025-08-26T20:38:05.2236426Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2236644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2236716Z return mod(**inputs) 2025-08-26T20:38:05.2237025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2237105Z outputs = self.mobilebert( 2025-08-26T20:38:05.2237409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2237496Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2237808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2237892Z layer_outputs = layer_module( 2025-08-26T20:38:05.2238218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2238356Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2238682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2238806Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2238810Z 2025-08-26T20:38:05.2238930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2239148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2239228Z return mod(**inputs) 2025-08-26T20:38:05.2239621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2239703Z outputs = self.mobilebert( 2025-08-26T20:38:05.2240020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2240099Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2240417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2240496Z layer_outputs = layer_module( 2025-08-26T20:38:05.2240808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2240984Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2241288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2241401Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2241407Z 2025-08-26T20:38:05.2241518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2241741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2241817Z return mod(**inputs) 2025-08-26T20:38:05.2242145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2242235Z outputs = self.mobilebert( 2025-08-26T20:38:05.2242556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2242649Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2242953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2243038Z layer_outputs = layer_module( 2025-08-26T20:38:05.2243341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2243515Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2243829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2243964Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2244276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2244379Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2244383Z 2025-08-26T20:38:05.2244499Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2244712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2244784Z return mod(**inputs) 2025-08-26T20:38:05.2245126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2245203Z outputs = self.mobilebert( 2025-08-26T20:38:05.2245537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2245617Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2245920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2246008Z layer_outputs = layer_module( 2025-08-26T20:38:05.2246308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2246485Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2246789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2246932Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2247238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2247332Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2247336Z 2025-08-26T20:38:05.2247463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2247664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2247739Z return mod(**inputs) 2025-08-26T20:38:05.2248026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2248098Z outputs = self.mobilebert( 2025-08-26T20:38:05.2248385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2248462Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2248747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2248821Z layer_outputs = layer_module( 2025-08-26T20:38:05.2249117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2249299Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2249582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2249714Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2249993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2250125Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2250405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2250501Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2250512Z 2025-08-26T20:38:05.2250615Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2250815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2250890Z return mod(**inputs) 2025-08-26T20:38:05.2251176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2251255Z outputs = self.mobilebert( 2025-08-26T20:38:05.2251534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2251623Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2251915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2252004Z layer_outputs = layer_module( 2025-08-26T20:38:05.2252293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2252456Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2252738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2252857Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2253135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2253229Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2253233Z 2025-08-26T20:38:05.2253334Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2253541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2253609Z return mod(**inputs) 2025-08-26T20:38:05.2253896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2253978Z outputs = self.mobilebert( 2025-08-26T20:38:05.2254258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2254340Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2254622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2254695Z layer_outputs = layer_module( 2025-08-26T20:38:05.2254981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2255144Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2255447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2255574Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2255865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2255953Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2256233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2256337Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2256341Z 2025-08-26T20:38:05.2256442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2256646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2256716Z return mod(**inputs) 2025-08-26T20:38:05.2257001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2257083Z outputs = self.mobilebert( 2025-08-26T20:38:05.2257361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2257442Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2257720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2257800Z layer_outputs = layer_module( 2025-08-26T20:38:05.2258100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2258188Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2258529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2258601Z self_outputs = self.self( 2025-08-26T20:38:05.2258909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2258987Z self.query(query_tensor) 2025-08-26T20:38:05.2258991Z 2025-08-26T20:38:05.2259098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2259329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2259395Z return mod(**inputs) 2025-08-26T20:38:05.2259690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2259764Z outputs = self.mobilebert( 2025-08-26T20:38:05.2260054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2260129Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2260410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2260493Z layer_outputs = layer_module( 2025-08-26T20:38:05.2260790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2260889Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2261184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2261261Z self_outputs = self.self( 2025-08-26T20:38:05.2261566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2261638Z self.key(key_tensor) 2025-08-26T20:38:05.2261642Z 2025-08-26T20:38:05.2261788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2261997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2262093Z return mod(**inputs) 2025-08-26T20:38:05.2262395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2262469Z outputs = self.mobilebert( 2025-08-26T20:38:05.2262782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2262860Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2263163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2263241Z layer_outputs = layer_module( 2025-08-26T20:38:05.2263536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2263635Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2263939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2264015Z self_outputs = self.self( 2025-08-26T20:38:05.2264301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2264373Z self.value(value_tensor) 2025-08-26T20:38:05.2264384Z 2025-08-26T20:38:05.2264468Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2264567Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2264679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2264878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2264969Z return mod(**inputs) 2025-08-26T20:38:05.2265255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2265329Z outputs = self.mobilebert( 2025-08-26T20:38:05.2265617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2265689Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2265976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2266048Z layer_outputs = layer_module( 2025-08-26T20:38:05.2266331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2266421Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2266704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2266835Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2267121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2267207Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2267218Z 2025-08-26T20:38:05.2267321Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2267520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2267597Z return mod(**inputs) 2025-08-26T20:38:05.2267882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2267963Z outputs = self.mobilebert( 2025-08-26T20:38:05.2268263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2268338Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2268645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2268718Z layer_outputs = layer_module( 2025-08-26T20:38:05.2269008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2269169Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2269463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2269578Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2269862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2269956Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2269960Z 2025-08-26T20:38:05.2270064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2270278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2270345Z return mod(**inputs) 2025-08-26T20:38:05.2270628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2270710Z outputs = self.mobilebert( 2025-08-26T20:38:05.2271016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2271120Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2271427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2272295Z layer_outputs = layer_module( 2025-08-26T20:38:05.2272619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2272712Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2273034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2273168Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2273494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2273631Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2273940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2274047Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2274053Z 2025-08-26T20:38:05.2274160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2274383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2274455Z return mod(**inputs) 2025-08-26T20:38:05.2274774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2274853Z outputs = self.mobilebert( 2025-08-26T20:38:05.2275173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2275263Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2275574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2275662Z layer_outputs = layer_module( 2025-08-26T20:38:05.2275997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2276124Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2276436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2276559Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2276879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2276973Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2276977Z 2025-08-26T20:38:05.2277094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2277307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2277380Z return mod(**inputs) 2025-08-26T20:38:05.2277693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2277773Z outputs = self.mobilebert( 2025-08-26T20:38:05.2278082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2278162Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2278472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2278557Z layer_outputs = layer_module( 2025-08-26T20:38:05.2278877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2278986Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2279312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2279552Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2279883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2280009Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2280013Z 2025-08-26T20:38:05.2280133Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2280348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2280431Z return mod(**inputs) 2025-08-26T20:38:05.2280743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2280820Z outputs = self.mobilebert( 2025-08-26T20:38:05.2281135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2281207Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2281496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2281568Z layer_outputs = layer_module( 2025-08-26T20:38:05.2281844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2281948Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2282227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2282364Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2282644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2282757Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2282761Z 2025-08-26T20:38:05.2282866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2283085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2283163Z return mod(**inputs) 2025-08-26T20:38:05.2283447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2283528Z outputs = self.mobilebert( 2025-08-26T20:38:05.2283807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2283888Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2284165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2284239Z layer_outputs = layer_module( 2025-08-26T20:38:05.2284543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2284641Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2284928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2285055Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2285333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2285488Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2285767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2285897Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2285902Z 2025-08-26T20:38:05.2286004Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2286213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2286280Z return mod(**inputs) 2025-08-26T20:38:05.2286566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2286645Z outputs = self.mobilebert( 2025-08-26T20:38:05.2286927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2287010Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2287292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2287366Z layer_outputs = layer_module( 2025-08-26T20:38:05.2287657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2287752Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2288039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2288153Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2288448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2288540Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2288544Z 2025-08-26T20:38:05.2288650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2288869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2288941Z return mod(**inputs) 2025-08-26T20:38:05.2289267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2289361Z outputs = self.mobilebert( 2025-08-26T20:38:05.2289664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2289758Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2290036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2290118Z layer_outputs = layer_module( 2025-08-26T20:38:05.2290407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2290514Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2290814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2290932Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2291237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2291356Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2291360Z 2025-08-26T20:38:05.2291475Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2291686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2291775Z return mod(**inputs) 2025-08-26T20:38:05.2292085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2292161Z outputs = self.mobilebert( 2025-08-26T20:38:05.2292481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2292560Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2292863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2292939Z layer_outputs = layer_module( 2025-08-26T20:38:05.2293232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2293339Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2293633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2293775Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2294077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2294164Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2294167Z 2025-08-26T20:38:05.2294276Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2294476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2294548Z return mod(**inputs) 2025-08-26T20:38:05.2294827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2294904Z outputs = self.mobilebert( 2025-08-26T20:38:05.2295183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2295258Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2295545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2295619Z layer_outputs = layer_module( 2025-08-26T20:38:05.2295920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2296031Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2296467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2296610Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2296904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2297043Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2297340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2297447Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2297452Z 2025-08-26T20:38:05.2297561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2297773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2297862Z return mod(**inputs) 2025-08-26T20:38:05.2298147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2298228Z outputs = self.mobilebert( 2025-08-26T20:38:05.2298511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2298638Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2298930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2299030Z layer_outputs = layer_module( 2025-08-26T20:38:05.2299317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2299412Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2299696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2299809Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2300087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2300184Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2300188Z 2025-08-26T20:38:05.2300291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2300499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2300568Z return mod(**inputs) 2025-08-26T20:38:05.2300850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2300933Z outputs = self.mobilebert( 2025-08-26T20:38:05.2301210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2301293Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2301570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2301649Z layer_outputs = layer_module( 2025-08-26T20:38:05.2301930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2302023Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2302338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2302453Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2302779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2302901Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2302904Z 2025-08-26T20:38:05.2303012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2303228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2303299Z return mod(**inputs) 2025-08-26T20:38:05.2303603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2303677Z outputs = self.mobilebert( 2025-08-26T20:38:05.2303963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2304036Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2304315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2304394Z layer_outputs = layer_module( 2025-08-26T20:38:05.2304670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2304770Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2305043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2305185Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2305471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2305574Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2305577Z 2025-08-26T20:38:05.2305685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2305897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2305974Z return mod(**inputs) 2025-08-26T20:38:05.2306274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2306353Z outputs = self.mobilebert( 2025-08-26T20:38:05.2306659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2306737Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2307041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2307116Z layer_outputs = layer_module( 2025-08-26T20:38:05.2307413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2307523Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2307817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2307955Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2308254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2308392Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2308688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2308787Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2308792Z 2025-08-26T20:38:05.2308926Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2309154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2309234Z return mod(**inputs) 2025-08-26T20:38:05.2309535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2309611Z outputs = self.mobilebert( 2025-08-26T20:38:05.2309920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2309999Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2310304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2310379Z layer_outputs = layer_module( 2025-08-26T20:38:05.2310689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2310823Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2311123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2311220Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2311224Z 2025-08-26T20:38:05.2311331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2311546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2311642Z return mod(**inputs) 2025-08-26T20:38:05.2311940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2312043Z outputs = self.mobilebert( 2025-08-26T20:38:05.2312341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2312424Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2312721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2312803Z layer_outputs = layer_module( 2025-08-26T20:38:05.2313097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2313225Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2313530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2313647Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2313653Z 2025-08-26T20:38:05.2313768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2313977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2314048Z return mod(**inputs) 2025-08-26T20:38:05.2314358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2314435Z outputs = self.mobilebert( 2025-08-26T20:38:05.2314742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2314819Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2315122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2315197Z layer_outputs = layer_module( 2025-08-26T20:38:05.2315492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2315693Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2316009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2316127Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2316131Z 2025-08-26T20:38:05.2316234Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2316432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2316507Z return mod(**inputs) 2025-08-26T20:38:05.2316794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2316875Z outputs = self.mobilebert( 2025-08-26T20:38:05.2317171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2317256Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2317553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2317629Z layer_outputs = layer_module( 2025-08-26T20:38:05.2317933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2318104Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2318409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2318561Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2318889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2318999Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2319003Z 2025-08-26T20:38:05.2319114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2319342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2319415Z return mod(**inputs) 2025-08-26T20:38:05.2320091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2320175Z outputs = self.mobilebert( 2025-08-26T20:38:05.2320482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2320573Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2320870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2320953Z layer_outputs = layer_module( 2025-08-26T20:38:05.2321238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2321400Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2321690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2321816Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2322106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2322195Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2322199Z 2025-08-26T20:38:05.2322312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2322514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2322604Z return mod(**inputs) 2025-08-26T20:38:05.2322915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2322991Z outputs = self.mobilebert( 2025-08-26T20:38:05.2323282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2323355Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2323634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2323716Z layer_outputs = layer_module( 2025-08-26T20:38:05.2323995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2324162Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2324450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2324585Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2324868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2324992Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2325282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2325396Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2325399Z 2025-08-26T20:38:05.2325509Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2325729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2325804Z return mod(**inputs) 2025-08-26T20:38:05.2326088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2326160Z outputs = self.mobilebert( 2025-08-26T20:38:05.2326452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2326525Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2326812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2326885Z layer_outputs = layer_module( 2025-08-26T20:38:05.2327166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2327339Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2327621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2327743Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2328029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2328131Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2328134Z 2025-08-26T20:38:05.2328230Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2328418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2328490Z return mod(**inputs) 2025-08-26T20:38:05.2328761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2328839Z outputs = self.mobilebert( 2025-08-26T20:38:05.2329129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2329217Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2329499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2329570Z layer_outputs = layer_module( 2025-08-26T20:38:05.2329849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2330008Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2330291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2330400Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2330673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2330765Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2331048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2331146Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2331149Z 2025-08-26T20:38:05.2331250Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2331459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2331592Z return mod(**inputs) 2025-08-26T20:38:05.2331874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2331980Z outputs = self.mobilebert( 2025-08-26T20:38:05.2332255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2332334Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2332607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2332678Z layer_outputs = layer_module( 2025-08-26T20:38:05.2332962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2333049Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2333395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2333465Z self_outputs = self.self( 2025-08-26T20:38:05.2333739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2333820Z self.query(query_tensor) 2025-08-26T20:38:05.2333824Z 2025-08-26T20:38:05.2333923Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2334124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2334189Z return mod(**inputs) 2025-08-26T20:38:05.2334468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2334540Z outputs = self.mobilebert( 2025-08-26T20:38:05.2334819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2334898Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2335162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2335242Z layer_outputs = layer_module( 2025-08-26T20:38:05.2335531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2335642Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2335929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2336000Z self_outputs = self.self( 2025-08-26T20:38:05.2336282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2336350Z self.key(key_tensor) 2025-08-26T20:38:05.2336353Z 2025-08-26T20:38:05.2336453Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2336659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2336727Z return mod(**inputs) 2025-08-26T20:38:05.2337019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2337101Z outputs = self.mobilebert( 2025-08-26T20:38:05.2337382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2337453Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2337729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2337806Z layer_outputs = layer_module( 2025-08-26T20:38:05.2338094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2338191Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2338484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2338554Z self_outputs = self.self( 2025-08-26T20:38:05.2338836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2338906Z self.value(value_tensor) 2025-08-26T20:38:05.2338910Z 2025-08-26T20:38:05.2339001Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2339081Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2339187Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2339393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2339462Z return mod(**inputs) 2025-08-26T20:38:05.2339756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2339830Z outputs = self.mobilebert( 2025-08-26T20:38:05.2340118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2340191Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2340473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2340554Z layer_outputs = layer_module( 2025-08-26T20:38:05.2340833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2340924Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2341216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2341337Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2341635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2341722Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2341725Z 2025-08-26T20:38:05.2341850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2342047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2342121Z return mod(**inputs) 2025-08-26T20:38:05.2342404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2342481Z outputs = self.mobilebert( 2025-08-26T20:38:05.2342775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2342852Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2343142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2343218Z layer_outputs = layer_module( 2025-08-26T20:38:05.2343510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2343678Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2343954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2344072Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2344351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2344458Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2344462Z 2025-08-26T20:38:05.2344578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2344774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2344846Z return mod(**inputs) 2025-08-26T20:38:05.2345122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2345200Z outputs = self.mobilebert( 2025-08-26T20:38:05.2345472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2345543Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2345826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2345898Z layer_outputs = layer_module( 2025-08-26T20:38:05.2346177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2346261Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2346543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2346665Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2346938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2347067Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2347353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2347454Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2347457Z 2025-08-26T20:38:05.2347560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2347774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2347863Z return mod(**inputs) 2025-08-26T20:38:05.2348183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2348273Z outputs = self.mobilebert( 2025-08-26T20:38:05.2348570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2348655Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2348954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2349032Z layer_outputs = layer_module( 2025-08-26T20:38:05.2349333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2349439Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2349752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2349868Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2350152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2350245Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2350249Z 2025-08-26T20:38:05.2350353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2350562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2350649Z return mod(**inputs) 2025-08-26T20:38:05.2350945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2351050Z outputs = self.mobilebert( 2025-08-26T20:38:05.2351362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2351452Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2351760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2351847Z layer_outputs = layer_module( 2025-08-26T20:38:05.2352151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2352255Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2352575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2352699Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2353026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2353148Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2353152Z 2025-08-26T20:38:05.2353270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2353481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2353550Z return mod(**inputs) 2025-08-26T20:38:05.2353857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2353935Z outputs = self.mobilebert( 2025-08-26T20:38:05.2354236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2354312Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2354609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2354709Z layer_outputs = layer_module( 2025-08-26T20:38:05.2355021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2355133Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2355434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2355575Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2355874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2355967Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2355971Z 2025-08-26T20:38:05.2356091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2356304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2358845Z return mod(**inputs) 2025-08-26T20:38:05.2359179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2359262Z outputs = self.mobilebert( 2025-08-26T20:38:05.2359689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2359776Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2360093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2360202Z layer_outputs = layer_module( 2025-08-26T20:38:05.2360508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2360621Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2360928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2361108Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2361390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2361510Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2361820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2361925Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2361930Z 2025-08-26T20:38:05.2362044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2362272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2362345Z return mod(**inputs) 2025-08-26T20:38:05.2362661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2362743Z outputs = self.mobilebert( 2025-08-26T20:38:05.2363048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2363137Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2363444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2363533Z layer_outputs = layer_module( 2025-08-26T20:38:05.2363838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2363951Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2364287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2364416Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2364752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2364847Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2364851Z 2025-08-26T20:38:05.2364971Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2365191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2365268Z return mod(**inputs) 2025-08-26T20:38:05.2365586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2365665Z outputs = self.mobilebert( 2025-08-26T20:38:05.2365984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2366152Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2366464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2366551Z layer_outputs = layer_module( 2025-08-26T20:38:05.2366856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2366967Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2367290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2367419Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2367730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2367846Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2367851Z 2025-08-26T20:38:05.2367963Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2368165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2368239Z return mod(**inputs) 2025-08-26T20:38:05.2368537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2368622Z outputs = self.mobilebert( 2025-08-26T20:38:05.2368923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2369001Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2369308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2369385Z layer_outputs = layer_module( 2025-08-26T20:38:05.2369692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2369793Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2370087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2370228Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2370522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2370622Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2370626Z 2025-08-26T20:38:05.2370733Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2370951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2371039Z return mod(**inputs) 2025-08-26T20:38:05.2371357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2371445Z outputs = self.mobilebert( 2025-08-26T20:38:05.2371743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2371828Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2372126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2372202Z layer_outputs = layer_module( 2025-08-26T20:38:05.2372509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2372610Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2372914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2373080Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2373380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2373510Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2373804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2373928Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2373932Z 2025-08-26T20:38:05.2374040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2374255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2374325Z return mod(**inputs) 2025-08-26T20:38:05.2374624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2374710Z outputs = self.mobilebert( 2025-08-26T20:38:05.2375003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2375085Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2375380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2375464Z layer_outputs = layer_module( 2025-08-26T20:38:05.2375759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2375858Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2376166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2376287Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2376588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2376681Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2376685Z 2025-08-26T20:38:05.2376796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2377020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2377095Z return mod(**inputs) 2025-08-26T20:38:05.2377409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2377491Z outputs = self.mobilebert( 2025-08-26T20:38:05.2377816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2377900Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2378221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2378308Z layer_outputs = layer_module( 2025-08-26T20:38:05.2378615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2378726Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2379036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2379157Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2379477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2379597Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2379622Z 2025-08-26T20:38:05.2379741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2379953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2380033Z return mod(**inputs) 2025-08-26T20:38:05.2380330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2380409Z outputs = self.mobilebert( 2025-08-26T20:38:05.2380731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2380809Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2381108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2381185Z layer_outputs = layer_module( 2025-08-26T20:38:05.2381482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2381590Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2381885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2382025Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2382329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2382428Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2382431Z 2025-08-26T20:38:05.2382540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2382748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2382828Z return mod(**inputs) 2025-08-26T20:38:05.2383129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2383212Z outputs = self.mobilebert( 2025-08-26T20:38:05.2383504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2383582Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2383958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2384038Z layer_outputs = layer_module( 2025-08-26T20:38:05.2384348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2384450Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2384794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2384944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2385243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2385381Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2385686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2385793Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2385797Z 2025-08-26T20:38:05.2385905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2386116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2386193Z return mod(**inputs) 2025-08-26T20:38:05.2386496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2386602Z outputs = self.mobilebert( 2025-08-26T20:38:05.2386897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2386982Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2387289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2387398Z layer_outputs = layer_module( 2025-08-26T20:38:05.2387709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2387841Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2388149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2388246Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2388251Z 2025-08-26T20:38:05.2388363Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2388583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2388655Z return mod(**inputs) 2025-08-26T20:38:05.2388964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2389056Z outputs = self.mobilebert( 2025-08-26T20:38:05.2389357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2389435Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2389730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2389814Z layer_outputs = layer_module( 2025-08-26T20:38:05.2390107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2390243Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2390549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2390670Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2390676Z 2025-08-26T20:38:05.2390795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2391008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2391086Z return mod(**inputs) 2025-08-26T20:38:05.2391412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2391502Z outputs = self.mobilebert( 2025-08-26T20:38:05.2391826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2391906Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2392216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2392295Z layer_outputs = layer_module( 2025-08-26T20:38:05.2392607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2392784Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2393091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2393203Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2393226Z 2025-08-26T20:38:05.2393340Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2393562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2393634Z return mod(**inputs) 2025-08-26T20:38:05.2393948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2394027Z outputs = self.mobilebert( 2025-08-26T20:38:05.2394335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2394445Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2394754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2394844Z layer_outputs = layer_module( 2025-08-26T20:38:05.2395151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2395328Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2395645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2395780Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2396099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2396348Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2396355Z 2025-08-26T20:38:05.2396481Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2396704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2396780Z return mod(**inputs) 2025-08-26T20:38:05.2397102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2397182Z outputs = self.mobilebert( 2025-08-26T20:38:05.2397497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2397577Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2397882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2397973Z layer_outputs = layer_module( 2025-08-26T20:38:05.2398281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2398512Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2398847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2398992Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2399297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2399391Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2399395Z 2025-08-26T20:38:05.2399564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2399789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2399878Z return mod(**inputs) 2025-08-26T20:38:05.2400184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2400264Z outputs = self.mobilebert( 2025-08-26T20:38:05.2400591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2400711Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2401016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2401094Z layer_outputs = layer_module( 2025-08-26T20:38:05.2401409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2401614Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2401920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2402067Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2402379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2402525Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2402836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2402945Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2402949Z 2025-08-26T20:38:05.2403061Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2403279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2403360Z return mod(**inputs) 2025-08-26T20:38:05.2403672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2403758Z outputs = self.mobilebert( 2025-08-26T20:38:05.2404068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2404150Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2404463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2404541Z layer_outputs = layer_module( 2025-08-26T20:38:05.2404854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2405033Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2405346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2405467Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2405790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2405913Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2405917Z 2025-08-26T20:38:05.2406031Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2406256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2406327Z return mod(**inputs) 2025-08-26T20:38:05.2406638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2406728Z outputs = self.mobilebert( 2025-08-26T20:38:05.2407035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2407119Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2407429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2407528Z layer_outputs = layer_module( 2025-08-26T20:38:05.2407848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2408025Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2408350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2408474Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2408798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2408885Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2409171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2409279Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2409285Z 2025-08-26T20:38:05.2409407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2409616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2409686Z return mod(**inputs) 2025-08-26T20:38:05.2409995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2410073Z outputs = self.mobilebert( 2025-08-26T20:38:05.2410376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2410453Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2410748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2410833Z layer_outputs = layer_module( 2025-08-26T20:38:05.2411129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2411235Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2411518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2411599Z self_outputs = self.self( 2025-08-26T20:38:05.2411892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2411971Z self.query(query_tensor) 2025-08-26T20:38:05.2411975Z 2025-08-26T20:38:05.2412093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2412303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2412399Z return mod(**inputs) 2025-08-26T20:38:05.2412714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2412792Z outputs = self.mobilebert( 2025-08-26T20:38:05.2413094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2413170Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2413475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2413553Z layer_outputs = layer_module( 2025-08-26T20:38:05.2413849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2413943Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2414227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2414337Z self_outputs = self.self( 2025-08-26T20:38:05.2414619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2414694Z self.key(key_tensor) 2025-08-26T20:38:05.2414697Z 2025-08-26T20:38:05.2414799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2414995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2415092Z return mod(**inputs) 2025-08-26T20:38:05.2415377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2415454Z outputs = self.mobilebert( 2025-08-26T20:38:05.2415755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2415834Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2416137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2416211Z layer_outputs = layer_module( 2025-08-26T20:38:05.2416514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2416603Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2416902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2416975Z self_outputs = self.self( 2025-08-26T20:38:05.2417253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2417334Z self.value(value_tensor) 2025-08-26T20:38:05.2417337Z 2025-08-26T20:38:05.2417422Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2417510Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2417612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2417812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2417889Z return mod(**inputs) 2025-08-26T20:38:05.2418182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2418267Z outputs = self.mobilebert( 2025-08-26T20:38:05.2418564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2418640Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2418960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2419038Z layer_outputs = layer_module( 2025-08-26T20:38:05.2419355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2419446Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2419750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2419881Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2420181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2420277Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2420281Z 2025-08-26T20:38:05.2420384Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2420592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2420676Z return mod(**inputs) 2025-08-26T20:38:05.2420959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2421041Z outputs = self.mobilebert( 2025-08-26T20:38:05.2421318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2421399Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2421685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2421782Z layer_outputs = layer_module( 2025-08-26T20:38:05.2422074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2422248Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2422562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2422686Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2423010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2423099Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2423103Z 2025-08-26T20:38:05.2423218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2423430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2423500Z return mod(**inputs) 2025-08-26T20:38:05.2423810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2423884Z outputs = self.mobilebert( 2025-08-26T20:38:05.2424175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2424247Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2424525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2424605Z layer_outputs = layer_module( 2025-08-26T20:38:05.2424884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2424976Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2425256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2425379Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2425679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2425826Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2426114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2426205Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2426209Z 2025-08-26T20:38:05.2426315Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2426513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2426581Z return mod(**inputs) 2025-08-26T20:38:05.2426869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2426942Z outputs = self.mobilebert( 2025-08-26T20:38:05.2427230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2427324Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2427608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2427689Z layer_outputs = layer_module( 2025-08-26T20:38:05.2427966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2428072Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2428374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2428496Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2428800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2428893Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2428897Z 2025-08-26T20:38:05.2429015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2429226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2429305Z return mod(**inputs) 2025-08-26T20:38:05.2429601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2429680Z outputs = self.mobilebert( 2025-08-26T20:38:05.2429993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2430068Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2430354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2430429Z layer_outputs = layer_module( 2025-08-26T20:38:05.2430729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2430830Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2431127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2431257Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2431562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2431692Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2431696Z 2025-08-26T20:38:05.2431803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2432035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2432107Z return mod(**inputs) 2025-08-26T20:38:05.2432422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2432507Z outputs = self.mobilebert( 2025-08-26T20:38:05.2432801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2432884Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2433180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2433259Z layer_outputs = layer_module( 2025-08-26T20:38:05.2433562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2433666Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2433985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2434118Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2434412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2434510Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2434513Z 2025-08-26T20:38:05.2434620Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2434858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2434927Z return mod(**inputs) 2025-08-26T20:38:05.2435235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2435313Z outputs = self.mobilebert( 2025-08-26T20:38:05.2435613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2435698Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2435995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2436077Z layer_outputs = layer_module( 2025-08-26T20:38:05.2436373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2436477Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2436783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2436915Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2437222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2437355Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2437666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2437768Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2437772Z 2025-08-26T20:38:05.2437884Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2438111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2438186Z return mod(**inputs) 2025-08-26T20:38:05.2438505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2438585Z outputs = self.mobilebert( 2025-08-26T20:38:05.2438910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2439027Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2439338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2439425Z layer_outputs = layer_module( 2025-08-26T20:38:05.2439804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2439921Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2440231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2440354Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2440669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2440789Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2440793Z 2025-08-26T20:38:05.2440917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2441149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2441221Z return mod(**inputs) 2025-08-26T20:38:05.2441533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2441610Z outputs = self.mobilebert( 2025-08-26T20:38:05.2441939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2442017Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2442324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2442403Z layer_outputs = layer_module( 2025-08-26T20:38:05.2442706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2442814Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2443109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2443234Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2443529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2443651Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2443663Z 2025-08-26T20:38:05.2443772Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2443986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2444071Z return mod(**inputs) 2025-08-26T20:38:05.2444348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2444428Z outputs = self.mobilebert( 2025-08-26T20:38:05.2444701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2444771Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2445049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2445120Z layer_outputs = layer_module( 2025-08-26T20:38:05.2445399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2445509Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2445808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2445943Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2446224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2446319Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2446323Z 2025-08-26T20:38:05.2446426Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2446637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2446704Z return mod(**inputs) 2025-08-26T20:38:05.2446989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2447073Z outputs = self.mobilebert( 2025-08-26T20:38:05.2447374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2447454Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2447732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2447805Z layer_outputs = layer_module( 2025-08-26T20:38:05.2448090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2448202Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2448491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2448626Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2448905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2449027Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2449302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2449402Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2449405Z 2025-08-26T20:38:05.2449505Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2449709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2449778Z return mod(**inputs) 2025-08-26T20:38:05.2450067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2450140Z outputs = self.mobilebert( 2025-08-26T20:38:05.2450423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2450505Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2450787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2450865Z layer_outputs = layer_module( 2025-08-26T20:38:05.2451143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2451237Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2451526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2451637Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2451941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2452029Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2452032Z 2025-08-26T20:38:05.2452155Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2452361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2452432Z return mod(**inputs) 2025-08-26T20:38:05.2452735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2452812Z outputs = self.mobilebert( 2025-08-26T20:38:05.2453115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2453192Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2453485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2453598Z layer_outputs = layer_module( 2025-08-26T20:38:05.2453895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2454003Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2454300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2454412Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2454697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2454838Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2454842Z 2025-08-26T20:38:05.2454949Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2455147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2455220Z return mod(**inputs) 2025-08-26T20:38:05.2455496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2455567Z outputs = self.mobilebert( 2025-08-26T20:38:05.2455846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2455918Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2456202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2456276Z layer_outputs = layer_module( 2025-08-26T20:38:05.2456553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2456656Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2456935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2457070Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2457349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2457441Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2457445Z 2025-08-26T20:38:05.2457549Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2457747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2457823Z return mod(**inputs) 2025-08-26T20:38:05.2458123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2458209Z outputs = self.mobilebert( 2025-08-26T20:38:05.2458520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2458619Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2458927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2459004Z layer_outputs = layer_module( 2025-08-26T20:38:05.2459307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2459408Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2459712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2459843Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2460141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2460298Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2460593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2460698Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2460702Z 2025-08-26T20:38:05.2460811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2461029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2461121Z return mod(**inputs) 2025-08-26T20:38:05.2461427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2461514Z outputs = self.mobilebert( 2025-08-26T20:38:05.2461816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2461905Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2462208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2462284Z layer_outputs = layer_module( 2025-08-26T20:38:05.2462595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2462726Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2463037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2463127Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2463130Z 2025-08-26T20:38:05.2463246Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2463462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2463533Z return mod(**inputs) 2025-08-26T20:38:05.2463851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2463929Z outputs = self.mobilebert( 2025-08-26T20:38:05.2464242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2464320Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2464626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2464710Z layer_outputs = layer_module( 2025-08-26T20:38:05.2465011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2465171Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2465489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2465618Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2465622Z 2025-08-26T20:38:05.2465728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2465940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2466019Z return mod(**inputs) 2025-08-26T20:38:05.2466323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2466407Z outputs = self.mobilebert( 2025-08-26T20:38:05.2466704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2466784Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2467109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2467184Z layer_outputs = layer_module( 2025-08-26T20:38:05.2467487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2467657Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2467962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2468085Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2468090Z 2025-08-26T20:38:05.2468199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2468421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2468492Z return mod(**inputs) 2025-08-26T20:38:05.2468799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2468876Z outputs = self.mobilebert( 2025-08-26T20:38:05.2469169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2469253Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2469552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2469639Z layer_outputs = layer_module( 2025-08-26T20:38:05.2469934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2470112Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2470415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2470549Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2470856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2470954Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2470958Z 2025-08-26T20:38:05.2471073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2471287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2471357Z return mod(**inputs) 2025-08-26T20:38:05.2471669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2471765Z outputs = self.mobilebert( 2025-08-26T20:38:05.2472117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2472197Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2472500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2472573Z layer_outputs = layer_module( 2025-08-26T20:38:05.2472857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2473271Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2473712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2473881Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2474202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2474346Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2474358Z 2025-08-26T20:38:05.2474470Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2474680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2474760Z return mod(**inputs) 2025-08-26T20:38:05.2475061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2475162Z outputs = self.mobilebert( 2025-08-26T20:38:05.2475467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2475543Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2475853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2475932Z layer_outputs = layer_module( 2025-08-26T20:38:05.2476245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2476411Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2476720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2476872Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2477180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2477317Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2477629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2477737Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2477741Z 2025-08-26T20:38:05.2477852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2478065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2478143Z return mod(**inputs) 2025-08-26T20:38:05.2478452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2478541Z outputs = self.mobilebert( 2025-08-26T20:38:05.2478857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2478944Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2479276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2479358Z layer_outputs = layer_module( 2025-08-26T20:38:05.2479747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2479933Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2480260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2480382Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2480703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2480799Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2480803Z 2025-08-26T20:38:05.2480914Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2481151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2481222Z return mod(**inputs) 2025-08-26T20:38:05.2481533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2481610Z outputs = self.mobilebert( 2025-08-26T20:38:05.2481919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2482028Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2482327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2482408Z layer_outputs = layer_module( 2025-08-26T20:38:05.2482710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2482884Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2483204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2483322Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2483633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2483728Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2484038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2484136Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2484140Z 2025-08-26T20:38:05.2484248Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2484471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2484543Z return mod(**inputs) 2025-08-26T20:38:05.2484853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2484931Z outputs = self.mobilebert( 2025-08-26T20:38:05.2485231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2485315Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2485625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2485707Z layer_outputs = layer_module( 2025-08-26T20:38:05.2486008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2486123Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2486438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2486515Z self_outputs = self.self( 2025-08-26T20:38:05.2486823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2486900Z self.query(query_tensor) 2025-08-26T20:38:05.2486904Z 2025-08-26T20:38:05.2487019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2487233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2487303Z return mod(**inputs) 2025-08-26T20:38:05.2487626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2487702Z outputs = self.mobilebert( 2025-08-26T20:38:05.2488007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2488104Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2488400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2488484Z layer_outputs = layer_module( 2025-08-26T20:38:05.2488787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2488913Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2489208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2489289Z self_outputs = self.self( 2025-08-26T20:38:05.2489583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2489656Z self.key(key_tensor) 2025-08-26T20:38:05.2489659Z 2025-08-26T20:38:05.2489776Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2489981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2490057Z return mod(**inputs) 2025-08-26T20:38:05.2490354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2490434Z outputs = self.mobilebert( 2025-08-26T20:38:05.2490738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2490814Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2491123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2491199Z layer_outputs = layer_module( 2025-08-26T20:38:05.2491505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2491595Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2491890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2491972Z self_outputs = self.self( 2025-08-26T20:38:05.2492276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2492360Z self.value(value_tensor) 2025-08-26T20:38:05.2492364Z 2025-08-26T20:38:05.2492452Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2492538Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2492655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2492877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2492957Z return mod(**inputs) 2025-08-26T20:38:05.2493276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2493355Z outputs = self.mobilebert( 2025-08-26T20:38:05.2493657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2493733Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2494037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2494112Z layer_outputs = layer_module( 2025-08-26T20:38:05.2494411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2494502Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2494818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2494956Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2495250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2495348Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2495351Z 2025-08-26T20:38:05.2495458Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2495688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2495765Z return mod(**inputs) 2025-08-26T20:38:05.2496066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2496323Z outputs = self.mobilebert( 2025-08-26T20:38:05.2496638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2496725Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2497025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2497102Z layer_outputs = layer_module( 2025-08-26T20:38:05.2497408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2497581Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2497888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2498009Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2498310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2498409Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2498413Z 2025-08-26T20:38:05.2498525Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2498744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2498815Z return mod(**inputs) 2025-08-26T20:38:05.2499123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2499203Z outputs = self.mobilebert( 2025-08-26T20:38:05.2499501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2499591Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2499949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2500060Z layer_outputs = layer_module( 2025-08-26T20:38:05.2500362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2500453Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2500761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2500892Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2501195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2501330Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2501639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2501768Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2501772Z 2025-08-26T20:38:05.2501881Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2502098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2502169Z return mod(**inputs) 2025-08-26T20:38:05.2502477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2502581Z outputs = self.mobilebert( 2025-08-26T20:38:05.2502876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2502954Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2503228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2503308Z layer_outputs = layer_module( 2025-08-26T20:38:05.2503580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2503681Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2503963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2504072Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2504347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2504428Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2504431Z 2025-08-26T20:38:05.2504534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2504721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2504784Z return mod(**inputs) 2025-08-26T20:38:05.2505064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2505134Z outputs = self.mobilebert( 2025-08-26T20:38:05.2505411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2505482Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2505759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2505829Z layer_outputs = layer_module( 2025-08-26T20:38:05.2506104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2506221Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2506512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2506630Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2506904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2507019Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2507029Z 2025-08-26T20:38:05.2507134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2507335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2507409Z return mod(**inputs) 2025-08-26T20:38:05.2507713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2507799Z outputs = self.mobilebert( 2025-08-26T20:38:05.2508113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2508191Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2508495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2508571Z layer_outputs = layer_module( 2025-08-26T20:38:05.2508880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2509012Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2509293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2509430Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2509714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2509809Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2509812Z 2025-08-26T20:38:05.2509914Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2510116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2510181Z return mod(**inputs) 2025-08-26T20:38:05.2510465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2510549Z outputs = self.mobilebert( 2025-08-26T20:38:05.2510836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2510914Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2511191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2511264Z layer_outputs = layer_module( 2025-08-26T20:38:05.2511549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2511646Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2511934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2512068Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2512368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2512496Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2512811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2512932Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2512936Z 2025-08-26T20:38:05.2513039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2513242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2513310Z return mod(**inputs) 2025-08-26T20:38:05.2513592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2513672Z outputs = self.mobilebert( 2025-08-26T20:38:05.2513956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2514041Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2514340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2514453Z layer_outputs = layer_module( 2025-08-26T20:38:05.2514756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2514857Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2515162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2515282Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2515606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2515697Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2515701Z 2025-08-26T20:38:05.2515817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2516026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2516098Z return mod(**inputs) 2025-08-26T20:38:05.2516399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2516476Z outputs = self.mobilebert( 2025-08-26T20:38:05.2516774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2516850Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2517146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2517227Z layer_outputs = layer_module( 2025-08-26T20:38:05.2517518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2517626Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2517920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2518035Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2518339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2518455Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2518459Z 2025-08-26T20:38:05.2518578Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2518784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2518859Z return mod(**inputs) 2025-08-26T20:38:05.2519158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2519251Z outputs = self.mobilebert( 2025-08-26T20:38:05.2519803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2519887Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2520195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2520271Z layer_outputs = layer_module( 2025-08-26T20:38:05.2520565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2520675Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2520970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2521112Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2521409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2521531Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2521535Z 2025-08-26T20:38:05.2521647Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2521860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2521939Z return mod(**inputs) 2025-08-26T20:38:05.2522239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2522346Z outputs = self.mobilebert( 2025-08-26T20:38:05.2522642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2522718Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2523022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2523103Z layer_outputs = layer_module( 2025-08-26T20:38:05.2523406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2523506Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2523808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2523944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2524238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2524375Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2524671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2524781Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2524785Z 2025-08-26T20:38:05.2524894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2525113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2525183Z return mod(**inputs) 2025-08-26T20:38:05.2525483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2525569Z outputs = self.mobilebert( 2025-08-26T20:38:05.2525864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2525948Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2526278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2526358Z layer_outputs = layer_module( 2025-08-26T20:38:05.2526681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2526784Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2527086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2527205Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2527506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2527607Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2527610Z 2025-08-26T20:38:05.2527710Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2527911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2527994Z return mod(**inputs) 2025-08-26T20:38:05.2528285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2528358Z outputs = self.mobilebert( 2025-08-26T20:38:05.2528640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2528721Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2529022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2529099Z layer_outputs = layer_module( 2025-08-26T20:38:05.2529368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2529461Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2529749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2529861Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2530156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2530264Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2530267Z 2025-08-26T20:38:05.2530372Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2530567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2530632Z return mod(**inputs) 2025-08-26T20:38:05.2530914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2530987Z outputs = self.mobilebert( 2025-08-26T20:38:05.2531268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2531339Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2531611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2531688Z layer_outputs = layer_module( 2025-08-26T20:38:05.2531958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2532060Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2532330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2532457Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2532748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2532848Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2532851Z 2025-08-26T20:38:05.2532960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2533154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2533226Z return mod(**inputs) 2025-08-26T20:38:05.2533497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2533569Z outputs = self.mobilebert( 2025-08-26T20:38:05.2533852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2533923Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2534205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2534950Z layer_outputs = layer_module( 2025-08-26T20:38:05.2535230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2535322Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2535595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2535745Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2536016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2536141Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2536414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2536515Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2536518Z 2025-08-26T20:38:05.2536619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2536814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2536886Z return mod(**inputs) 2025-08-26T20:38:05.2537160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2537237Z outputs = self.mobilebert( 2025-08-26T20:38:05.2537511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2537581Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2537862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2537932Z layer_outputs = layer_module( 2025-08-26T20:38:05.2538217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2538336Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2538609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2538699Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2538704Z 2025-08-26T20:38:05.2538803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2539005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2539069Z return mod(**inputs) 2025-08-26T20:38:05.2539368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2539444Z outputs = self.mobilebert( 2025-08-26T20:38:05.2539742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2539823Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2540096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2540173Z layer_outputs = layer_module( 2025-08-26T20:38:05.2540444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2540563Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2540843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2540955Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2540975Z 2025-08-26T20:38:05.2541088Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2541284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2541357Z return mod(**inputs) 2025-08-26T20:38:05.2541639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2541710Z outputs = self.mobilebert( 2025-08-26T20:38:05.2541992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2542081Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2542360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2542433Z layer_outputs = layer_module( 2025-08-26T20:38:05.2542712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2542876Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2543149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2543251Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2543255Z 2025-08-26T20:38:05.2543355Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2543566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2543636Z return mod(**inputs) 2025-08-26T20:38:05.2543933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2544024Z outputs = self.mobilebert( 2025-08-26T20:38:05.2544325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2544406Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2544684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2544757Z layer_outputs = layer_module( 2025-08-26T20:38:05.2545043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2545203Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2545490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2545631Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2545939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2546045Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2546048Z 2025-08-26T20:38:05.2546146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2546345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2546410Z return mod(**inputs) 2025-08-26T20:38:05.2546692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2546764Z outputs = self.mobilebert( 2025-08-26T20:38:05.2547039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2547110Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2547387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2547486Z layer_outputs = layer_module( 2025-08-26T20:38:05.2547764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2547929Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2548210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2548353Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2548652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2548744Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2548747Z 2025-08-26T20:38:05.2548866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2549089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2549164Z return mod(**inputs) 2025-08-26T20:38:05.2549447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2549518Z outputs = self.mobilebert( 2025-08-26T20:38:05.2549805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2549878Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2550163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2550235Z layer_outputs = layer_module( 2025-08-26T20:38:05.2550514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2550682Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2550962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2551099Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2551370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2551495Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2551766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2551857Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2551860Z 2025-08-26T20:38:05.2551986Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2552199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2552271Z return mod(**inputs) 2025-08-26T20:38:05.2552549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2552619Z outputs = self.mobilebert( 2025-08-26T20:38:05.2552912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2552986Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2553273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2553344Z layer_outputs = layer_module( 2025-08-26T20:38:05.2553642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2553833Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2554132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2554264Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2554568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2554696Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2554700Z 2025-08-26T20:38:05.2554808Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2555024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2555096Z return mod(**inputs) 2025-08-26T20:38:05.2555396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2555486Z outputs = self.mobilebert( 2025-08-26T20:38:05.2555789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2555875Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2556179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2556259Z layer_outputs = layer_module( 2025-08-26T20:38:05.2556575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2556751Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2557074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2557196Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2557522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2557617Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2557929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2558037Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2558042Z 2025-08-26T20:38:05.2558153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2558386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2558458Z return mod(**inputs) 2025-08-26T20:38:05.2558782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2558873Z outputs = self.mobilebert( 2025-08-26T20:38:05.2559207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2559296Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2559692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2559774Z layer_outputs = layer_module( 2025-08-26T20:38:05.2560102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2560199Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2560528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2560607Z self_outputs = self.self( 2025-08-26T20:38:05.2560946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2561025Z self.query(query_tensor) 2025-08-26T20:38:05.2561029Z 2025-08-26T20:38:05.2561143Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2561378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2561451Z return mod(**inputs) 2025-08-26T20:38:05.2561777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2561875Z outputs = self.mobilebert( 2025-08-26T20:38:05.2562193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2562284Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2562603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2562694Z layer_outputs = layer_module( 2025-08-26T20:38:05.2563012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2563115Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2563431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2563508Z self_outputs = self.self( 2025-08-26T20:38:05.2563824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2563898Z self.key(key_tensor) 2025-08-26T20:38:05.2563902Z 2025-08-26T20:38:05.2564019Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2564249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2564322Z return mod(**inputs) 2025-08-26T20:38:05.2564642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2564719Z outputs = self.mobilebert( 2025-08-26T20:38:05.2565034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2565114Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2565441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2565519Z layer_outputs = layer_module( 2025-08-26T20:38:05.2565838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2565970Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2566305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2566392Z self_outputs = self.self( 2025-08-26T20:38:05.2566703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2566780Z self.value(value_tensor) 2025-08-26T20:38:05.2566784Z 2025-08-26T20:38:05.2566883Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2566970Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2567091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2567305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2567376Z return mod(**inputs) 2025-08-26T20:38:05.2567694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2567788Z outputs = self.mobilebert( 2025-08-26T20:38:05.2568106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2568185Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2568503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2568581Z layer_outputs = layer_module( 2025-08-26T20:38:05.2568899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2569019Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2569331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2569472Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2569771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2569861Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2569865Z 2025-08-26T20:38:05.2569980Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2570188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2570264Z return mod(**inputs) 2025-08-26T20:38:05.2570559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2570644Z outputs = self.mobilebert( 2025-08-26T20:38:05.2570939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2571018Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2571322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2571398Z layer_outputs = layer_module( 2025-08-26T20:38:05.2571697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2571870Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2572175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2572301Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2572594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2572688Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2572709Z 2025-08-26T20:38:05.2572821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2573056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2573129Z return mod(**inputs) 2025-08-26T20:38:05.2573432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2573518Z outputs = self.mobilebert( 2025-08-26T20:38:05.2573815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2573901Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2574201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2574278Z layer_outputs = layer_module( 2025-08-26T20:38:05.2574581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2574693Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2574999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2575129Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2575442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2575598Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2575895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2576002Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2576006Z 2025-08-26T20:38:05.2576115Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2576335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2576407Z return mod(**inputs) 2025-08-26T20:38:05.2576709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2576795Z outputs = self.mobilebert( 2025-08-26T20:38:05.2577102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2577190Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2577488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2577571Z layer_outputs = layer_module( 2025-08-26T20:38:05.2577868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2577975Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2578278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2578399Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2578703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2578793Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2578799Z 2025-08-26T20:38:05.2578907Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2579123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2579195Z return mod(**inputs) 2025-08-26T20:38:05.2579553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2579634Z outputs = self.mobilebert( 2025-08-26T20:38:05.2579952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2580031Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2580327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2580413Z layer_outputs = layer_module( 2025-08-26T20:38:05.2580707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2580820Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2581115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2581238Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2581567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2581687Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2581691Z 2025-08-26T20:38:05.2581808Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2582018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2582097Z return mod(**inputs) 2025-08-26T20:38:05.2582396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2582492Z outputs = self.mobilebert( 2025-08-26T20:38:05.2582797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2582876Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2583184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2583263Z layer_outputs = layer_module( 2025-08-26T20:38:05.2583570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2583684Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2583988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2584136Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2584441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2584551Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2584554Z 2025-08-26T20:38:05.2584663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2584875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2584954Z return mod(**inputs) 2025-08-26T20:38:05.2585258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2585343Z outputs = self.mobilebert( 2025-08-26T20:38:05.2585648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2585729Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2586040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2586117Z layer_outputs = layer_module( 2025-08-26T20:38:05.2586442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2586565Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2586876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2587014Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2587317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2587460Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2587759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2587868Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2587872Z 2025-08-26T20:38:05.2587986Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2588220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2588301Z return mod(**inputs) 2025-08-26T20:38:05.2588609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2588693Z outputs = self.mobilebert( 2025-08-26T20:38:05.2588997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2589112Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2589416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2589494Z layer_outputs = layer_module( 2025-08-26T20:38:05.2589812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2589918Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2590232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2590354Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2590709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2590814Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2590821Z 2025-08-26T20:38:05.2590932Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2591156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2591229Z return mod(**inputs) 2025-08-26T20:38:05.2591544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2591624Z outputs = self.mobilebert( 2025-08-26T20:38:05.2591929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2592016Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2592319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2592405Z layer_outputs = layer_module( 2025-08-26T20:38:05.2592710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2592815Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2593128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2593272Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2593606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2593732Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2593735Z 2025-08-26T20:38:05.2593857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2594076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2594150Z return mod(**inputs) 2025-08-26T20:38:05.2594467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2594548Z outputs = self.mobilebert( 2025-08-26T20:38:05.2594860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2594941Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2595267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2595352Z layer_outputs = layer_module( 2025-08-26T20:38:05.2595654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2595764Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2596067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2596390Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2596702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2596796Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2596801Z 2025-08-26T20:38:05.2596927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2597148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2597228Z return mod(**inputs) 2025-08-26T20:38:05.2597533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2597613Z outputs = self.mobilebert( 2025-08-26T20:38:05.2597929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2598010Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2598321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2598413Z layer_outputs = layer_module( 2025-08-26T20:38:05.2598857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2598964Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2599366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2599560Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2599885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2600054Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2600361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2600463Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2600475Z 2025-08-26T20:38:05.2600649Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2600873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2600980Z return mod(**inputs) 2025-08-26T20:38:05.2601290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2601377Z outputs = self.mobilebert( 2025-08-26T20:38:05.2601682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2601763Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2602076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2602153Z layer_outputs = layer_module( 2025-08-26T20:38:05.2602469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2602604Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2602911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2603044Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2603352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2603451Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2603482Z 2025-08-26T20:38:05.2603595Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2603826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2603896Z return mod(**inputs) 2025-08-26T20:38:05.2604199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2604286Z outputs = self.mobilebert( 2025-08-26T20:38:05.2604586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2604666Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2604942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2605013Z layer_outputs = layer_module( 2025-08-26T20:38:05.2605301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2605396Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2605683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2605796Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2606085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2606199Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2606202Z 2025-08-26T20:38:05.2606307Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2606517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2606582Z return mod(**inputs) 2025-08-26T20:38:05.2606878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2606956Z outputs = self.mobilebert( 2025-08-26T20:38:05.2607252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2607356Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2607671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2607756Z layer_outputs = layer_module( 2025-08-26T20:38:05.2608057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2608163Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2608459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2608597Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2608904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2608995Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2608999Z 2025-08-26T20:38:05.2609119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2609349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2609420Z return mod(**inputs) 2025-08-26T20:38:05.2609729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2609806Z outputs = self.mobilebert( 2025-08-26T20:38:05.2610110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2610207Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2610510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2610593Z layer_outputs = layer_module( 2025-08-26T20:38:05.2610877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2610982Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2611267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2611407Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2611705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2611835Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2612142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2612240Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2612243Z 2025-08-26T20:38:05.2612363Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2612576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2612656Z return mod(**inputs) 2025-08-26T20:38:05.2612958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2613035Z outputs = self.mobilebert( 2025-08-26T20:38:05.2613357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2613430Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2613721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2613793Z layer_outputs = layer_module( 2025-08-26T20:38:05.2614106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2614248Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2614560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2614658Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2614661Z 2025-08-26T20:38:05.2614770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2614988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2615057Z return mod(**inputs) 2025-08-26T20:38:05.2615359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2615442Z outputs = self.mobilebert( 2025-08-26T20:38:05.2615744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2615867Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2616150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2616222Z layer_outputs = layer_module( 2025-08-26T20:38:05.2616509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2616630Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2616917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2617051Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2617054Z 2025-08-26T20:38:05.2617173Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2617366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2617433Z return mod(**inputs) 2025-08-26T20:38:05.2617715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2617787Z outputs = self.mobilebert( 2025-08-26T20:38:05.2618071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2618143Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2618418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2618497Z layer_outputs = layer_module( 2025-08-26T20:38:05.2618772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2618945Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2619238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2619347Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2619351Z 2025-08-26T20:38:05.2619459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2619666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2619743Z return mod(**inputs) 2025-08-26T20:38:05.2620038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2620123Z outputs = self.mobilebert( 2025-08-26T20:38:05.2620417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2620492Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2620807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2620911Z layer_outputs = layer_module( 2025-08-26T20:38:05.2621198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2621355Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2621637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2621763Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2622041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2622138Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2622144Z 2025-08-26T20:38:05.2622266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2622473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2622538Z return mod(**inputs) 2025-08-26T20:38:05.2622823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2622902Z outputs = self.mobilebert( 2025-08-26T20:38:05.2623185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2623282Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2623560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2623637Z layer_outputs = layer_module( 2025-08-26T20:38:05.2623920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2624090Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2624395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2624529Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2624831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2624919Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2624923Z 2025-08-26T20:38:05.2625033Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2625232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2625299Z return mod(**inputs) 2025-08-26T20:38:05.2625594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2625669Z outputs = self.mobilebert( 2025-08-26T20:38:05.2625954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2626028Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2626310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2626394Z layer_outputs = layer_module( 2025-08-26T20:38:05.2626687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2626861Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2627173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2627330Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2627630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2627769Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2628059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2628155Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2628159Z 2025-08-26T20:38:05.2628269Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2628469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2628536Z return mod(**inputs) 2025-08-26T20:38:05.2628830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2628922Z outputs = self.mobilebert( 2025-08-26T20:38:05.2629209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2629282Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2629569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2629641Z layer_outputs = layer_module( 2025-08-26T20:38:05.2629938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2630110Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2630392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2630514Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2630795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2630879Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2630890Z 2025-08-26T20:38:05.2630992Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2631190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2631268Z return mod(**inputs) 2025-08-26T20:38:05.2631552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2631632Z outputs = self.mobilebert( 2025-08-26T20:38:05.2631915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2631987Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2632274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2632346Z layer_outputs = layer_module( 2025-08-26T20:38:05.2632634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2632795Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2633077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2633199Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2633511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2633616Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2633934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2634040Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2634044Z 2025-08-26T20:38:05.2634153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2634364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2634441Z return mod(**inputs) 2025-08-26T20:38:05.2634743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2634826Z outputs = self.mobilebert( 2025-08-26T20:38:05.2635122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2635198Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2635521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2635595Z layer_outputs = layer_module( 2025-08-26T20:38:05.2635897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2635987Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2636290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2636386Z self_outputs = self.self( 2025-08-26T20:38:05.2636681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2636766Z self.query(query_tensor) 2025-08-26T20:38:05.2636770Z 2025-08-26T20:38:05.2636882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2637101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2637171Z return mod(**inputs) 2025-08-26T20:38:05.2637468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2637549Z outputs = self.mobilebert( 2025-08-26T20:38:05.2637845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2637931Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2638229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2638314Z layer_outputs = layer_module( 2025-08-26T20:38:05.2638618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2638713Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2639026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2639102Z self_outputs = self.self( 2025-08-26T20:38:05.2639422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2639563Z self.key(key_tensor) 2025-08-26T20:38:05.2639573Z 2025-08-26T20:38:05.2639689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2639912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2639984Z return mod(**inputs) 2025-08-26T20:38:05.2640320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2640413Z outputs = self.mobilebert( 2025-08-26T20:38:05.2640732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2640818Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2641114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2641198Z layer_outputs = layer_module( 2025-08-26T20:38:05.2641494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2641594Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2641890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2641966Z self_outputs = self.self( 2025-08-26T20:38:05.2642273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2642369Z self.value(value_tensor) 2025-08-26T20:38:05.2642373Z 2025-08-26T20:38:05.2642469Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2642553Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2642664Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2642882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2642951Z return mod(**inputs) 2025-08-26T20:38:05.2643275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2643352Z outputs = self.mobilebert( 2025-08-26T20:38:05.2643648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2643732Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2644028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2644111Z layer_outputs = layer_module( 2025-08-26T20:38:05.2644409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2644506Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2644802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2644933Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2645239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2645332Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2645338Z 2025-08-26T20:38:05.2645453Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2645666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2645736Z return mod(**inputs) 2025-08-26T20:38:05.2646044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2646122Z outputs = self.mobilebert( 2025-08-26T20:38:05.2646428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2646507Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2646811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2646886Z layer_outputs = layer_module( 2025-08-26T20:38:05.2647302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2647516Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2647814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2647944Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2648237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2648334Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2648338Z 2025-08-26T20:38:05.2648447Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2648664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2648740Z return mod(**inputs) 2025-08-26T20:38:05.2649045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2649128Z outputs = self.mobilebert( 2025-08-26T20:38:05.2649410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2649484Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2649776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2649867Z layer_outputs = layer_module( 2025-08-26T20:38:05.2650155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2650242Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2650521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2650654Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2650933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2651068Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2651345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2651447Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2651450Z 2025-08-26T20:38:05.2651553Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2651751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2651825Z return mod(**inputs) 2025-08-26T20:38:05.2652106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2652186Z outputs = self.mobilebert( 2025-08-26T20:38:05.2652468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2652553Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2652846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2652922Z layer_outputs = layer_module( 2025-08-26T20:38:05.2653223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2653325Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2653648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2653770Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2654084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2654182Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2654186Z 2025-08-26T20:38:05.2654293Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2654511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2654582Z return mod(**inputs) 2025-08-26T20:38:05.2654892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2654967Z outputs = self.mobilebert( 2025-08-26T20:38:05.2655262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2655349Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2655664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2655746Z layer_outputs = layer_module( 2025-08-26T20:38:05.2656041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2656143Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2656449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2656587Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2656889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2657011Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2657016Z 2025-08-26T20:38:05.2657133Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2657345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2657416Z return mod(**inputs) 2025-08-26T20:38:05.2657720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2657797Z outputs = self.mobilebert( 2025-08-26T20:38:05.2658097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2658176Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2658470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2658554Z layer_outputs = layer_module( 2025-08-26T20:38:05.2658854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2658966Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2659262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2659396Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2659699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2659792Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2659796Z 2025-08-26T20:38:05.2659913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2660124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2660203Z return mod(**inputs) 2025-08-26T20:38:05.2660522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2660618Z outputs = self.mobilebert( 2025-08-26T20:38:05.2660926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2661003Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2661316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2661393Z layer_outputs = layer_module( 2025-08-26T20:38:05.2661701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2661809Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2662118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2662278Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2662581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2662718Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2663019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2663115Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2663138Z 2025-08-26T20:38:05.2663257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2663466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2663544Z return mod(**inputs) 2025-08-26T20:38:05.2663841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2663927Z outputs = self.mobilebert( 2025-08-26T20:38:05.2664233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2664307Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2664595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2664668Z layer_outputs = layer_module( 2025-08-26T20:38:05.2664983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2665083Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2665392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2665519Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2665834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2665932Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2665935Z 2025-08-26T20:38:05.2666044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2666262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2666333Z return mod(**inputs) 2025-08-26T20:38:05.2666635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2666721Z outputs = self.mobilebert( 2025-08-26T20:38:05.2667033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2667143Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2667469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2667547Z layer_outputs = layer_module( 2025-08-26T20:38:05.2667860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2667961Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2668273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2668396Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2668711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2668833Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2668855Z 2025-08-26T20:38:05.2668966Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2669189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2669260Z return mod(**inputs) 2025-08-26T20:38:05.2669566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2669644Z outputs = self.mobilebert( 2025-08-26T20:38:05.2669947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2670057Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2670362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2670444Z layer_outputs = layer_module( 2025-08-26T20:38:05.2670758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2670863Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2671186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2671321Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2671629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2671723Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2671727Z 2025-08-26T20:38:05.2671841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2672051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2672122Z return mod(**inputs) 2025-08-26T20:38:05.2672436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2672518Z outputs = self.mobilebert( 2025-08-26T20:38:05.2672830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2672910Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2673217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2673302Z layer_outputs = layer_module( 2025-08-26T20:38:05.2673608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2673717Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2674046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2674192Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2674514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2674650Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2674961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2675062Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2675067Z 2025-08-26T20:38:05.2675187Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2675403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2675482Z return mod(**inputs) 2025-08-26T20:38:05.2675793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2675891Z outputs = self.mobilebert( 2025-08-26T20:38:05.2676203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2676283Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2676593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2676671Z layer_outputs = layer_module( 2025-08-26T20:38:05.2677002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2677113Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2677419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2677551Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2677860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2677960Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2677964Z 2025-08-26T20:38:05.2678077Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2678296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2678379Z return mod(**inputs) 2025-08-26T20:38:05.2678688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2678775Z outputs = self.mobilebert( 2025-08-26T20:38:05.2679078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2679160Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2679545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2679634Z layer_outputs = layer_module( 2025-08-26T20:38:05.2679949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2680053Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2680367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2680491Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2680797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2680956Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2680962Z 2025-08-26T20:38:05.2681076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2681332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2681406Z return mod(**inputs) 2025-08-26T20:38:05.2681713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2681803Z outputs = self.mobilebert( 2025-08-26T20:38:05.2682113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2682207Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2682513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2682600Z layer_outputs = layer_module( 2025-08-26T20:38:05.2682908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2683034Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2683346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2683483Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2683797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2683935Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2683939Z 2025-08-26T20:38:05.2684051Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2684275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2684348Z return mod(**inputs) 2025-08-26T20:38:05.2684665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2684746Z outputs = self.mobilebert( 2025-08-26T20:38:05.2685057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2685136Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2685439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2685527Z layer_outputs = layer_module( 2025-08-26T20:38:05.2685840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2685949Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2686256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2686397Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2686713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2686847Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2687161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2687262Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2687267Z 2025-08-26T20:38:05.2687386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2687602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2687674Z return mod(**inputs) 2025-08-26T20:38:05.2688009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2688090Z outputs = self.mobilebert( 2025-08-26T20:38:05.2688415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2688495Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2688801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2688888Z layer_outputs = layer_module( 2025-08-26T20:38:05.2689195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2689333Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2689643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2689740Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2689772Z 2025-08-26T20:38:05.2689885Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2690100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2690182Z return mod(**inputs) 2025-08-26T20:38:05.2690485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2690569Z outputs = self.mobilebert( 2025-08-26T20:38:05.2690897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2690977Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2691287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2691366Z layer_outputs = layer_module( 2025-08-26T20:38:05.2691682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2691815Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2692127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2692249Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2692253Z 2025-08-26T20:38:05.2692364Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2692590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2692662Z return mod(**inputs) 2025-08-26T20:38:05.2692978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2693058Z outputs = self.mobilebert( 2025-08-26T20:38:05.2693368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2693456Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2693760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2693847Z layer_outputs = layer_module( 2025-08-26T20:38:05.2694152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2694337Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2694642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2694745Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2694765Z 2025-08-26T20:38:05.2694888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2695122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2695204Z return mod(**inputs) 2025-08-26T20:38:05.2695513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2695592Z outputs = self.mobilebert( 2025-08-26T20:38:05.2695907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2695988Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2696485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2696569Z layer_outputs = layer_module( 2025-08-26T20:38:05.2696888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2697113Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2697417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2697560Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2697866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2698002Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2698006Z 2025-08-26T20:38:05.2698118Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2698338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2698419Z return mod(**inputs) 2025-08-26T20:38:05.2698796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2698885Z outputs = self.mobilebert( 2025-08-26T20:38:05.2699185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2699272Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2699569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2699647Z layer_outputs = layer_module( 2025-08-26T20:38:05.2699952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2700118Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2700426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2700562Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2700863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2700961Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2700965Z 2025-08-26T20:38:05.2701073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2701290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2701363Z return mod(**inputs) 2025-08-26T20:38:05.2701671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2701747Z outputs = self.mobilebert( 2025-08-26T20:38:05.2702075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2702163Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2702485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2702572Z layer_outputs = layer_module( 2025-08-26T20:38:05.2702874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2703039Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2703344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2703477Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2703783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2703935Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2704238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2704335Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2704339Z 2025-08-26T20:38:05.2704446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2704667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2704756Z return mod(**inputs) 2025-08-26T20:38:05.2705068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2705144Z outputs = self.mobilebert( 2025-08-26T20:38:05.2705453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2705531Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2705836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2705920Z layer_outputs = layer_module( 2025-08-26T20:38:05.2706224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2706402Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2706706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2706824Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2707132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2707221Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2707225Z 2025-08-26T20:38:05.2707341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2707554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2707629Z return mod(**inputs) 2025-08-26T20:38:05.2707931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2708007Z outputs = self.mobilebert( 2025-08-26T20:38:05.2708312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2708388Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2708714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2708792Z layer_outputs = layer_module( 2025-08-26T20:38:05.2709105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2709288Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2709589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-26T20:38:05.2709712Z shared_attention_input = self.attention(hidden_states) 2025-08-26T20:38:05.2710017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-26T20:38:05.2710118Z layer_input = self.LayerNorm(layer_input) 2025-08-26T20:38:05.2710421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2710518Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2710540Z 2025-08-26T20:38:05.2710659Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2710872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2710950Z return mod(**inputs) 2025-08-26T20:38:05.2711249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2711327Z outputs = self.mobilebert( 2025-08-26T20:38:05.2711637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2711729Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2712017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2712091Z layer_outputs = layer_module( 2025-08-26T20:38:05.2712389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2712482Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2712779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2712864Z self_outputs = self.self( 2025-08-26T20:38:05.2713161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-26T20:38:05.2713244Z self.query(query_tensor) 2025-08-26T20:38:05.2713248Z 2025-08-26T20:38:05.2713357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2713569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2713647Z return mod(**inputs) 2025-08-26T20:38:05.2713946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2714034Z outputs = self.mobilebert( 2025-08-26T20:38:05.2714327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2714405Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2714707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2714783Z layer_outputs = layer_module( 2025-08-26T20:38:05.2715088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2715180Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2715499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2715578Z self_outputs = self.self( 2025-08-26T20:38:05.2715889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-26T20:38:05.2715971Z self.key(key_tensor) 2025-08-26T20:38:05.2715975Z 2025-08-26T20:38:05.2716082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2716299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2716369Z return mod(**inputs) 2025-08-26T20:38:05.2716671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2716755Z outputs = self.mobilebert( 2025-08-26T20:38:05.2717054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2717142Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2717467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2717552Z layer_outputs = layer_module( 2025-08-26T20:38:05.2717849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2717939Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2718242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-26T20:38:05.2718335Z self_outputs = self.self( 2025-08-26T20:38:05.2718640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-26T20:38:05.2718716Z self.value(value_tensor) 2025-08-26T20:38:05.2718719Z 2025-08-26T20:38:05.2718808Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2718904Z cudagraph partition due to non gpu ops 2025-08-26T20:38:05.2719017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2719244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2719317Z return mod(**inputs) 2025-08-26T20:38:05.2719687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2719778Z outputs = self.mobilebert( 2025-08-26T20:38:05.2720084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2720175Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2720477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2720565Z layer_outputs = layer_module( 2025-08-26T20:38:05.2720875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2720972Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2721258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2721380Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2721666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-26T20:38:05.2721756Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2721760Z 2025-08-26T20:38:05.2721863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2722070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2722159Z return mod(**inputs) 2025-08-26T20:38:05.2722471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2722547Z outputs = self.mobilebert( 2025-08-26T20:38:05.2722832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2722909Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2723214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2723300Z layer_outputs = layer_module( 2025-08-26T20:38:05.2723595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-26T20:38:05.2723772Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-26T20:38:05.2724081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-26T20:38:05.2724220Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-26T20:38:05.2724532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-26T20:38:05.2724621Z layer_input = self.dense(hidden_states) 2025-08-26T20:38:05.2724626Z 2025-08-26T20:38:05.2724740Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2724957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2725049Z return mod(**inputs) 2025-08-26T20:38:05.2725332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2725404Z outputs = self.mobilebert( 2025-08-26T20:38:05.2725697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2725772Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2726059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2726129Z layer_outputs = layer_module( 2025-08-26T20:38:05.2726411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-26T20:38:05.2726507Z self_attention_outputs = self.attention( 2025-08-26T20:38:05.2726813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-26T20:38:05.2726950Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-26T20:38:05.2727259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-26T20:38:05.2727401Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2727699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2727799Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2727803Z 2025-08-26T20:38:05.2727918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2728128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2728207Z return mod(**inputs) 2025-08-26T20:38:05.2728506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2728584Z outputs = self.mobilebert( 2025-08-26T20:38:05.2728917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2728998Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2729323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2729401Z layer_outputs = layer_module( 2025-08-26T20:38:05.2729720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2729821Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2730131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2730260Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2730557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2730656Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2730679Z 2025-08-26T20:38:05.2730790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2731003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2731083Z return mod(**inputs) 2025-08-26T20:38:05.2731384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2731471Z outputs = self.mobilebert( 2025-08-26T20:38:05.2731781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2731884Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2732187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2732263Z layer_outputs = layer_module( 2025-08-26T20:38:05.2732568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2732669Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2732980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2733097Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2733404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2733534Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2733538Z 2025-08-26T20:38:05.2733644Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2733862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2733935Z return mod(**inputs) 2025-08-26T20:38:05.2734242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2734318Z outputs = self.mobilebert( 2025-08-26T20:38:05.2734620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2734706Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2735010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2735096Z layer_outputs = layer_module( 2025-08-26T20:38:05.2735391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2735493Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2735817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2735969Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2736280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2736371Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2736375Z 2025-08-26T20:38:05.2736492Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2736707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2736780Z return mod(**inputs) 2025-08-26T20:38:05.2737093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2737169Z outputs = self.mobilebert( 2025-08-26T20:38:05.2737475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2737571Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2737875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2737958Z layer_outputs = layer_module( 2025-08-26T20:38:05.2738264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2738373Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2738690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2738831Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2739129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2739259Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2739561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2739658Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2739661Z 2025-08-26T20:38:05.2739776Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2739983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2740055Z return mod(**inputs) 2025-08-26T20:38:05.2740360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2740437Z outputs = self.mobilebert( 2025-08-26T20:38:05.2740741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2740818Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2741137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2741208Z layer_outputs = layer_module( 2025-08-26T20:38:05.2741484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2741587Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2741873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2741997Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2742290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2742397Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2742409Z 2025-08-26T20:38:05.2742542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2742755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2742835Z return mod(**inputs) 2025-08-26T20:38:05.2743135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2743218Z outputs = self.mobilebert( 2025-08-26T20:38:05.2743516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2743595Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2743896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2743973Z layer_outputs = layer_module( 2025-08-26T20:38:05.2744305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2744399Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2744683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2744802Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2745092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2745240Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2745244Z 2025-08-26T20:38:05.2745353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2745571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2745644Z return mod(**inputs) 2025-08-26T20:38:05.2745944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2746030Z outputs = self.mobilebert( 2025-08-26T20:38:05.2746325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2746410Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2746702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2746780Z layer_outputs = layer_module( 2025-08-26T20:38:05.2747084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2747183Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2747486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2747620Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2747925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2748014Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2748018Z 2025-08-26T20:38:05.2748126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2748345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2748417Z return mod(**inputs) 2025-08-26T20:38:05.2748721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2748796Z outputs = self.mobilebert( 2025-08-26T20:38:05.2749107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2749212Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2749509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2749599Z layer_outputs = layer_module( 2025-08-26T20:38:05.2749893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2749998Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2750295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2750430Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2750735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2750877Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2751164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2751258Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2751261Z 2025-08-26T20:38:05.2751371Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2751573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2751658Z return mod(**inputs) 2025-08-26T20:38:05.2751960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2752036Z outputs = self.mobilebert( 2025-08-26T20:38:05.2752340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2752418Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2752717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2752800Z layer_outputs = layer_module( 2025-08-26T20:38:05.2753097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2753204Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2753504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2753625Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2753928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2754018Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2754023Z 2025-08-26T20:38:05.2754140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2754353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2754431Z return mod(**inputs) 2025-08-26T20:38:05.2754729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2754805Z outputs = self.mobilebert( 2025-08-26T20:38:05.2755107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2755185Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2755491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2755587Z layer_outputs = layer_module( 2025-08-26T20:38:05.2755911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2756025Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2756328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-26T20:38:05.2756456Z intermediate_output = self.intermediate(hidden_states) 2025-08-26T20:38:05.2756774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2756899Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2756902Z 2025-08-26T20:38:05.2757012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2757223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2757303Z return mod(**inputs) 2025-08-26T20:38:05.2757622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2757704Z outputs = self.mobilebert( 2025-08-26T20:38:05.2758004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2758081Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2758390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2758486Z layer_outputs = layer_module( 2025-08-26T20:38:05.2758795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2758898Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2759216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2759355Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2759740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-26T20:38:05.2759847Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2759852Z 2025-08-26T20:38:05.2759965Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2760190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2760266Z return mod(**inputs) 2025-08-26T20:38:05.2760578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2760667Z outputs = self.mobilebert( 2025-08-26T20:38:05.2760974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2761065Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2761375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2761461Z layer_outputs = layer_module( 2025-08-26T20:38:05.2761770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-26T20:38:05.2761874Z attention_output = ffn_module(attention_output) 2025-08-26T20:38:05.2762192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-26T20:38:05.2762328Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-26T20:38:05.2762680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-26T20:38:05.2762819Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2763150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2763253Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2763258Z 2025-08-26T20:38:05.2763369Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2763598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2763674Z return mod(**inputs) 2025-08-26T20:38:05.2763988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2764067Z outputs = self.mobilebert( 2025-08-26T20:38:05.2764374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2764484Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2764791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2764876Z layer_outputs = layer_module( 2025-08-26T20:38:05.2765184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2765317Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2765630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-26T20:38:05.2765766Z hidden_states = self.dense(hidden_states) 2025-08-26T20:38:05.2765769Z 2025-08-26T20:38:05.2765892Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2766110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2766190Z return mod(**inputs) 2025-08-26T20:38:05.2766501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2766581Z outputs = self.mobilebert( 2025-08-26T20:38:05.2766891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2766969Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2767289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2767369Z layer_outputs = layer_module( 2025-08-26T20:38:05.2767682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-26T20:38:05.2767823Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:38:05.2768137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-26T20:38:05.2768267Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:38:05.2768270Z 2025-08-26T20:38:05.2768378Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2768605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2768675Z return mod(**inputs) 2025-08-26T20:38:05.2768973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2769060Z outputs = self.mobilebert( 2025-08-26T20:38:05.2769367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2769452Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2769785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2769879Z layer_outputs = layer_module( 2025-08-26T20:38:05.2770195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2770366Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2770675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-26T20:38:05.2770777Z layer_output = self.dense(intermediate_states) 2025-08-26T20:38:05.2770781Z 2025-08-26T20:38:05.2770896Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2771115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2771187Z return mod(**inputs) 2025-08-26T20:38:05.2771515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2771593Z outputs = self.mobilebert( 2025-08-26T20:38:05.2771906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2771983Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2772292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2772397Z layer_outputs = layer_module( 2025-08-26T20:38:05.2772704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2772879Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2773188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-26T20:38:05.2773329Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-26T20:38:05.2773633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2773731Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2773735Z 2025-08-26T20:38:05.2773852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2774073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2774150Z return mod(**inputs) 2025-08-26T20:38:05.2774451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2774534Z outputs = self.mobilebert( 2025-08-26T20:38:05.2774842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2774920Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2775229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2775304Z layer_outputs = layer_module( 2025-08-26T20:38:05.2775615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2775781Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2776089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2776227Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2776548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-26T20:38:05.2776666Z layer_outputs = self.dense(hidden_states) 2025-08-26T20:38:05.2776670Z 2025-08-26T20:38:05.2776780Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2777003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2777075Z return mod(**inputs) 2025-08-26T20:38:05.2777383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-26T20:38:05.2777472Z outputs = self.mobilebert( 2025-08-26T20:38:05.2777772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-26T20:38:05.2777861Z encoder_outputs = self.encoder( 2025-08-26T20:38:05.2778162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-26T20:38:05.2778259Z layer_outputs = layer_module( 2025-08-26T20:38:05.2778566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-26T20:38:05.2778733Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-26T20:38:05.2779037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-26T20:38:05.2779168Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-26T20:38:05.2779488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-26T20:38:05.2779619Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-26T20:38:05.2779925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-26T20:38:05.2780035Z return input_tensor * self.weight + self.bias 2025-08-26T20:38:05.2780040Z 2025-08-26T20:38:05.2780152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2780378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2780451Z return mod(**inputs) 2025-08-26T20:38:05.2780767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1256, in forward 2025-08-26T20:38:05.2780870Z logits = self.qa_outputs(sequence_output) 2025-08-26T20:38:05.2780874Z 2025-08-26T20:38:05.2780981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2781199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2781269Z return mod(**inputs) 2025-08-26T20:38:05.2781584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1274, in forward 2025-08-26T20:38:05.2781702Z start_loss = loss_fct(start_logits, start_positions) 2025-08-26T20:38:05.2781706Z 2025-08-26T20:38:05.2781819Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:05.2782052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:05.2782122Z return mod(**inputs) 2025-08-26T20:38:05.2782431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1275, in forward 2025-08-26T20:38:05.2782531Z end_loss = loss_fct(end_logits, end_positions) 2025-08-26T20:38:05.2782536Z 2025-08-26T20:38:17.8949306Z Compilation time (from dynamo_timed): 39.83583792 2025-08-26T20:38:17.8949629Z pass 2025-08-26T20:38:17.8949945Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:17.8951126Z TIMING: _recursive_pre_grad_passes:0.02412 _recursive_joint_graph_passes:1.42352 _recursive_post_grad_passes:0.23408 async_compile.wait:0.28109 code_gen:9.39824 inductor_compile:14.38711 backend_compile:27.66496 gc:0.00129 entire_frame_compile:39.83584 total_wall_time:39.83584 2025-08-26T20:38:17.8952244Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:56755 | FakeTensor.__torch_dispatch__:15375 | ProxyTorchDispatchMode.__torch_dispatch__:21655 2025-08-26T20:38:17.8952820Z Dynamo produced 1 graphs covering 1453 ops with 0 graph breaks (0 unique) 2025-08-26T20:38:24.3398188Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:38:24.3399203Z from pkg_resources import resource_filename 2025-08-26T20:38:24.9509559Z 2025-08-26T20:38:26.9293329Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:38:26.9293620Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:38:26.9305812Z cpu eval OPTForCausalLM 2025-08-26T20:38:28.7342055Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:29.6141009Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:30.4800843Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:38.3511014Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3516577Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3516917Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3517395Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3517681Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3517968Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3518233Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3518474Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3518707Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3518937Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3519171Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3519404Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3520017Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3520470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3520903Z return mod(**inputs) 2025-08-26T20:38:38.3521329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3521747Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3522174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3522599Z outputs = self.model.decoder( 2025-08-26T20:38:38.3522989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3523379Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3523789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3524208Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3524595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3524998Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3525413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3525850Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3526629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3527108Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3527358Z 2025-08-26T20:38:38.3527483Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3527888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3528247Z return mod(**inputs) 2025-08-26T20:38:38.3528602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3529015Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3529449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3529877Z outputs = self.model.decoder( 2025-08-26T20:38:38.3530274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3530654Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3531127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3531543Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3531924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3532311Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3532724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3533219Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3533657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3534085Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3534232Z 2025-08-26T20:38:38.3534348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3534756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3535117Z return mod(**inputs) 2025-08-26T20:38:38.3535476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3535852Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3536260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3536665Z outputs = self.model.decoder( 2025-08-26T20:38:38.3537046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3537424Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3537801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3538188Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3538563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3538953Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3539359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3539827Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3540259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3540678Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3540828Z 2025-08-26T20:38:38.3540924Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3541148Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3541374Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3541672Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3541931Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3542351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3542687Z return mod(**inputs) 2025-08-26T20:38:38.3543041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3543412Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3543792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3544174Z outputs = self.model.decoder( 2025-08-26T20:38:38.3544527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3544888Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3545268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3545716Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3546088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3546504Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3546897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3547312Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3547717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3548156Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3548613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3549111Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3549303Z 2025-08-26T20:38:38.3549417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3549799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3550152Z return mod(**inputs) 2025-08-26T20:38:38.3550507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3550889Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3551290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3551685Z outputs = self.model.decoder( 2025-08-26T20:38:38.3552038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3552398Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3552783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3553180Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3553556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3553947Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3554349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3554780Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3555211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3555653Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3556139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3556666Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3556851Z 2025-08-26T20:38:38.3556989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3557383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3557758Z return mod(**inputs) 2025-08-26T20:38:38.3558129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3558522Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3558931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3559348Z outputs = self.model.decoder( 2025-08-26T20:38:38.3559868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3560595Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3561006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3561435Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3561814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3562206Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3562613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3563043Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3563486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3563904Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3564058Z 2025-08-26T20:38:38.3564170Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3564567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3564925Z return mod(**inputs) 2025-08-26T20:38:38.3565316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3565701Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3566142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3566568Z outputs = self.model.decoder( 2025-08-26T20:38:38.3566956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3567337Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3567735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3568138Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3568520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3568884Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3569264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3569651Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3569792Z 2025-08-26T20:38:38.3569904Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3570261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3570594Z return mod(**inputs) 2025-08-26T20:38:38.3570927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3571284Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3571690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3572067Z outputs = self.model.decoder( 2025-08-26T20:38:38.3572432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3572789Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3573168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3573548Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3573893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3574273Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3574674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3575090Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3575245Z 2025-08-26T20:38:38.3575352Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3575741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3576069Z return mod(**inputs) 2025-08-26T20:38:38.3576399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3576762Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3577135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3577538Z outputs = self.model.decoder( 2025-08-26T20:38:38.3577887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3578245Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3578615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3579000Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3579380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3579769Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3580176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3580582Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3580736Z 2025-08-26T20:38:38.3580848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3581241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3581600Z return mod(**inputs) 2025-08-26T20:38:38.3581946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3582330Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3582740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3583140Z outputs = self.model.decoder( 2025-08-26T20:38:38.3583510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3583884Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3584331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3584733Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3585113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3585510Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3585896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3586384Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3586841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3587293Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3587475Z 2025-08-26T20:38:38.3587595Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3587982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3588334Z return mod(**inputs) 2025-08-26T20:38:38.3588686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3589066Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3589461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3589868Z outputs = self.model.decoder( 2025-08-26T20:38:38.3590244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3590649Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3591048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3591444Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3591824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3592216Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3592653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3593077Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3593504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3593918Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3594068Z 2025-08-26T20:38:38.3594189Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3594577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3594922Z return mod(**inputs) 2025-08-26T20:38:38.3595269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3595646Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3596046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3596755Z outputs = self.model.decoder( 2025-08-26T20:38:38.3597127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3597506Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3597915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3598320Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3598692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3599085Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3599583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3600094Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3600547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3600970Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3601146Z 2025-08-26T20:38:38.3601235Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3601548Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3601778Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3601995Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3602286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3602660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3603004Z return mod(**inputs) 2025-08-26T20:38:38.3603356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3603741Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3604148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3604556Z outputs = self.model.decoder( 2025-08-26T20:38:38.3604930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3605305Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3605739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3606124Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3606479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3606849Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3607227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3607670Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3608074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3608480Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3608930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3609420Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3609615Z 2025-08-26T20:38:38.3609721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3610088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3610418Z return mod(**inputs) 2025-08-26T20:38:38.3610745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3611107Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3611490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3611871Z outputs = self.model.decoder( 2025-08-26T20:38:38.3612222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3612575Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3612994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3613373Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3613730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3614093Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3614480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3614888Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3615292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3615700Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3616162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3616653Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3616830Z 2025-08-26T20:38:38.3616939Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3617304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3617638Z return mod(**inputs) 2025-08-26T20:38:38.3617963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3618328Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3618709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3619091Z outputs = self.model.decoder( 2025-08-26T20:38:38.3619437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3619819Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3620203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3620586Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3620944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3621301Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3621683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3622107Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3622512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3622901Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3623039Z 2025-08-26T20:38:38.3623146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3623518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3623848Z return mod(**inputs) 2025-08-26T20:38:38.3624178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3624543Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3624945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3625341Z outputs = self.model.decoder( 2025-08-26T20:38:38.3625687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3626041Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3626413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3626793Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3627147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3627509Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3627896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3628308Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3628460Z 2025-08-26T20:38:38.3628572Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3628964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3629309Z return mod(**inputs) 2025-08-26T20:38:38.3629651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3630028Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3630416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3630826Z outputs = self.model.decoder( 2025-08-26T20:38:38.3631171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3631527Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3631902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3632285Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3632642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3633002Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3633382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3633788Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3633964Z 2025-08-26T20:38:38.3634078Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3634447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3634772Z return mod(**inputs) 2025-08-26T20:38:38.3635104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3635483Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3635883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3636902Z outputs = self.model.decoder( 2025-08-26T20:38:38.3637274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3637650Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3638051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3638455Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3638823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3639228Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3639756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3640197Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3640355Z 2025-08-26T20:38:38.3640479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3640882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3641240Z return mod(**inputs) 2025-08-26T20:38:38.3641597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3641983Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3642381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3642795Z outputs = self.model.decoder( 2025-08-26T20:38:38.3643169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3643553Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3643956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3644353Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3644733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3645127Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3645567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3646016Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3646447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3646892Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3647076Z 2025-08-26T20:38:38.3647190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3647555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3647888Z return mod(**inputs) 2025-08-26T20:38:38.3648217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3648590Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3648998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3649444Z outputs = self.model.decoder( 2025-08-26T20:38:38.3649801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3650157Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3650531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3650904Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3651270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3651685Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3652072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3652482Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3652889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3653273Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3653429Z 2025-08-26T20:38:38.3653533Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3653890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3654215Z return mod(**inputs) 2025-08-26T20:38:38.3654539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3654899Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3655274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3655656Z outputs = self.model.decoder( 2025-08-26T20:38:38.3656019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3656364Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3656737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3657108Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3657456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3657815Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3658180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3658585Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3658989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3659381Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3659524Z 2025-08-26T20:38:38.3659625Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3659856Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3660101Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3660327Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3660576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3660957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3661303Z return mod(**inputs) 2025-08-26T20:38:38.3661656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3662028Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3662407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3662779Z outputs = self.model.decoder( 2025-08-26T20:38:38.3663130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3663509Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3663892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3664265Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3664621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3664984Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3665370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3665790Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3666201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3666612Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3667070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3667559Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3667749Z 2025-08-26T20:38:38.3667856Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3668227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3668564Z return mod(**inputs) 2025-08-26T20:38:38.3668901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3669267Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3669641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3670025Z outputs = self.model.decoder( 2025-08-26T20:38:38.3670381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3670742Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3671111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3671492Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3671846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3672218Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3672602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3673008Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3673414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3673838Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3674313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3674782Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3674945Z 2025-08-26T20:38:38.3675052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3675436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3675783Z return mod(**inputs) 2025-08-26T20:38:38.3676138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3676510Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3676908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3677309Z outputs = self.model.decoder( 2025-08-26T20:38:38.3677704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3678082Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3678475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3678878Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3679259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3679776Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3680201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3680657Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3681109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3681506Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3681652Z 2025-08-26T20:38:38.3681768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3682145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3682511Z return mod(**inputs) 2025-08-26T20:38:38.3682881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3683290Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3683712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3684144Z outputs = self.model.decoder( 2025-08-26T20:38:38.3684541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3684947Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3685371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3685789Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3686185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3686591Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3687019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3687456Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3687612Z 2025-08-26T20:38:38.3687728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3688133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3688474Z return mod(**inputs) 2025-08-26T20:38:38.3688830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3689188Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3689578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3689962Z outputs = self.model.decoder( 2025-08-26T20:38:38.3690306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3690656Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3691017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3691390Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3691736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3692091Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3692466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3692876Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3693035Z 2025-08-26T20:38:38.3693136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3693494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3693830Z return mod(**inputs) 2025-08-26T20:38:38.3694162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3694529Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3694899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3695271Z outputs = self.model.decoder( 2025-08-26T20:38:38.3695612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3695954Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3696526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3696917Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3697274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3697638Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3698013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3698403Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3698542Z 2025-08-26T20:38:38.3698655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3699026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3699356Z return mod(**inputs) 2025-08-26T20:38:38.3699694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3700048Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3700427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3700813Z outputs = self.model.decoder( 2025-08-26T20:38:38.3701156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3701525Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3701891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3702259Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3702655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3703016Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3703427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-26T20:38:38.3703875Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-26T20:38:38.3704067Z 2025-08-26T20:38:38.3704184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3704543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3704875Z return mod(**inputs) 2025-08-26T20:38:38.3705209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3705577Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3705947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3706314Z outputs = self.model.decoder( 2025-08-26T20:38:38.3706727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3707074Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3707441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3707802Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3708149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3708547Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3708950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3709352Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3709744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3710161Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3710333Z 2025-08-26T20:38:38.3710439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3710809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3711140Z return mod(**inputs) 2025-08-26T20:38:38.3711465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3711825Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3712204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3712584Z outputs = self.model.decoder( 2025-08-26T20:38:38.3712925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3713302Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3713702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3714105Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3714474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3714857Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3715261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3715687Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3716109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3716511Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3716663Z 2025-08-26T20:38:38.3716800Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3717169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3717535Z return mod(**inputs) 2025-08-26T20:38:38.3717891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3718265Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3718667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3719071Z outputs = self.model.decoder( 2025-08-26T20:38:38.3719504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3719898Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3720294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3720714Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3721142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3721537Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3721940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3722371Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3722799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3723223Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3723368Z 2025-08-26T20:38:38.3723461Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3723672Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3723889Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3724100Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3724339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3724704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3725034Z return mod(**inputs) 2025-08-26T20:38:38.3725367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3725729Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3726100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3726485Z outputs = self.model.decoder( 2025-08-26T20:38:38.3726835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3727194Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3727569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3727944Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3728304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3728670Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3729051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3729459Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3729853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3730262Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3730712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3731210Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3770809Z 2025-08-26T20:38:38.3771094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3771586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3771964Z return mod(**inputs) 2025-08-26T20:38:38.3772346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3772740Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3773149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3773563Z outputs = self.model.decoder( 2025-08-26T20:38:38.3773936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3774303Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3774705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3775137Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3775495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3775865Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3776256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3776672Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3777102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3777574Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3778050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3778555Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3778743Z 2025-08-26T20:38:38.3778860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3779259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3779613Z return mod(**inputs) 2025-08-26T20:38:38.3779945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3780309Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3780704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3781116Z outputs = self.model.decoder( 2025-08-26T20:38:38.3781482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3781862Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3782269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3782669Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3783048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3783435Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3783842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3784270Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3784697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3785110Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3785259Z 2025-08-26T20:38:38.3785384Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3785798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3786146Z return mod(**inputs) 2025-08-26T20:38:38.3786518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3786896Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3787299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3787705Z outputs = self.model.decoder( 2025-08-26T20:38:38.3788072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3788454Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3788855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3789264Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3789635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3790044Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3790453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3790865Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3791016Z 2025-08-26T20:38:38.3791137Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3791517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3791887Z return mod(**inputs) 2025-08-26T20:38:38.3792240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3792618Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3793012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3793424Z outputs = self.model.decoder( 2025-08-26T20:38:38.3793796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3794195Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3794611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3795028Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3795416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3795814Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3796408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3796864Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3797036Z 2025-08-26T20:38:38.3797156Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3797574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3797939Z return mod(**inputs) 2025-08-26T20:38:38.3798303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3798696Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3799108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3799591Z outputs = self.model.decoder( 2025-08-26T20:38:38.3799984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3800382Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3800788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3801287Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3801729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3802136Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3802548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3802976Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3803136Z 2025-08-26T20:38:38.3803251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3803653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3804017Z return mod(**inputs) 2025-08-26T20:38:38.3804372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3804766Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3805178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3805638Z outputs = self.model.decoder( 2025-08-26T20:38:38.3806028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3806411Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3806823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3807253Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3807636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3808067Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3808474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3808906Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3809342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3809781Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3809961Z 2025-08-26T20:38:38.3810074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3810461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3810808Z return mod(**inputs) 2025-08-26T20:38:38.3811159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3811528Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3811923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3812330Z outputs = self.model.decoder( 2025-08-26T20:38:38.3812702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3813092Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3813498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3813896Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3814266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3814651Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3815054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3815473Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3815896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3816337Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3816492Z 2025-08-26T20:38:38.3816611Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3817021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3817384Z return mod(**inputs) 2025-08-26T20:38:38.3817734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3818113Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3818510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3818905Z outputs = self.model.decoder( 2025-08-26T20:38:38.3819273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3819653Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3820053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3820501Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3820872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3821266Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3821678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3822109Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3822547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3822962Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3823120Z 2025-08-26T20:38:38.3823210Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3823446Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3823680Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3823906Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3824166Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3824562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3824931Z return mod(**inputs) 2025-08-26T20:38:38.3825275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3825651Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3826050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3826463Z outputs = self.model.decoder( 2025-08-26T20:38:38.3826822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3827197Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3827594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3827996Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3828370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3828747Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3829149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3829572Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3829998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3830421Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3830943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3831478Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3831683Z 2025-08-26T20:38:38.3831813Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3832208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3832567Z return mod(**inputs) 2025-08-26T20:38:38.3832895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3833257Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3833638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3834025Z outputs = self.model.decoder( 2025-08-26T20:38:38.3834392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3834770Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3835196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3835600Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3835973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3836354Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3836761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3837211Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3837640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3838058Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3838536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3839039Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3839219Z 2025-08-26T20:38:38.3839343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3839839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3840202Z return mod(**inputs) 2025-08-26T20:38:38.3840565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3840964Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3841380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3841805Z outputs = self.model.decoder( 2025-08-26T20:38:38.3842170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3842568Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3842977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3843374Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3843741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3844134Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3844542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3844973Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3845397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3845800Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3845957Z 2025-08-26T20:38:38.3846096Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3846505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3846861Z return mod(**inputs) 2025-08-26T20:38:38.3847209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3847591Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3847989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3848396Z outputs = self.model.decoder( 2025-08-26T20:38:38.3848766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3849140Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3849540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3849937Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3850312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3850677Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3851050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3851437Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3851577Z 2025-08-26T20:38:38.3851688Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3852079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3852408Z return mod(**inputs) 2025-08-26T20:38:38.3852759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3853136Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3853534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3853944Z outputs = self.model.decoder( 2025-08-26T20:38:38.3854354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3854712Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3855105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3855507Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3855877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3856264Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3856670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3857101Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3857268Z 2025-08-26T20:38:38.3857388Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3857774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3858125Z return mod(**inputs) 2025-08-26T20:38:38.3858476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3858858Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3859294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3859694Z outputs = self.model.decoder( 2025-08-26T20:38:38.3860066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3860444Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3860882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3861280Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3861673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3862063Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3862470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3862883Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3863033Z 2025-08-26T20:38:38.3863146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3863534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3863881Z return mod(**inputs) 2025-08-26T20:38:38.3864218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3864592Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3864988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3865400Z outputs = self.model.decoder( 2025-08-26T20:38:38.3865780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3866174Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3866579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3867001Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3867372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3867757Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3868159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-26T20:38:38.3868618Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-26T20:38:38.3868829Z 2025-08-26T20:38:38.3868941Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3869332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3869678Z return mod(**inputs) 2025-08-26T20:38:38.3870019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3870403Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3870803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3871207Z outputs = self.model.decoder( 2025-08-26T20:38:38.3871579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3871956Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3872363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3872767Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3873147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3873526Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3873930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3874362Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3874795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3875242Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3875421Z 2025-08-26T20:38:38.3875556Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3875973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3876336Z return mod(**inputs) 2025-08-26T20:38:38.3876698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3877099Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3877512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3877939Z outputs = self.model.decoder( 2025-08-26T20:38:38.3878328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3878726Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3879139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3879658Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3880053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3880462Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3880875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3881307Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3881753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3882210Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3882357Z 2025-08-26T20:38:38.3882475Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3882866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3883216Z return mod(**inputs) 2025-08-26T20:38:38.3883568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3883947Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3884345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3884777Z outputs = self.model.decoder( 2025-08-26T20:38:38.3885145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3885526Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3885933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3886311Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3886654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3887019Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3887401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3887800Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3888199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3888590Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3888738Z 2025-08-26T20:38:38.3888821Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3889043Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3889257Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3889461Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3889699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3890096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3890436Z return mod(**inputs) 2025-08-26T20:38:38.3890796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3891183Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3891567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3891953Z outputs = self.model.decoder( 2025-08-26T20:38:38.3892306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3892660Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3893039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3893421Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3893780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3894162Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3894544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3894965Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3895389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3895816Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3896486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3897069Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3897269Z 2025-08-26T20:38:38.3897374Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3897768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3898122Z return mod(**inputs) 2025-08-26T20:38:38.3898470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3898862Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3899268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3899676Z outputs = self.model.decoder( 2025-08-26T20:38:38.3900043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3900425Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3900828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3901208Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3901564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3901927Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3902312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3902735Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3903160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3903587Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3904057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3904552Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3904732Z 2025-08-26T20:38:38.3904846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3905269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3905619Z return mod(**inputs) 2025-08-26T20:38:38.3905997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3906382Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3906784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3907186Z outputs = self.model.decoder( 2025-08-26T20:38:38.3907548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3907930Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3908329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3908729Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3909105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3909521Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3909926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3910353Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3910756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3911139Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3911320Z 2025-08-26T20:38:38.3911426Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3911795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3912146Z return mod(**inputs) 2025-08-26T20:38:38.3912498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3912872Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3913282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3913684Z outputs = self.model.decoder( 2025-08-26T20:38:38.3914053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3914427Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3914827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3915236Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3915621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3916022Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3916443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3916869Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3917031Z 2025-08-26T20:38:38.3917146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3917545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3917917Z return mod(**inputs) 2025-08-26T20:38:38.3918269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3918662Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3919079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3919561Z outputs = self.model.decoder( 2025-08-26T20:38:38.3919974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3920367Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3920799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3921207Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3921562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3921923Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3922306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3922717Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3922878Z 2025-08-26T20:38:38.3922991Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3923363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3923689Z return mod(**inputs) 2025-08-26T20:38:38.3924043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3924398Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3924774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3925148Z outputs = self.model.decoder( 2025-08-26T20:38:38.3925496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3925877Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3926254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3926634Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3926982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3927349Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3927733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3928119Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3928258Z 2025-08-26T20:38:38.3928361Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3928732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3929062Z return mod(**inputs) 2025-08-26T20:38:38.3929393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3929749Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3930119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3930499Z outputs = self.model.decoder( 2025-08-26T20:38:38.3930848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3931212Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3931579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3931954Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3932311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3932679Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3933061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3933478Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3933955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3934384Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3934557Z 2025-08-26T20:38:38.3934697Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3935067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3935393Z return mod(**inputs) 2025-08-26T20:38:38.3935729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3936090Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3936470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3936852Z outputs = self.model.decoder( 2025-08-26T20:38:38.3937203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3937561Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3937938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3938346Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3938693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3939079Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3939493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3939899Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3940325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3940704Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3940850Z 2025-08-26T20:38:38.3940956Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3941332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3941667Z return mod(**inputs) 2025-08-26T20:38:38.3942001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3942380Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3942784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3943189Z outputs = self.model.decoder( 2025-08-26T20:38:38.3943559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3943933Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3944336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3944738Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3945116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3945480Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3945874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3946270Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3946669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3947080Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3947231Z 2025-08-26T20:38:38.3947321Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3947554Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3947644Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3947723Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3947855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3948084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3948172Z return mod(**inputs) 2025-08-26T20:38:38.3948417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3948497Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3948756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3948846Z outputs = self.model.decoder( 2025-08-26T20:38:38.3949083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3949171Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3949428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3949508Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3949776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3949864Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3950132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3950239Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3950506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3950635Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3950947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3951104Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3951108Z 2025-08-26T20:38:38.3951222Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3951447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3951518Z return mod(**inputs) 2025-08-26T20:38:38.3951753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3951842Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3952096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3952184Z outputs = self.model.decoder( 2025-08-26T20:38:38.3952416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3952495Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3952780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3952858Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3953101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3953186Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3953447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3953550Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3953809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3953923Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3954235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3954363Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3954390Z 2025-08-26T20:38:38.3954504Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3954734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3954812Z return mod(**inputs) 2025-08-26T20:38:38.3955046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3955132Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3955390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3955479Z outputs = self.model.decoder( 2025-08-26T20:38:38.3955710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3955790Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3956057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3956167Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3956421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3956510Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3956782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3956898Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3957171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3957286Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3957290Z 2025-08-26T20:38:38.3957404Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3957643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3957718Z return mod(**inputs) 2025-08-26T20:38:38.3957965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3958054Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3958377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3958464Z outputs = self.model.decoder( 2025-08-26T20:38:38.3958695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3958776Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3959042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3959120Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3959364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3959521Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3959794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3959891Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3959896Z 2025-08-26T20:38:38.3960007Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3960243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3960316Z return mod(**inputs) 2025-08-26T20:38:38.3960561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3960652Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3960925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3961044Z outputs = self.model.decoder( 2025-08-26T20:38:38.3961287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3961406Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3961651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3961726Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3961966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3962047Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3962292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3962389Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3962393Z 2025-08-26T20:38:38.3962494Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3962700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3962783Z return mod(**inputs) 2025-08-26T20:38:38.3963006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3963079Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3963324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3963399Z outputs = self.model.decoder( 2025-08-26T20:38:38.3963647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3963734Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3963989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3964071Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3964306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3964393Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3964660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3964744Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3964748Z 2025-08-26T20:38:38.3964866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3965079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3965150Z return mod(**inputs) 2025-08-26T20:38:38.3965391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3965470Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3965737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3965817Z outputs = self.model.decoder( 2025-08-26T20:38:38.3966054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3966134Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3966388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3966474Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3966709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3966801Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3967057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-26T20:38:38.3967223Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-26T20:38:38.3967227Z 2025-08-26T20:38:38.3967350Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3967580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3967659Z return mod(**inputs) 2025-08-26T20:38:38.3967897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3967976Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3968241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3968320Z outputs = self.model.decoder( 2025-08-26T20:38:38.3968555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3968633Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3968894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3968995Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3969231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3969323Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3969577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3969689Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3969945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3970088Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3970093Z 2025-08-26T20:38:38.3970208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3970424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3970503Z return mod(**inputs) 2025-08-26T20:38:38.3970737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3970816Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3971079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3971158Z outputs = self.model.decoder( 2025-08-26T20:38:38.3971393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3971473Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3971737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3971814Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3972051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3972147Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3972401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3972515Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3972768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3972854Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3972858Z 2025-08-26T20:38:38.3972978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3973189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3973265Z return mod(**inputs) 2025-08-26T20:38:38.3973496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3973607Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3973924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3974005Z outputs = self.model.decoder( 2025-08-26T20:38:38.3974242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3974320Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3974586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3974667Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3974903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3974997Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3975254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3975387Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3975645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.3975736Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.3975748Z 2025-08-26T20:38:38.3975837Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3975923Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3976015Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3976119Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.3976231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3976452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3976523Z return mod(**inputs) 2025-08-26T20:38:38.3976763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3976846Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3977122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3977205Z outputs = self.model.decoder( 2025-08-26T20:38:38.3977447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3977535Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3977807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3977892Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3978129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3978212Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3978479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3978588Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3978851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3978956Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3979266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.3979421Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.3979426Z 2025-08-26T20:38:38.3979537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3979759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3979829Z return mod(**inputs) 2025-08-26T20:38:38.3980094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3980178Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3980467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3980559Z outputs = self.model.decoder( 2025-08-26T20:38:38.3980791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3980879Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3981136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3981216Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3981458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3981542Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3981804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3981931Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3982182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.3982282Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.3982593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.3982743Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.3982747Z 2025-08-26T20:38:38.3982858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3983076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3983142Z return mod(**inputs) 2025-08-26T20:38:38.3983359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3983444Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3983683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3983763Z outputs = self.model.decoder( 2025-08-26T20:38:38.3983979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3984052Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3984299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3984372Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3984598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3984681Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3984929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3985027Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3985266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.3985355Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.3985359Z 2025-08-26T20:38:38.3985462Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3985673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3985740Z return mod(**inputs) 2025-08-26T20:38:38.3985954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3986034Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3986294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3986392Z outputs = self.model.decoder( 2025-08-26T20:38:38.3986612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3986694Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3986938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3987013Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3987242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3987324Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3987571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.3987654Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.3987675Z 2025-08-26T20:38:38.3987786Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3988009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3988080Z return mod(**inputs) 2025-08-26T20:38:38.3988320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3988400Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3988656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3988763Z outputs = self.model.decoder( 2025-08-26T20:38:38.3988995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3989082Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3989343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3989428Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3989666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3989759Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3990010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.3990109Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.3990114Z 2025-08-26T20:38:38.3990225Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3990426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3990492Z return mod(**inputs) 2025-08-26T20:38:38.3990720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3990797Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3991048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3991123Z outputs = self.model.decoder( 2025-08-26T20:38:38.3991340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3991421Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3991661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3991743Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3991966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3992050Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3992312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.3992397Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.3992421Z 2025-08-26T20:38:38.3992535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3992733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3992806Z return mod(**inputs) 2025-08-26T20:38:38.3993023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3993099Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3993348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3993423Z outputs = self.model.decoder( 2025-08-26T20:38:38.3993651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3993729Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3994007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3994090Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3994312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3994399Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3994636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3994797Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3995037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.3995151Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.3995155Z 2025-08-26T20:38:38.3995267Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3995470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3995546Z return mod(**inputs) 2025-08-26T20:38:38.3995766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3995843Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3996106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3996310Z outputs = self.model.decoder( 2025-08-26T20:38:38.3996564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3996645Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3996909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3996989Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.3997228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.3997323Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.3997581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.3997697Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.3997958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.3998050Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.3998054Z 2025-08-26T20:38:38.3998176Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.3998404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.3998481Z return mod(**inputs) 2025-08-26T20:38:38.3998764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3998889Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3999153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.3999232Z outputs = self.model.decoder( 2025-08-26T20:38:38.3999522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.3999611Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.3999888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.3999967Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4000211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4000308Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4000615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4000731Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4000997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.4001090Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.4001095Z 2025-08-26T20:38:38.4001193Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4001313Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4001406Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4001500Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4001608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4001820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4001887Z return mod(**inputs) 2025-08-26T20:38:38.4002126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4002199Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4002434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4002515Z outputs = self.model.decoder( 2025-08-26T20:38:38.4002729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4002812Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4003056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4003139Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4003364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4003448Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4003701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4003800Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4004052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4004152Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4004449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.4004594Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.4004598Z 2025-08-26T20:38:38.4004704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4004943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4005013Z return mod(**inputs) 2025-08-26T20:38:38.4005258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4005337Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4005579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4005663Z outputs = self.model.decoder( 2025-08-26T20:38:38.4005880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4005964Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4006218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4006295Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4006539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4006645Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4006911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4007016Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4007268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4007378Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4007707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.4007836Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.4007840Z 2025-08-26T20:38:38.4007951Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4008170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4008243Z return mod(**inputs) 2025-08-26T20:38:38.4008477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4008565Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4008822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4008910Z outputs = self.model.decoder( 2025-08-26T20:38:38.4009140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4009221Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4009485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4009563Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4009808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4009896Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4010152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4010264Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4010517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.4010615Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.4010620Z 2025-08-26T20:38:38.4010731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4010951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4011021Z return mod(**inputs) 2025-08-26T20:38:38.4011275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4011368Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4011640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4011728Z outputs = self.model.decoder( 2025-08-26T20:38:38.4011961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4012039Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4012302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4012382Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4012625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4012709Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4012968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.4013085Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.4013091Z 2025-08-26T20:38:38.4013202Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4013425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4013495Z return mod(**inputs) 2025-08-26T20:38:38.4013734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4013832Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4014087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4014174Z outputs = self.model.decoder( 2025-08-26T20:38:38.4014406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4014492Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4014750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4014829Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4015068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4015152Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4015411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.4015516Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.4015520Z 2025-08-26T20:38:38.4015635Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4015847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4015919Z return mod(**inputs) 2025-08-26T20:38:38.4016154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4016237Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4016500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4016580Z outputs = self.model.decoder( 2025-08-26T20:38:38.4016811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4016897Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4017155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4017238Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4017473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4017578Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4017863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.4017954Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.4017959Z 2025-08-26T20:38:38.4018081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4018300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4018379Z return mod(**inputs) 2025-08-26T20:38:38.4018619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4018703Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4018975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4019057Z outputs = self.model.decoder( 2025-08-26T20:38:38.4019312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4019412Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4019671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4019757Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4019992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4020084Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4020370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-26T20:38:38.4020514Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-26T20:38:38.4020525Z 2025-08-26T20:38:38.4020635Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4020849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4020930Z return mod(**inputs) 2025-08-26T20:38:38.4021161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4021249Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4021504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4021582Z outputs = self.model.decoder( 2025-08-26T20:38:38.4021818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4021898Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4022160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4022237Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4022471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4022567Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4022821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4022934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4023191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.4023310Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.4023322Z 2025-08-26T20:38:38.4023430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4023642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4023722Z return mod(**inputs) 2025-08-26T20:38:38.4023973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4024064Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4024336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4024417Z outputs = self.model.decoder( 2025-08-26T20:38:38.4024659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4024738Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4024999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4025079Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4025312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4025407Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4025662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4025797Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4026052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.4026148Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.4026151Z 2025-08-26T20:38:38.4026263Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4026476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4026580Z return mod(**inputs) 2025-08-26T20:38:38.4026815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4026901Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4027161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4027243Z outputs = self.model.decoder( 2025-08-26T20:38:38.4027484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4027562Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4027823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4027901Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4028133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4028228Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4028484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4028593Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4028851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.4028949Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.4028953Z 2025-08-26T20:38:38.4029038Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4029122Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4029210Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4029293Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4029411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4029626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4029697Z return mod(**inputs) 2025-08-26T20:38:38.4029937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4030015Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4030302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4030384Z outputs = self.model.decoder( 2025-08-26T20:38:38.4030636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4030725Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4030985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4031071Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4031311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4031401Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4031664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4031773Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4032056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4032163Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4032485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.4032628Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.4032632Z 2025-08-26T20:38:38.4032743Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4032983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4033053Z return mod(**inputs) 2025-08-26T20:38:38.4033294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4033374Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4033631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4033721Z outputs = self.model.decoder( 2025-08-26T20:38:38.4033954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4034041Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4034294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4034378Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4034615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4034700Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4034962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4035070Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4035333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4035436Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4035745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.4035872Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.4035876Z 2025-08-26T20:38:38.4035988Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4036211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4036283Z return mod(**inputs) 2025-08-26T20:38:38.4036521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4036641Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4036926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4037020Z outputs = self.model.decoder( 2025-08-26T20:38:38.4037258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4037347Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4037611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4037693Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4037944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4038031Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4038303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4038433Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4038694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.4038792Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.4038796Z 2025-08-26T20:38:38.4038910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4039136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4039210Z return mod(**inputs) 2025-08-26T20:38:38.4039750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4039846Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4040123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4040220Z outputs = self.model.decoder( 2025-08-26T20:38:38.4040461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4040552Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4040815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4040901Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4041134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4041212Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4041463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.4041545Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.4041551Z 2025-08-26T20:38:38.4041656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4041869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4041937Z return mod(**inputs) 2025-08-26T20:38:38.4042169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4042246Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4042512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4042594Z outputs = self.model.decoder( 2025-08-26T20:38:38.4042825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4042915Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4043171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4043255Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4043517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4043623Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4043888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.4043992Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.4043996Z 2025-08-26T20:38:38.4044111Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4044328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4044396Z return mod(**inputs) 2025-08-26T20:38:38.4044622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4044698Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4044967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4045073Z outputs = self.model.decoder( 2025-08-26T20:38:38.4045314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4045394Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4045651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4045736Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4045973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4046081Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4046338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.4046423Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.4046426Z 2025-08-26T20:38:38.4046546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4046760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4046836Z return mod(**inputs) 2025-08-26T20:38:38.4047069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4047147Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4047410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4047490Z outputs = self.model.decoder( 2025-08-26T20:38:38.4047731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4047810Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4048075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4048155Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4048390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4048482Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4048739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4048850Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4049111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.4049232Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.4049236Z 2025-08-26T20:38:38.4049353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4049564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4049663Z return mod(**inputs) 2025-08-26T20:38:38.4049918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4049999Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4050263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4050343Z outputs = self.model.decoder( 2025-08-26T20:38:38.4050579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4050661Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4050924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4051002Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4051238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4051367Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4051623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4051734Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4051993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.4052082Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.4052086Z 2025-08-26T20:38:38.4052208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4052449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4052527Z return mod(**inputs) 2025-08-26T20:38:38.4052763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4052852Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4053119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4053201Z outputs = self.model.decoder( 2025-08-26T20:38:38.4053444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4053526Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4053807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4053887Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4054121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4054213Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4054472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4054584Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4054840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.4054931Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.4054942Z 2025-08-26T20:38:38.4055028Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4055113Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4055202Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4055283Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4055396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4055615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4055682Z return mod(**inputs) 2025-08-26T20:38:38.4055940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4056024Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4056303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4056383Z outputs = self.model.decoder( 2025-08-26T20:38:38.4056615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4056701Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4056956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4057043Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4057278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4057362Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4057626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4057755Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4058018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4058123Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4058434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.4058585Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.4058606Z 2025-08-26T20:38:38.4058718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4058939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4059008Z return mod(**inputs) 2025-08-26T20:38:38.4059247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4059328Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4059588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4059673Z outputs = self.model.decoder( 2025-08-26T20:38:38.4059902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4059985Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4060240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4060322Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4060562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4060645Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4060909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4061017Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4061267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4061376Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4061817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.4062021Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.4062027Z 2025-08-26T20:38:38.4062182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4062521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4062616Z return mod(**inputs) 2025-08-26T20:38:38.4063003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4063152Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4063570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4063666Z outputs = self.model.decoder( 2025-08-26T20:38:38.4063905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4063987Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4064268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4064349Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4064611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4064698Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4064963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4065097Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4065360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.4065461Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.4065465Z 2025-08-26T20:38:38.4065579Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4065805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4065899Z return mod(**inputs) 2025-08-26T20:38:38.4066140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4066232Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4066500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4066592Z outputs = self.model.decoder( 2025-08-26T20:38:38.4066831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4066912Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4067184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4067266Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4067516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4067605Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4067879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.4067970Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.4067975Z 2025-08-26T20:38:38.4068092Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4068320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4068393Z return mod(**inputs) 2025-08-26T20:38:38.4068640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4068722Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4068985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4069075Z outputs = self.model.decoder( 2025-08-26T20:38:38.4069314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4069404Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4069685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4069769Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4070038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4070127Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4070399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.4070505Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.4070509Z 2025-08-26T20:38:38.4070629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4070845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4070919Z return mod(**inputs) 2025-08-26T20:38:38.4071165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4071249Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4071541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4071623Z outputs = self.model.decoder( 2025-08-26T20:38:38.4071860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4071949Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4072210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4072317Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4072564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4072649Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4072925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.4073015Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.4073019Z 2025-08-26T20:38:38.4073142Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4073357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4073436Z return mod(**inputs) 2025-08-26T20:38:38.4073677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4073759Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4074032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4074113Z outputs = self.model.decoder( 2025-08-26T20:38:38.4074481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4074591Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4074981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4075096Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4075402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4075497Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4075773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-26T20:38:38.4075964Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-26T20:38:38.4075968Z 2025-08-26T20:38:38.4076083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4076302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4076381Z return mod(**inputs) 2025-08-26T20:38:38.4076634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4076753Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4077021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4077103Z outputs = self.model.decoder( 2025-08-26T20:38:38.4077347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4077430Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4077703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4077781Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4078023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4078118Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4079223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4079340Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4079691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-26T20:38:38.4079828Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:38:38.4079834Z 2025-08-26T20:38:38.4079949Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4080203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4080286Z return mod(**inputs) 2025-08-26T20:38:38.4080525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4080615Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4080882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4080967Z outputs = self.model.decoder( 2025-08-26T20:38:38.4081213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4081296Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4081567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4081646Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4081892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4081988Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4082252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4082371Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4082635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-26T20:38:38.4082731Z key_states = self.k_proj(hidden_states) 2025-08-26T20:38:38.4082736Z 2025-08-26T20:38:38.4082848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4083065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4083147Z return mod(**inputs) 2025-08-26T20:38:38.4083384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4083479Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4083745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4083827Z outputs = self.model.decoder( 2025-08-26T20:38:38.4084097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4084184Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4084474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4084555Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4084806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4084892Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4085158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4085271Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4085541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-26T20:38:38.4085639Z value_states = self.v_proj(hidden_states) 2025-08-26T20:38:38.4085661Z 2025-08-26T20:38:38.4085751Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4085836Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4085925Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4086006Z cudagraph partition due to non gpu ops 2025-08-26T20:38:38.4086123Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4086335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4086405Z return mod(**inputs) 2025-08-26T20:38:38.4086662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4086741Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4087006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4087089Z outputs = self.model.decoder( 2025-08-26T20:38:38.4087326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4087422Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4087695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4087785Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4088030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4088127Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4088402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4088513Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4088791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4088904Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4089247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:38:38.4089394Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:38:38.4089398Z 2025-08-26T20:38:38.4089513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4089736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4089813Z return mod(**inputs) 2025-08-26T20:38:38.4090063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4090148Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4090419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4090526Z outputs = self.model.decoder( 2025-08-26T20:38:38.4090785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4090876Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4091141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4091236Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4091472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4091558Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4091825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4091929Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4092197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-26T20:38:38.4092320Z attn_output, attn_weights = attention_interface( 2025-08-26T20:38:38.4092634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:38:38.4092762Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:38:38.4092766Z 2025-08-26T20:38:38.4092874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4093094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4093185Z return mod(**inputs) 2025-08-26T20:38:38.4093424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4093502Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4093759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4093847Z outputs = self.model.decoder( 2025-08-26T20:38:38.4094082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4094167Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4094426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4094505Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4094759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4094848Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4095118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-26T20:38:38.4095226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:38:38.4095495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-26T20:38:38.4095597Z attn_output = self.out_proj(attn_output) 2025-08-26T20:38:38.4095602Z 2025-08-26T20:38:38.4095715Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4095943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4096016Z return mod(**inputs) 2025-08-26T20:38:38.4096534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4096658Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4097022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4097117Z outputs = self.model.decoder( 2025-08-26T20:38:38.4097358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4097526Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4097821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4097902Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4098151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4098237Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4098501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-26T20:38:38.4098592Z hidden_states = self.fc1(hidden_states) 2025-08-26T20:38:38.4098596Z 2025-08-26T20:38:38.4098714Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4098927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4098999Z return mod(**inputs) 2025-08-26T20:38:38.4099239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4099394Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4099660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4099740Z outputs = self.model.decoder( 2025-08-26T20:38:38.4099970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4100068Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4100356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4100444Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4100679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4100767Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4101038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-26T20:38:38.4101142Z hidden_states = self.activation_fn(hidden_states) 2025-08-26T20:38:38.4101146Z 2025-08-26T20:38:38.4101266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4101481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4101560Z return mod(**inputs) 2025-08-26T20:38:38.4101791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4101872Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4102133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-26T20:38:38.4102212Z outputs = self.model.decoder( 2025-08-26T20:38:38.4102451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4102533Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4102791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-26T20:38:38.4102876Z layer_outputs = decoder_layer( 2025-08-26T20:38:38.4103110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:38:38.4103202Z return super().__call__(*args, **kwargs) 2025-08-26T20:38:38.4103460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-26T20:38:38.4103546Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:38:38.4103550Z 2025-08-26T20:38:38.4103664Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4103907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4103989Z return mod(**inputs) 2025-08-26T20:38:38.4104238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4104328Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4104586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 841, in forward 2025-08-26T20:38:38.4104688Z logits = self.lm_head(outputs[0]).contiguous() 2025-08-26T20:38:38.4104692Z 2025-08-26T20:38:38.4104809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:38:38.4105020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:38:38.4105097Z return mod(**inputs) 2025-08-26T20:38:38.4105329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-26T20:38:38.4105411Z output = func(self, *args, **kwargs) 2025-08-26T20:38:38.4105696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 847, in forward 2025-08-26T20:38:38.4105776Z loss = self.loss_function( 2025-08-26T20:38:38.4106044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-26T20:38:38.4106234Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-26T20:38:38.4106505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-26T20:38:38.4106738Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-26T20:38:38.4106742Z 2025-08-26T20:38:49.2688557Z Compilation time (from dynamo_timed): 16.474647327 2025-08-26T20:38:49.3278503Z pass 2025-08-26T20:38:49.3279010Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:49.3279972Z TIMING: _recursive_pre_grad_passes:0.00836 _recursive_joint_graph_passes:0.6466 _recursive_post_grad_passes:0.10162 async_compile.wait:0.8266 code_gen:9.12111 inductor_compile:10.43024 backend_compile:13.69075 gc:0.00118 entire_frame_compile:16.47465 total_wall_time:16.47465 2025-08-26T20:38:49.3281058Z STATS: call_* op count: 415 | FakeTensorMode.__torch_dispatch__:12795 | FakeTensor.__torch_dispatch__:4179 | ProxyTorchDispatchMode.__torch_dispatch__:4707 2025-08-26T20:38:49.3281615Z Dynamo produced 1 graphs covering 415 ops with 0 graph breaks (0 unique) 2025-08-26T20:38:54.7637264Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:38:54.7638267Z from pkg_resources import resource_filename 2025-08-26T20:38:55.6867081Z 2025-08-26T20:38:57.1586874Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:38:57.1587197Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:38:57.1592825Z cpu eval PLBartForCausalLM 2025-08-26T20:38:57.8395283Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:58.1478263Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:38:58.4454608Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:03.6465440Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6465874Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6466104Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6466316Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6466517Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6467060Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6467319Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6467786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6468135Z return mod(**inputs) 2025-08-26T20:39:03.6468565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6469001Z outputs = self.model.decoder( 2025-08-26T20:39:03.6469426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6469837Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6470193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6470570Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6470989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6471473Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6471901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:03.6472383Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:03.6472608Z 2025-08-26T20:39:03.6472726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6473113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6473507Z return mod(**inputs) 2025-08-26T20:39:03.6473904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6474336Z outputs = self.model.decoder( 2025-08-26T20:39:03.6474774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6475224Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6475603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6476001Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6476444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6476900Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6477357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:03.6477790Z key_states = self.k_proj(current_states) 2025-08-26T20:39:03.6477942Z 2025-08-26T20:39:03.6478055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6478453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6478808Z return mod(**inputs) 2025-08-26T20:39:03.6479247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6479908Z outputs = self.model.decoder( 2025-08-26T20:39:03.6480330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6480761Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6481150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6481587Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6481982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6482407Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6482855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:03.6483337Z value_states = self.v_proj(current_states) 2025-08-26T20:39:03.6483483Z 2025-08-26T20:39:03.6483574Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6483783Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6483996Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6484207Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6484573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6484935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6485268Z return mod(**inputs) 2025-08-26T20:39:03.6485697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6486101Z outputs = self.model.decoder( 2025-08-26T20:39:03.6486494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6486912Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6487269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6487640Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6488043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6488460Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6489021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6489454Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6489916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:03.6490415Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:03.6490606Z 2025-08-26T20:39:03.6490713Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6491095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6491422Z return mod(**inputs) 2025-08-26T20:39:03.6491801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6492213Z outputs = self.model.decoder( 2025-08-26T20:39:03.6492603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6493008Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6493366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6493741Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6494144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6494566Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6494973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6495383Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6495824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:03.6496490Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:03.6496664Z 2025-08-26T20:39:03.6496768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6497171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6497500Z return mod(**inputs) 2025-08-26T20:39:03.6497898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6498286Z outputs = self.model.decoder( 2025-08-26T20:39:03.6498678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6499073Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6499431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6499810Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6500212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6500646Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6501073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:03.6501516Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:03.6501655Z 2025-08-26T20:39:03.6501760Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6502132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6502461Z return mod(**inputs) 2025-08-26T20:39:03.6502835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6503314Z outputs = self.model.decoder( 2025-08-26T20:39:03.6503734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6504127Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6504476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6504846Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6505269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6505744Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6505930Z 2025-08-26T20:39:03.6506048Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6506428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6506780Z return mod(**inputs) 2025-08-26T20:39:03.6507154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6507537Z outputs = self.model.decoder( 2025-08-26T20:39:03.6507919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6508333Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6508710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6509097Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6509520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6509978Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6510394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:03.6510760Z return self.act(input) 2025-08-26T20:39:03.6510880Z 2025-08-26T20:39:03.6510999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6511384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6511786Z return mod(**inputs) 2025-08-26T20:39:03.6512214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6512639Z outputs = self.model.decoder( 2025-08-26T20:39:03.6513055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6513476Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6513843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6514233Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6514658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:03.6515084Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:03.6515232Z 2025-08-26T20:39:03.6515344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6515736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6516110Z return mod(**inputs) 2025-08-26T20:39:03.6516507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6516927Z outputs = self.model.decoder( 2025-08-26T20:39:03.6517335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6517754Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6518153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6518540Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6518958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6519410Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6519930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:03.6520439Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:03.6520659Z 2025-08-26T20:39:03.6520778Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6521158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6521499Z return mod(**inputs) 2025-08-26T20:39:03.6521882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6522283Z outputs = self.model.decoder( 2025-08-26T20:39:03.6522693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6523109Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6523468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6523838Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6524241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6524660Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6525075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:03.6525484Z key_states = self.k_proj(current_states) 2025-08-26T20:39:03.6525628Z 2025-08-26T20:39:03.6525734Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6526098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6526420Z return mod(**inputs) 2025-08-26T20:39:03.6526816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6527241Z outputs = self.model.decoder( 2025-08-26T20:39:03.6527634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6528027Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6528370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6528736Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6529134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6529550Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6529969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:03.6530397Z value_states = self.v_proj(current_states) 2025-08-26T20:39:03.6530549Z 2025-08-26T20:39:03.6530634Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6530855Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6531071Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6531276Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6531515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6531929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6532293Z return mod(**inputs) 2025-08-26T20:39:03.6532665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6533056Z outputs = self.model.decoder( 2025-08-26T20:39:03.6533446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6533850Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6534195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6534545Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6534936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6535358Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6535778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6536201Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6536675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:03.6537188Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:03.6537387Z 2025-08-26T20:39:03.6537492Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6537860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6538226Z return mod(**inputs) 2025-08-26T20:39:03.6538595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6538995Z outputs = self.model.decoder( 2025-08-26T20:39:03.6539389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6539792Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6540140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6540507Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6541555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6542024Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6542457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6542907Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6543387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:03.6543890Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:03.6544065Z 2025-08-26T20:39:03.6544191Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6544558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6544885Z return mod(**inputs) 2025-08-26T20:39:03.6545264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6545699Z outputs = self.model.decoder( 2025-08-26T20:39:03.6546090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6546482Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6546835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6547201Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6547622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6548037Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6548446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:03.6548854Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:03.6549000Z 2025-08-26T20:39:03.6549107Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6549472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6549799Z return mod(**inputs) 2025-08-26T20:39:03.6550166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6550561Z outputs = self.model.decoder( 2025-08-26T20:39:03.6550953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6551352Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6551693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6552061Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6552459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6552911Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6553097Z 2025-08-26T20:39:03.6553215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6553595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6553945Z return mod(**inputs) 2025-08-26T20:39:03.6554334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6554733Z outputs = self.model.decoder( 2025-08-26T20:39:03.6555117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6555528Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6555929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6556366Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6556806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6557302Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6557728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:03.6558102Z return self.act(input) 2025-08-26T20:39:03.6558227Z 2025-08-26T20:39:03.6558351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6558747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6559094Z return mod(**inputs) 2025-08-26T20:39:03.6559596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6560075Z outputs = self.model.decoder( 2025-08-26T20:39:03.6560503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6560928Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6561319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6561691Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6562095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:03.6562546Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:03.6562688Z 2025-08-26T20:39:03.6562793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6563160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6563490Z return mod(**inputs) 2025-08-26T20:39:03.6563864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6564264Z outputs = self.model.decoder( 2025-08-26T20:39:03.6564645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6565042Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6565392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6565761Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6566156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6566613Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6567056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:03.6567557Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:03.6567783Z 2025-08-26T20:39:03.6567905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6568304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6568656Z return mod(**inputs) 2025-08-26T20:39:03.6569051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6569478Z outputs = self.model.decoder( 2025-08-26T20:39:03.6569889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6570302Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6570698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6571095Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6571536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6571977Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6572425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:03.6572851Z key_states = self.k_proj(current_states) 2025-08-26T20:39:03.6572999Z 2025-08-26T20:39:03.6573121Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6573509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6573852Z return mod(**inputs) 2025-08-26T20:39:03.6574249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6574758Z outputs = self.model.decoder( 2025-08-26T20:39:03.6575175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6575578Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6575931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6576321Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6576742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6577199Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6577607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:03.6578017Z value_states = self.v_proj(current_states) 2025-08-26T20:39:03.6578165Z 2025-08-26T20:39:03.6578247Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6578465Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6578676Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6578874Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6579106Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6579460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6579786Z return mod(**inputs) 2025-08-26T20:39:03.6580153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6580554Z outputs = self.model.decoder( 2025-08-26T20:39:03.6580940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6581333Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6581689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6582051Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6582451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6582871Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6583283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6583698Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6584151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:03.6584635Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:03.6584826Z 2025-08-26T20:39:03.6584961Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6585352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6585713Z return mod(**inputs) 2025-08-26T20:39:03.6586112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6586536Z outputs = self.model.decoder( 2025-08-26T20:39:03.6586952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6587375Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6587753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6588128Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6588527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6588949Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6589398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6589836Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6590317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:03.6590814Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:03.6590985Z 2025-08-26T20:39:03.6591131Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6591516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6591847Z return mod(**inputs) 2025-08-26T20:39:03.6592225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6592633Z outputs = self.model.decoder( 2025-08-26T20:39:03.6593036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6593425Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6593785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6594168Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6594598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6595034Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6595448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:03.6595863Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:03.6596017Z 2025-08-26T20:39:03.6596132Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6596729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6597076Z return mod(**inputs) 2025-08-26T20:39:03.6597479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6597904Z outputs = self.model.decoder( 2025-08-26T20:39:03.6598323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6598747Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6599113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6599563Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6600082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6600577Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6600769Z 2025-08-26T20:39:03.6600935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6601319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6601669Z return mod(**inputs) 2025-08-26T20:39:03.6602069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6602497Z outputs = self.model.decoder( 2025-08-26T20:39:03.6602906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6603328Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6603707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6604102Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6604562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6605025Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6605450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:03.6605816Z return self.act(input) 2025-08-26T20:39:03.6605937Z 2025-08-26T20:39:03.6606055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6606499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6606839Z return mod(**inputs) 2025-08-26T20:39:03.6607235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6607657Z outputs = self.model.decoder( 2025-08-26T20:39:03.6608072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6608486Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6608855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6609239Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6609641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:03.6610048Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:03.6610188Z 2025-08-26T20:39:03.6610296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6610664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6610992Z return mod(**inputs) 2025-08-26T20:39:03.6611368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6611755Z outputs = self.model.decoder( 2025-08-26T20:39:03.6612159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6612575Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6612946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6613335Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6613753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6614204Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6614648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:03.6615176Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:03.6615401Z 2025-08-26T20:39:03.6615543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6615930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6616279Z return mod(**inputs) 2025-08-26T20:39:03.6616660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6617065Z outputs = self.model.decoder( 2025-08-26T20:39:03.6617458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6617869Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6618225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6618603Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6619010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6619471Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6619891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:03.6620295Z key_states = self.k_proj(current_states) 2025-08-26T20:39:03.6620432Z 2025-08-26T20:39:03.6620544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6620912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6621256Z return mod(**inputs) 2025-08-26T20:39:03.6621630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6622030Z outputs = self.model.decoder( 2025-08-26T20:39:03.6622424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6622826Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6623176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6623544Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6623946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6624371Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6624784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:03.6625201Z value_states = self.v_proj(current_states) 2025-08-26T20:39:03.6625360Z 2025-08-26T20:39:03.6625451Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6625687Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6625917Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6626135Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6626395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6626783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6627134Z return mod(**inputs) 2025-08-26T20:39:03.6627524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6627948Z outputs = self.model.decoder( 2025-08-26T20:39:03.6628363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6628780Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6629154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6629573Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6630027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6630478Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6630920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6631367Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6631859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:03.6632387Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:03.6632587Z 2025-08-26T20:39:03.6632708Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6633094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6633466Z return mod(**inputs) 2025-08-26T20:39:03.6633863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6634285Z outputs = self.model.decoder( 2025-08-26T20:39:03.6634700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6635122Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6635488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6635906Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6636357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6636807Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6637250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6637702Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6638185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:03.6638697Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:03.6638878Z 2025-08-26T20:39:03.6639005Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6639403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6639876Z return mod(**inputs) 2025-08-26T20:39:03.6640307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6640744Z outputs = self.model.decoder( 2025-08-26T20:39:03.6641181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6641595Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6641975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6642350Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6642747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6643166Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6643614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:03.6644043Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:03.6644195Z 2025-08-26T20:39:03.6644316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6644744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6645100Z return mod(**inputs) 2025-08-26T20:39:03.6645499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6645899Z outputs = self.model.decoder( 2025-08-26T20:39:03.6646291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6646704Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6647093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6647465Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6647864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6648315Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6648500Z 2025-08-26T20:39:03.6648646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6649030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6649381Z return mod(**inputs) 2025-08-26T20:39:03.6649777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6650201Z outputs = self.model.decoder( 2025-08-26T20:39:03.6650600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6651020Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6651373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6651751Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6652179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6652643Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6653061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:03.6653424Z return self.act(input) 2025-08-26T20:39:03.6653548Z 2025-08-26T20:39:03.6653656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6654009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6654330Z return mod(**inputs) 2025-08-26T20:39:03.6654696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6655094Z outputs = self.model.decoder( 2025-08-26T20:39:03.6655485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6655873Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6656228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6656604Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6657027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:03.6657447Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:03.6657586Z 2025-08-26T20:39:03.6657690Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6658054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6658382Z return mod(**inputs) 2025-08-26T20:39:03.6658756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6659190Z outputs = self.model.decoder( 2025-08-26T20:39:03.6659645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6660068Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6660444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6660833Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6661254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6661705Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6662151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:03.6662655Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:03.6662874Z 2025-08-26T20:39:03.6662993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6663402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6663747Z return mod(**inputs) 2025-08-26T20:39:03.6664140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6664559Z outputs = self.model.decoder( 2025-08-26T20:39:03.6664948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6665387Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6665760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6666139Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6666548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6666966Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6667389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:03.6667796Z key_states = self.k_proj(current_states) 2025-08-26T20:39:03.6667935Z 2025-08-26T20:39:03.6668048Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6668415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6668739Z return mod(**inputs) 2025-08-26T20:39:03.6669115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6669564Z outputs = self.model.decoder( 2025-08-26T20:39:03.6669983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6670395Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6670777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6671163Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6671585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6672030Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6672464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:03.6672902Z value_states = self.v_proj(current_states) 2025-08-26T20:39:03.6673060Z 2025-08-26T20:39:03.6673147Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6673376Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6673605Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6673852Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6674110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6674528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6674880Z return mod(**inputs) 2025-08-26T20:39:03.6675271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6675698Z outputs = self.model.decoder( 2025-08-26T20:39:03.6676116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6676540Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6676909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6677299Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6677721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6678255Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6678714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6679160Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6679743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:03.6680306Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:03.6680539Z 2025-08-26T20:39:03.6680661Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6681054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6681408Z return mod(**inputs) 2025-08-26T20:39:03.6681815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6682245Z outputs = self.model.decoder( 2025-08-26T20:39:03.6682669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6683087Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6683458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6683845Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6684270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6684719Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6685151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6685598Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6686075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:03.6686568Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:03.6686742Z 2025-08-26T20:39:03.6686864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6687242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6687592Z return mod(**inputs) 2025-08-26T20:39:03.6687993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6688417Z outputs = self.model.decoder( 2025-08-26T20:39:03.6688865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6689316Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6689705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6690120Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6690557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6691018Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6691492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:03.6691931Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:03.6692081Z 2025-08-26T20:39:03.6692203Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6692596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6692947Z return mod(**inputs) 2025-08-26T20:39:03.6693354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6693821Z outputs = self.model.decoder( 2025-08-26T20:39:03.6694236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6694649Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6695019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6695418Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6695884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6696500Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6696697Z 2025-08-26T20:39:03.6696811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6697215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6697583Z return mod(**inputs) 2025-08-26T20:39:03.6697985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6698413Z outputs = self.model.decoder( 2025-08-26T20:39:03.6698824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6699248Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6699627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6699997Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6700425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6700909Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6701330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:03.6701698Z return self.act(input) 2025-08-26T20:39:03.6701819Z 2025-08-26T20:39:03.6701937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6702314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6702662Z return mod(**inputs) 2025-08-26T20:39:03.6703063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6703490Z outputs = self.model.decoder( 2025-08-26T20:39:03.6703905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6704319Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6704766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6705160Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6705612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:03.6706047Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:03.6706195Z 2025-08-26T20:39:03.6706305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6706696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6707049Z return mod(**inputs) 2025-08-26T20:39:03.6707446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6707859Z outputs = self.model.decoder( 2025-08-26T20:39:03.6708274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6708725Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6709101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6709487Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6709897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6710342Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6710783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:03.6711319Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:03.6711534Z 2025-08-26T20:39:03.6711651Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6712039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6712399Z return mod(**inputs) 2025-08-26T20:39:03.6712814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6713245Z outputs = self.model.decoder( 2025-08-26T20:39:03.6713669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6714103Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6714475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6714867Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6715306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6715758Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6716225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:03.6716679Z key_states = self.k_proj(current_states) 2025-08-26T20:39:03.6716829Z 2025-08-26T20:39:03.6716952Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6717348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6717703Z return mod(**inputs) 2025-08-26T20:39:03.6718100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6718527Z outputs = self.model.decoder( 2025-08-26T20:39:03.6718959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6719398Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6719890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6720315Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6720786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6721239Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6721678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:03.6722127Z value_states = self.v_proj(current_states) 2025-08-26T20:39:03.6722293Z 2025-08-26T20:39:03.6722385Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6722624Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6722851Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6723082Z cudagraph partition due to non gpu ops 2025-08-26T20:39:03.6723342Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6723746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6724137Z return mod(**inputs) 2025-08-26T20:39:03.6724562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6725004Z outputs = self.model.decoder( 2025-08-26T20:39:03.6725450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6725894Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6726300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6726697Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6727130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6727589Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6728042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6728492Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6728981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:03.6729518Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:03.6729729Z 2025-08-26T20:39:03.6729847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6730235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6730572Z return mod(**inputs) 2025-08-26T20:39:03.6730971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6731404Z outputs = self.model.decoder( 2025-08-26T20:39:03.6731822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6732233Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6732609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6732996Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6733421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6733871Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6734306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:03.6734755Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:03.6735271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:03.6735792Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:03.6735968Z 2025-08-26T20:39:03.6736087Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6736468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6736817Z return mod(**inputs) 2025-08-26T20:39:03.6737214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6737643Z outputs = self.model.decoder( 2025-08-26T20:39:03.6738049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6738469Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6738843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6739252Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6739681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:03.6740123Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:03.6740572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:03.6741004Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:03.6741151Z 2025-08-26T20:39:03.6741296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6741681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6742019Z return mod(**inputs) 2025-08-26T20:39:03.6742419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6742853Z outputs = self.model.decoder( 2025-08-26T20:39:03.6743287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6743704Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6744077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6744468Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6744890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6745358Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6745544Z 2025-08-26T20:39:03.6745655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6746039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6746388Z return mod(**inputs) 2025-08-26T20:39:03.6746785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6747207Z outputs = self.model.decoder( 2025-08-26T20:39:03.6747616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6748040Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6748416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6748807Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6749219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:03.6749710Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:03.6750158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:03.6750541Z return self.act(input) 2025-08-26T20:39:03.6750663Z 2025-08-26T20:39:03.6750804Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6751196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6751572Z return mod(**inputs) 2025-08-26T20:39:03.6751977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-26T20:39:03.6752425Z outputs = self.model.decoder( 2025-08-26T20:39:03.6752854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:03.6753295Z layer_outputs = decoder_layer( 2025-08-26T20:39:03.6753680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:03.6754081Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:03.6754543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:03.6754976Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:03.6755134Z 2025-08-26T20:39:03.6755247Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6755648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6756006Z return mod(**inputs) 2025-08-26T20:39:03.6756408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1694, in forward 2025-08-26T20:39:03.6756861Z logits = self.lm_head(outputs[0]) 2025-08-26T20:39:03.6757017Z 2025-08-26T20:39:03.6757130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:03.6757523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:03.6757878Z return mod(**inputs) 2025-08-26T20:39:03.6758286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1700, in forward 2025-08-26T20:39:03.6758793Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:39:03.6759019Z 2025-08-26T20:39:11.8797677Z Compilation time (from dynamo_timed): 11.797717227 2025-08-26T20:39:11.9108866Z pass 2025-08-26T20:39:11.9111097Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:11.9112057Z TIMING: _recursive_pre_grad_passes:0.00598 _recursive_joint_graph_passes:0.26338 _recursive_post_grad_passes:0.05248 async_compile.wait:0.77557 code_gen:7.52943 inductor_compile:8.56939 backend_compile:10.47161 gc:0.001 entire_frame_compile:11.79772 total_wall_time:11.79772 2025-08-26T20:39:11.9113048Z STATS: call_* op count: 198 | FakeTensorMode.__torch_dispatch__:7096 | FakeTensor.__torch_dispatch__:2414 | ProxyTorchDispatchMode.__torch_dispatch__:2533 2025-08-26T20:39:11.9113598Z Dynamo produced 1 graphs covering 198 ops with 0 graph breaks (0 unique) 2025-08-26T20:39:17.2677401Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:39:17.2678412Z from pkg_resources import resource_filename 2025-08-26T20:39:17.9655694Z 2025-08-26T20:39:20.4920615Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:39:20.4924388Z loading model: 0it [00:02, ?it/s] 2025-08-26T20:39:20.4935576Z cpu eval PLBartForConditionalGeneration 2025-08-26T20:39:21.6904443Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:22.2315659Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:22.7779971Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:32.9921100Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9921597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9922012Z return mod(**inputs) 2025-08-26T20:39:32.9922455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1357, in forward 2025-08-26T20:39:32.9923067Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-08-26T20:39:32.9924852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1084, in shift_tokens_right 2025-08-26T20:39:32.9925441Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-08-26T20:39:32.9925668Z 2025-08-26T20:39:32.9925772Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9926332Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9926576Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9926806Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9927044Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9927282Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9927554Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9928017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9928441Z return mod(**inputs) 2025-08-26T20:39:32.9928854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9929299Z outputs = self.model( 2025-08-26T20:39:32.9929715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9930178Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9930625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9931059Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9931444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9931842Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9932281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9932747Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9933237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:32.9933771Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:32.9934026Z 2025-08-26T20:39:32.9934153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9934542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9934916Z return mod(**inputs) 2025-08-26T20:39:32.9935320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9935764Z outputs = self.model( 2025-08-26T20:39:32.9936188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9936618Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9937056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9937492Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9937947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9938363Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9938846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9939352Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9939797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:32.9940243Z key_states = self.k_proj(current_states) 2025-08-26T20:39:32.9940402Z 2025-08-26T20:39:32.9940516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9940906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9941253Z return mod(**inputs) 2025-08-26T20:39:32.9941650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9942105Z outputs = self.model( 2025-08-26T20:39:32.9942529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9942967Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9943408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9943849Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9944244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9944673Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9945112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9945553Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9946007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:32.9946447Z value_states = self.v_proj(current_states) 2025-08-26T20:39:32.9946599Z 2025-08-26T20:39:32.9946695Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9947088Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9947310Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9947535Z cudagraph partition due to non gpu ops 2025-08-26T20:39:32.9947788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9948181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9948528Z return mod(**inputs) 2025-08-26T20:39:32.9948935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9949357Z outputs = self.model( 2025-08-26T20:39:32.9949760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9950187Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9950614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9951055Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9951450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9951860Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9952306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9952769Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9953235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:32.9953711Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:32.9954236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:32.9954766Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:32.9954980Z 2025-08-26T20:39:32.9955097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9955495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9955878Z return mod(**inputs) 2025-08-26T20:39:32.9956294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9956730Z outputs = self.model( 2025-08-26T20:39:32.9957144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9957586Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9958039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9958468Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9958867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9959279Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9959945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9960437Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9960881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:32.9961351Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:32.9961843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:32.9962358Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:32.9962544Z 2025-08-26T20:39:32.9962668Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9963059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9963420Z return mod(**inputs) 2025-08-26T20:39:32.9963829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9964266Z outputs = self.model( 2025-08-26T20:39:32.9964667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9965099Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9965528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9965961Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9966350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9966742Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9967178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9967630Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9968078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:32.9968640Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:32.9968797Z 2025-08-26T20:39:32.9968914Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9969338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9969703Z return mod(**inputs) 2025-08-26T20:39:32.9970175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9970604Z outputs = self.model( 2025-08-26T20:39:32.9971032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9971465Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9971889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9972322Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9972699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9973112Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9973571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:32.9974086Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:32.9974281Z 2025-08-26T20:39:32.9974405Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9974801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9975150Z return mod(**inputs) 2025-08-26T20:39:32.9975541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9975979Z outputs = self.model( 2025-08-26T20:39:32.9976367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9976786Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9977205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9977627Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9978001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9978380Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9978800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:32.9979269Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:32.9979705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:32.9980099Z return self.act(input) 2025-08-26T20:39:32.9980220Z 2025-08-26T20:39:32.9980330Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9980715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9981065Z return mod(**inputs) 2025-08-26T20:39:32.9981464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9981874Z outputs = self.model( 2025-08-26T20:39:32.9982267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9982684Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9983094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9983521Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9983896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9984292Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9984743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-26T20:39:32.9985198Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:32.9985349Z 2025-08-26T20:39:32.9985489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9985872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9986218Z return mod(**inputs) 2025-08-26T20:39:32.9986612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9987030Z outputs = self.model( 2025-08-26T20:39:32.9987419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9987846Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9988262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9988680Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9989080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9989461Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9989886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9990323Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9990756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:32.9991279Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:32.9991498Z 2025-08-26T20:39:32.9991610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9991993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9992342Z return mod(**inputs) 2025-08-26T20:39:32.9992739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9993162Z outputs = self.model( 2025-08-26T20:39:32.9993591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:32.9994022Z encoder_outputs = self.encoder( 2025-08-26T20:39:32.9994446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:32.9994880Z layer_outputs = encoder_layer( 2025-08-26T20:39:32.9995323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:32.9995732Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:32.9996534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:32.9997009Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:32.9997473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:32.9997919Z key_states = self.k_proj(current_states) 2025-08-26T20:39:32.9998081Z 2025-08-26T20:39:32.9998198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:32.9998598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:32.9998963Z return mod(**inputs) 2025-08-26T20:39:32.9999370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:32.9999884Z outputs = self.model( 2025-08-26T20:39:33.0000315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0001613Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0002093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0002538Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0002927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0003331Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0003768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0004215Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0004664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0005118Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0005277Z 2025-08-26T20:39:33.0005377Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0005664Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0005892Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0006124Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0006380Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0006744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0007074Z return mod(**inputs) 2025-08-26T20:39:33.0007475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0007935Z outputs = self.model( 2025-08-26T20:39:33.0008348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0008754Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0009145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0009547Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0009911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0010287Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0010685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0011107Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0011527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0011960Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0012416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0012904Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0013107Z 2025-08-26T20:39:33.0013215Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0013593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0013931Z return mod(**inputs) 2025-08-26T20:39:33.0014311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0014707Z outputs = self.model( 2025-08-26T20:39:33.0015088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0015496Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0015893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0016296Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0016673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0017066Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0017475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0017899Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0018310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0018740Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0019198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0019672Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0019838Z 2025-08-26T20:39:33.0019952Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0020336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0020668Z return mod(**inputs) 2025-08-26T20:39:33.0021041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0021445Z outputs = self.model( 2025-08-26T20:39:33.0021841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0022248Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0022659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0023054Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0023407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0023768Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0024173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0024591Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0025005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0025434Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0025589Z 2025-08-26T20:39:33.0025695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0026062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0026392Z return mod(**inputs) 2025-08-26T20:39:33.0026766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0027172Z outputs = self.model( 2025-08-26T20:39:33.0027555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0027986Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0028401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0028822Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0029191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0029588Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0030015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0030503Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0030691Z 2025-08-26T20:39:33.0030810Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0031233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0031622Z return mod(**inputs) 2025-08-26T20:39:33.0032023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0032438Z outputs = self.model( 2025-08-26T20:39:33.0032826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0033243Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0033657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0034087Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0034460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0034850Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0035299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0035767Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0036160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0036508Z return self.act(input) 2025-08-26T20:39:33.0036621Z 2025-08-26T20:39:33.0036725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0037090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0037441Z return mod(**inputs) 2025-08-26T20:39:33.0037811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0038198Z outputs = self.model( 2025-08-26T20:39:33.0038578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0039003Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0039417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0039956Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0040328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0040718Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0041126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-26T20:39:33.0041540Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0041682Z 2025-08-26T20:39:33.0041795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0042161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0042503Z return mod(**inputs) 2025-08-26T20:39:33.0042883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0043285Z outputs = self.model( 2025-08-26T20:39:33.0043655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0044057Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0044445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0044846Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0045201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0045564Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0045990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0046427Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0046844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0047322Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0047531Z 2025-08-26T20:39:33.0047637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0048004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0048336Z return mod(**inputs) 2025-08-26T20:39:33.0048711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0049102Z outputs = self.model( 2025-08-26T20:39:33.0049480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0049901Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0050297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0050693Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0051040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0051407Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0051830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0052261Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0052669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0053081Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0053237Z 2025-08-26T20:39:33.0053348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0053736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0054068Z return mod(**inputs) 2025-08-26T20:39:33.0054433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0054828Z outputs = self.model( 2025-08-26T20:39:33.0055202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0055602Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0055992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0056377Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0056729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0057099Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0057504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0057912Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0058320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0058731Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0058883Z 2025-08-26T20:39:33.0058965Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0059185Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0059403Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0059626Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0059906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0060277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0060626Z return mod(**inputs) 2025-08-26T20:39:33.0061003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0061399Z outputs = self.model( 2025-08-26T20:39:33.0061772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0062166Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0062562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0062979Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0063351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0063746Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0064195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0064650Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0065099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0065544Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0066015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0066520Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0066715Z 2025-08-26T20:39:33.0066821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0067194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0067529Z return mod(**inputs) 2025-08-26T20:39:33.0067911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0068303Z outputs = self.model( 2025-08-26T20:39:33.0068687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0069091Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0069500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0069927Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0070300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0070693Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0071134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0071577Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0071991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0072426Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0072878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0073377Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0073553Z 2025-08-26T20:39:33.0073671Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0074063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0074411Z return mod(**inputs) 2025-08-26T20:39:33.0074849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0075273Z outputs = self.model( 2025-08-26T20:39:33.0075686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0076106Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0076517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0076945Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0077322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0077706Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0078129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0078585Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0079055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0079612Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0079772Z 2025-08-26T20:39:33.0079890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0080291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0080670Z return mod(**inputs) 2025-08-26T20:39:33.0081065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0081531Z outputs = self.model( 2025-08-26T20:39:33.0081900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0082303Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0082693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0083109Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0083477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0083869Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0084290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0084778Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0084954Z 2025-08-26T20:39:33.0085066Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0085435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0085783Z return mod(**inputs) 2025-08-26T20:39:33.0086175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0086594Z outputs = self.model( 2025-08-26T20:39:33.0086993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0087402Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0087819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0088239Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0088613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0089004Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0089427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0089891Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0090329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0090721Z return self.act(input) 2025-08-26T20:39:33.0090840Z 2025-08-26T20:39:33.0090952Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0091336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0091681Z return mod(**inputs) 2025-08-26T20:39:33.0092069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0092484Z outputs = self.model( 2025-08-26T20:39:33.0093054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0093495Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0093917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0094361Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0094734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0095125Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0095556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-26T20:39:33.0095987Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0096138Z 2025-08-26T20:39:33.0096391Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0096840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0097188Z return mod(**inputs) 2025-08-26T20:39:33.0097580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0098000Z outputs = self.model( 2025-08-26T20:39:33.0098373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0098774Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0099167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0099563Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0099920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0100286Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0100681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0101097Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0101509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0102009Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0102239Z 2025-08-26T20:39:33.0102344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0102712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0103042Z return mod(**inputs) 2025-08-26T20:39:33.0103420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0103811Z outputs = self.model( 2025-08-26T20:39:33.0104185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0104583Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0105008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0105408Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0105781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0106155Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0106556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0106972Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0107387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0107789Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0107936Z 2025-08-26T20:39:33.0108042Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0108409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0108815Z return mod(**inputs) 2025-08-26T20:39:33.0109188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0109594Z outputs = self.model( 2025-08-26T20:39:33.0109958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0110349Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0110743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0111154Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0111511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0111876Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0112274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0112692Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0113098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0113507Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0113660Z 2025-08-26T20:39:33.0113742Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0113961Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0114181Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0114404Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0114655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0115043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0115384Z return mod(**inputs) 2025-08-26T20:39:33.0115781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0116196Z outputs = self.model( 2025-08-26T20:39:33.0116592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0117013Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0117419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0117837Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0118211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0118600Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0119019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0119530Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0119982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0120449Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0120942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0121455Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0121650Z 2025-08-26T20:39:33.0121756Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0122122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0122456Z return mod(**inputs) 2025-08-26T20:39:33.0122852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0123262Z outputs = self.model( 2025-08-26T20:39:33.0123641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0124070Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0124473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0124862Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0125212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0125582Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0126000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0126419Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0126815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0127237Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0127685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0128147Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0128315Z 2025-08-26T20:39:33.0128426Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0128784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0129113Z return mod(**inputs) 2025-08-26T20:39:33.0129484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0129878Z outputs = self.model( 2025-08-26T20:39:33.0130251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0130641Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0131032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0131445Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0131827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0132188Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0132587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0133004Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0133417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0133820Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0133958Z 2025-08-26T20:39:33.0134083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0134469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0134805Z return mod(**inputs) 2025-08-26T20:39:33.0135176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0135568Z outputs = self.model( 2025-08-26T20:39:33.0135938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0136339Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0136726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0137123Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0137472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0137876Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0138277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0138726Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0138900Z 2025-08-26T20:39:33.0139012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0139369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0139698Z return mod(**inputs) 2025-08-26T20:39:33.0140093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0140486Z outputs = self.model( 2025-08-26T20:39:33.0140863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0141264Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0141673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0142079Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0142434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0142802Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0143209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0143655Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0144049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0144397Z return self.act(input) 2025-08-26T20:39:33.0144507Z 2025-08-26T20:39:33.0144614Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0144982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0145316Z return mod(**inputs) 2025-08-26T20:39:33.0145718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0146144Z outputs = self.model( 2025-08-26T20:39:33.0146536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0146965Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0147392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0161364Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0161836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0162377Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0162889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-26T20:39:33.0163332Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0163492Z 2025-08-26T20:39:33.0163626Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0164029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0164473Z return mod(**inputs) 2025-08-26T20:39:33.0164946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0165389Z outputs = self.model( 2025-08-26T20:39:33.0165804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0166241Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0166677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0167136Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0167516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0167895Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0168302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0168746Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0169216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0169726Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0169957Z 2025-08-26T20:39:33.0170075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0170469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0170826Z return mod(**inputs) 2025-08-26T20:39:33.0171213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0171614Z outputs = self.model( 2025-08-26T20:39:33.0171993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0172392Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0172811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0173230Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0173604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0173997Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0174415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0174852Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0175288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0175717Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0175864Z 2025-08-26T20:39:33.0175977Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0176367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0176711Z return mod(**inputs) 2025-08-26T20:39:33.0177101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0177495Z outputs = self.model( 2025-08-26T20:39:33.0177881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0178327Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0178755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0179178Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0179550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0179943Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0180374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0180832Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0181292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0181756Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0181919Z 2025-08-26T20:39:33.0182024Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0182261Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0182490Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0182713Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0182966Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0183359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0183741Z return mod(**inputs) 2025-08-26T20:39:33.0184152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0184567Z outputs = self.model( 2025-08-26T20:39:33.0184968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0185400Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0185821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0186244Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0186615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0187016Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0187447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0187889Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0188320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0188779Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0189266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0189786Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0189983Z 2025-08-26T20:39:33.0190103Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0190485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0190832Z return mod(**inputs) 2025-08-26T20:39:33.0191231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0191652Z outputs = self.model( 2025-08-26T20:39:33.0192054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0192472Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0192915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0193347Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0193743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0194143Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0194588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0195046Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0195505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0195956Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0196633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0197163Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0197425Z 2025-08-26T20:39:33.0197544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0197950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0198310Z return mod(**inputs) 2025-08-26T20:39:33.0198719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0199153Z outputs = self.model( 2025-08-26T20:39:33.0199627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0200142Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0200570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0201025Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0201417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0201826Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0202264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0202727Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0203201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0203654Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0203808Z 2025-08-26T20:39:33.0203931Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0204340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0204698Z return mod(**inputs) 2025-08-26T20:39:33.0205116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0205561Z outputs = self.model( 2025-08-26T20:39:33.0205987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0206422Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0206858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0207306Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0207699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0208109Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0208607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0209211Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0209417Z 2025-08-26T20:39:33.0209565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0209973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0210333Z return mod(**inputs) 2025-08-26T20:39:33.0210739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0211177Z outputs = self.model( 2025-08-26T20:39:33.0211595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0212039Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0212470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0212895Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0213284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0213727Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0214174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0214668Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0215119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0215509Z return self.act(input) 2025-08-26T20:39:33.0215626Z 2025-08-26T20:39:33.0215746Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0216140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0216487Z return mod(**inputs) 2025-08-26T20:39:33.0216894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0217324Z outputs = self.model( 2025-08-26T20:39:33.0217727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0218145Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0218564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0218998Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0219389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0219799Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0220245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-26T20:39:33.0220679Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0220836Z 2025-08-26T20:39:33.0220951Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0221345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0221693Z return mod(**inputs) 2025-08-26T20:39:33.0222098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0222519Z outputs = self.model( 2025-08-26T20:39:33.0222921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0223355Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0223774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0224198Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0224605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0225003Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0225448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0225892Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0226345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0226861Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0227086Z 2025-08-26T20:39:33.0227206Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0227590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0227950Z return mod(**inputs) 2025-08-26T20:39:33.0228352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0228810Z outputs = self.model( 2025-08-26T20:39:33.0229225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0229656Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0230085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0230516Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0230929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0231377Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0231810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0232267Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0232721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0233168Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0233317Z 2025-08-26T20:39:33.0233433Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0233834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0234193Z return mod(**inputs) 2025-08-26T20:39:33.0234600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0235033Z outputs = self.model( 2025-08-26T20:39:33.0235447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0235885Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0236319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0236759Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0237138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0237536Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0237974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0238430Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0238891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0239335Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0239573Z 2025-08-26T20:39:33.0239673Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0239931Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0240203Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0240435Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0240718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0241122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0241486Z return mod(**inputs) 2025-08-26T20:39:33.0241897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0242331Z outputs = self.model( 2025-08-26T20:39:33.0242744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0243184Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0243625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0244014Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0244387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0244759Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0245156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0245561Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0245958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0246397Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0246858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0247325Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0247506Z 2025-08-26T20:39:33.0247616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0247963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0248288Z return mod(**inputs) 2025-08-26T20:39:33.0248658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0249058Z outputs = self.model( 2025-08-26T20:39:33.0249434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0249851Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0250239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0250631Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0250979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0251338Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0251742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0252143Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0252539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0252935Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0253357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0253804Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0253968Z 2025-08-26T20:39:33.0254069Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0254440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0254755Z return mod(**inputs) 2025-08-26T20:39:33.0255125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0255502Z outputs = self.model( 2025-08-26T20:39:33.0255866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0256260Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0256640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0257034Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0257378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0257747Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0258132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-26T20:39:33.0258557Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:39:33.0258959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0259360Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0259499Z 2025-08-26T20:39:33.0259609Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0259975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0260335Z return mod(**inputs) 2025-08-26T20:39:33.0260711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0261105Z outputs = self.model( 2025-08-26T20:39:33.0261482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0261879Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0262277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0262685Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0263044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0263418Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0263813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0264271Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0264454Z 2025-08-26T20:39:33.0264561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0264931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0265260Z return mod(**inputs) 2025-08-26T20:39:33.0265642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0266046Z outputs = self.model( 2025-08-26T20:39:33.0266425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0266832Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0267219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0267622Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0267983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0268358Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0268798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-26T20:39:33.0269261Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0269659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0270017Z return self.act(input) 2025-08-26T20:39:33.0270127Z 2025-08-26T20:39:33.0270240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0270604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0270942Z return mod(**inputs) 2025-08-26T20:39:33.0271321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0271716Z outputs = self.model( 2025-08-26T20:39:33.0272090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-26T20:39:33.0272513Z encoder_outputs = self.encoder( 2025-08-26T20:39:33.0272907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-26T20:39:33.0273304Z layer_outputs = encoder_layer( 2025-08-26T20:39:33.0273656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0274024Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0274417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-26T20:39:33.0274851Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0274999Z 2025-08-26T20:39:33.0275105Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0275472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0275799Z return mod(**inputs) 2025-08-26T20:39:33.0276187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0276595Z outputs = self.model( 2025-08-26T20:39:33.0277014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0277434Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0277845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0278266Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0278642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0279040Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0279548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0280015Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0280484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0280990Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0281210Z 2025-08-26T20:39:33.0281337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0281721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0282077Z return mod(**inputs) 2025-08-26T20:39:33.0282472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0282891Z outputs = self.model( 2025-08-26T20:39:33.0283321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0283743Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0284182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0284605Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0284982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0285374Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0285799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0286255Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0286708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0287183Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0287329Z 2025-08-26T20:39:33.0287445Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0287861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0288212Z return mod(**inputs) 2025-08-26T20:39:33.0288605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0289016Z outputs = self.model( 2025-08-26T20:39:33.0289407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0289855Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0290278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0290683Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0291055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0291458Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0291863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0292293Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0292721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0293161Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0293324Z 2025-08-26T20:39:33.0293416Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0293655Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0293885Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0294110Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0294366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0294758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0295092Z return mod(**inputs) 2025-08-26T20:39:33.0295473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0295865Z outputs = self.model( 2025-08-26T20:39:33.0296372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0296791Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0297213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0297642Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0298013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0298405Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0298874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0299338Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0299756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0300189Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0300667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0301187Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0301385Z 2025-08-26T20:39:33.0301507Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0301888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0302234Z return mod(**inputs) 2025-08-26T20:39:33.0302613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0303047Z outputs = self.model( 2025-08-26T20:39:33.0303420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0303818Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0304236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0304663Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0305071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0305462Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0305891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0306316Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0306749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0307175Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0307620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0308093Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0308274Z 2025-08-26T20:39:33.0308389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0308776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0309125Z return mod(**inputs) 2025-08-26T20:39:33.0309516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0309941Z outputs = self.model( 2025-08-26T20:39:33.0310348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0310775Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0311192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0311614Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0311991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0312386Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0312812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0313261Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0313729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0314180Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0314330Z 2025-08-26T20:39:33.0314448Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0314836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0315176Z return mod(**inputs) 2025-08-26T20:39:33.0315570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0315994Z outputs = self.model( 2025-08-26T20:39:33.0316391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0316820Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0317345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0317816Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0318197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0318602Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0319035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0319583Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0320075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0320805Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0321036Z 2025-08-26T20:39:33.0321162Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0321561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0321934Z return mod(**inputs) 2025-08-26T20:39:33.0322333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0322758Z outputs = self.model( 2025-08-26T20:39:33.0323151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0323589Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0324014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0324440Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0324817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0325201Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0325608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0326041Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0326473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0326878Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0327017Z 2025-08-26T20:39:33.0327124Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0327490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0327819Z return mod(**inputs) 2025-08-26T20:39:33.0328205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0328614Z outputs = self.model( 2025-08-26T20:39:33.0329037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0329466Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0329911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0330336Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0330701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0331100Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0331522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0331977Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0332427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0332864Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0333044Z 2025-08-26T20:39:33.0333131Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0333367Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0333582Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0333785Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0334025Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0334400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0334752Z return mod(**inputs) 2025-08-26T20:39:33.0335143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0335593Z outputs = self.model( 2025-08-26T20:39:33.0335967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0336365Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0336762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0337189Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0337567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0337953Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0338378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0338838Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0339284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0339725Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0340197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0340720Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0340920Z 2025-08-26T20:39:33.0341305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0341687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0342040Z return mod(**inputs) 2025-08-26T20:39:33.0342440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0342862Z outputs = self.model( 2025-08-26T20:39:33.0343253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0343677Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0344129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0344565Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0344956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0345347Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0345787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0346253Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0346715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0347170Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0347652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0348147Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0348346Z 2025-08-26T20:39:33.0348460Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0348848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0349200Z return mod(**inputs) 2025-08-26T20:39:33.0349587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0350003Z outputs = self.model( 2025-08-26T20:39:33.0350405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0350847Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0351253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0351651Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0352007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0352378Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0352782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0353210Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0353662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0354094Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0354247Z 2025-08-26T20:39:33.0354365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0354751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0355091Z return mod(**inputs) 2025-08-26T20:39:33.0355491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0355911Z outputs = self.model( 2025-08-26T20:39:33.0356317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0356737Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0357159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0357582Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0357964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0358370Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0358806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0359362Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0359644Z 2025-08-26T20:39:33.0359768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0360199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0360578Z return mod(**inputs) 2025-08-26T20:39:33.0360982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0361403Z outputs = self.model( 2025-08-26T20:39:33.0361779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0362176Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0362587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0362986Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0363338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0363735Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0364138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0364573Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0364969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0365318Z return self.act(input) 2025-08-26T20:39:33.0365451Z 2025-08-26T20:39:33.0365567Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0365935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0366275Z return mod(**inputs) 2025-08-26T20:39:33.0366681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0367114Z outputs = self.model( 2025-08-26T20:39:33.0367540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0367967Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0368393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0368814Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0369197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0369592Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0370016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:33.0370452Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0370604Z 2025-08-26T20:39:33.0370727Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0371127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0371475Z return mod(**inputs) 2025-08-26T20:39:33.0371876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0372295Z outputs = self.model( 2025-08-26T20:39:33.0372696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0373121Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0373536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0373964Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0374370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0374765Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0375217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0375672Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0376119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0376632Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0376856Z 2025-08-26T20:39:33.0376975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0377356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0377708Z return mod(**inputs) 2025-08-26T20:39:33.0378106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0378549Z outputs = self.model( 2025-08-26T20:39:33.0378951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0379369Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0379786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0380210Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0380591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0381019Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0381448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0381908Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0382368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0382815Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0382962Z 2025-08-26T20:39:33.0383073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0383462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0383816Z return mod(**inputs) 2025-08-26T20:39:33.0384223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0384657Z outputs = self.model( 2025-08-26T20:39:33.0385058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0385490Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0385908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0386305Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0386656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0387028Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0387453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0387900Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0388341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0388769Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0388929Z 2025-08-26T20:39:33.0389019Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0389252Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0389506Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0389728Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0389995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0390389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0390743Z return mod(**inputs) 2025-08-26T20:39:33.0391138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0391535Z outputs = self.model( 2025-08-26T20:39:33.0391914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0392330Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0392758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0393187Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0393561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0393985Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0394409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0394857Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0395290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0395760Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0396387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0396939Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0397143Z 2025-08-26T20:39:33.0397272Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0397671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0398040Z return mod(**inputs) 2025-08-26T20:39:33.0398440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0398863Z outputs = self.model( 2025-08-26T20:39:33.0399264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0399760Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0400194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0400630Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0401017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0401413Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0401856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0402306Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0402766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0403260Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0403730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0404222Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0404404Z 2025-08-26T20:39:33.0404516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0404961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0405330Z return mod(**inputs) 2025-08-26T20:39:33.0405747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0406164Z outputs = self.model( 2025-08-26T20:39:33.0406557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0406978Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0407385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0407817Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0408170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0408538Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0408938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0409391Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0409814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0410226Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0410366Z 2025-08-26T20:39:33.0410478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0410845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0411200Z return mod(**inputs) 2025-08-26T20:39:33.0411571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0411971Z outputs = self.model( 2025-08-26T20:39:33.0412350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0412746Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0413165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0413587Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0413965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0414364Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0414796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0415230Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0415665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0416145Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0416357Z 2025-08-26T20:39:33.0416470Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0416830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0417158Z return mod(**inputs) 2025-08-26T20:39:33.0417531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0417929Z outputs = self.model( 2025-08-26T20:39:33.0418298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0418698Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0419094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0419494Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0419890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0420301Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0420711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0421168Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0421623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0422022Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0422161Z 2025-08-26T20:39:33.0422266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0422627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0422955Z return mod(**inputs) 2025-08-26T20:39:33.0423327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0423736Z outputs = self.model( 2025-08-26T20:39:33.0424110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0424516Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0424901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0425291Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0425638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0426024Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0426424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0426853Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0427283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0427689Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0427840Z 2025-08-26T20:39:33.0427922Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0428142Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0428356Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0428558Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0428795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0429163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0429491Z return mod(**inputs) 2025-08-26T20:39:33.0429857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0430250Z outputs = self.model( 2025-08-26T20:39:33.0430625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0431039Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0431462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0431852Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0432206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0432585Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0432975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0433398Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0433836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0434286Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0434777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0435291Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0435487Z 2025-08-26T20:39:33.0435606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0435986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0436338Z return mod(**inputs) 2025-08-26T20:39:33.0436732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0437149Z outputs = self.model( 2025-08-26T20:39:33.0437538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0437980Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0438399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0438824Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0439201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0439681Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0440148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0440646Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0441122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0441555Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0441992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0442460Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0442642Z 2025-08-26T20:39:33.0442761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0443156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0443510Z return mod(**inputs) 2025-08-26T20:39:33.0443912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0444337Z outputs = self.model( 2025-08-26T20:39:33.0444743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0445176Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0445597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0446029Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0446413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0446809Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0447245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0447705Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0448174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0448625Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0448760Z 2025-08-26T20:39:33.0448871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0449241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0449567Z return mod(**inputs) 2025-08-26T20:39:33.0449956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0450346Z outputs = self.model( 2025-08-26T20:39:33.0450720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0451114Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0451509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0451910Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0452264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0452640Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0453034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0453489Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0453665Z 2025-08-26T20:39:33.0453768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0454121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0454433Z return mod(**inputs) 2025-08-26T20:39:33.0454797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0455212Z outputs = self.model( 2025-08-26T20:39:33.0455579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0455967Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0456349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0456762Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0457109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0457468Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0457855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0458289Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0458673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0459014Z return self.act(input) 2025-08-26T20:39:33.0459121Z 2025-08-26T20:39:33.0459231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0459580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0459906Z return mod(**inputs) 2025-08-26T20:39:33.0460283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0460676Z outputs = self.model( 2025-08-26T20:39:33.0461050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0461448Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0461841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0462240Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0462599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0462954Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0463370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:33.0463785Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0463923Z 2025-08-26T20:39:33.0464034Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0464398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0464718Z return mod(**inputs) 2025-08-26T20:39:33.0465090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0465485Z outputs = self.model( 2025-08-26T20:39:33.0465877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0466324Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0466712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0467133Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0467497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0467888Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0468295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0468723Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0469147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0469672Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0469891Z 2025-08-26T20:39:33.0470018Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0470377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0470710Z return mod(**inputs) 2025-08-26T20:39:33.0471081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0471489Z outputs = self.model( 2025-08-26T20:39:33.0471898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0472323Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0472750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0473183Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0473568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0473969Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0474427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0474886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0475338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0475790Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0475939Z 2025-08-26T20:39:33.0476054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0476448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0476808Z return mod(**inputs) 2025-08-26T20:39:33.0477211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0477642Z outputs = self.model( 2025-08-26T20:39:33.0478056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0478497Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0478945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0479377Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0480071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0480477Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0480915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0481398Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0481840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0482281Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0482475Z 2025-08-26T20:39:33.0482565Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0482806Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0483041Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0483267Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0483529Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0483931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0484295Z return mod(**inputs) 2025-08-26T20:39:33.0484701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0485165Z outputs = self.model( 2025-08-26T20:39:33.0485572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0486005Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0486431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0486860Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0487250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0487645Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0488080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0488542Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0488992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0489453Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0489947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0490471Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0490672Z 2025-08-26T20:39:33.0490793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0491183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0491541Z return mod(**inputs) 2025-08-26T20:39:33.0491944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0492375Z outputs = self.model( 2025-08-26T20:39:33.0492773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0493205Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0493646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0494090Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0494497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0494889Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0495322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0495785Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0496382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0497933Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0498522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0499069Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0499467Z 2025-08-26T20:39:33.0499588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0500004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0500368Z return mod(**inputs) 2025-08-26T20:39:33.0500782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0501219Z outputs = self.model( 2025-08-26T20:39:33.0501660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0502276Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0502727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0503175Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0503577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0503994Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0504452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0504904Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0505374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0505818Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0505978Z 2025-08-26T20:39:33.0506108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0506519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0506875Z return mod(**inputs) 2025-08-26T20:39:33.0507293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0507735Z outputs = self.model( 2025-08-26T20:39:33.0508149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0508586Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0509022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0509449Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0509832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0510229Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0510677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0511154Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0511665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0512222Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0512451Z 2025-08-26T20:39:33.0512589Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0512978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0513335Z return mod(**inputs) 2025-08-26T20:39:33.0513786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0514230Z outputs = self.model( 2025-08-26T20:39:33.0514640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0515090Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0515568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0516036Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0516436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0516840Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0517287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0517772Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0518266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0518718Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0518868Z 2025-08-26T20:39:33.0518982Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0519384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0520038Z return mod(**inputs) 2025-08-26T20:39:33.0520450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0520873Z outputs = self.model( 2025-08-26T20:39:33.0521285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0521720Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0522154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0522588Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0522969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0523373Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0523813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0524281Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0524738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0525184Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0525348Z 2025-08-26T20:39:33.0525439Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0525680Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0525916Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0526137Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0526395Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0526791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0527175Z return mod(**inputs) 2025-08-26T20:39:33.0527603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0528039Z outputs = self.model( 2025-08-26T20:39:33.0528450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0528900Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0529328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0529759Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0530148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0530551Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0530978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0531461Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0531909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0532357Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0532842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0533364Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0533585Z 2025-08-26T20:39:33.0533698Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0534086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0534434Z return mod(**inputs) 2025-08-26T20:39:33.0534829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0535249Z outputs = self.model( 2025-08-26T20:39:33.0535640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0536062Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0536473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0536893Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0537269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0537656Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0538081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0538536Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0538988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0539429Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0539909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0540398Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0540572Z 2025-08-26T20:39:33.0540693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0541085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0541420Z return mod(**inputs) 2025-08-26T20:39:33.0541813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0542231Z outputs = self.model( 2025-08-26T20:39:33.0542645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0543091Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0543504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0543930Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0544311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0544713Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0545142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0545608Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0546069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0546537Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0546687Z 2025-08-26T20:39:33.0546809Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0547190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0547545Z return mod(**inputs) 2025-08-26T20:39:33.0547940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0548361Z outputs = self.model( 2025-08-26T20:39:33.0548774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0549192Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0549605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0550029Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0550267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0550363Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0550638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0550774Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0550778Z 2025-08-26T20:39:33.0550888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0551104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0551183Z return mod(**inputs) 2025-08-26T20:39:33.0551459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0551537Z outputs = self.model( 2025-08-26T20:39:33.0551814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0551895Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0552181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0552260Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0552503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0552589Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0552875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0553006Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0553238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0553342Z return self.act(input) 2025-08-26T20:39:33.0553349Z 2025-08-26T20:39:33.0553465Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0553720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0553796Z return mod(**inputs) 2025-08-26T20:39:33.0554081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0554165Z outputs = self.model( 2025-08-26T20:39:33.0554462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0554552Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0554896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0554974Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0555227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0555335Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0555622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:33.0555712Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0555716Z 2025-08-26T20:39:33.0555836Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0556060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0556151Z return mod(**inputs) 2025-08-26T20:39:33.0556467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0556540Z outputs = self.model( 2025-08-26T20:39:33.0556831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0556913Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0557199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0557286Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0557527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0557622Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0557906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0558019Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0558310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0558478Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0558483Z 2025-08-26T20:39:33.0558604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0558824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0558903Z return mod(**inputs) 2025-08-26T20:39:33.0559187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0559261Z outputs = self.model( 2025-08-26T20:39:33.0559748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0559846Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0560139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0560219Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0560488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0560606Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0560890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0561010Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0561301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0561400Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0561404Z 2025-08-26T20:39:33.0561520Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0561737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0561818Z return mod(**inputs) 2025-08-26T20:39:33.0562102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0562206Z outputs = self.model( 2025-08-26T20:39:33.0562493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0562575Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0562868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0562945Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0563216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0563303Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0563597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0563707Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0563996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0564098Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0564102Z 2025-08-26T20:39:33.0564190Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0564285Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0564370Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0564454Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0564577Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0564798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0564878Z return mod(**inputs) 2025-08-26T20:39:33.0565165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0565243Z outputs = self.model( 2025-08-26T20:39:33.0565545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0565625Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0565918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0565997Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0566240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0566334Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0566619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0566733Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0567043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0567164Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0567512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0567660Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0567665Z 2025-08-26T20:39:33.0567789Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0568007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0568092Z return mod(**inputs) 2025-08-26T20:39:33.0568378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0568453Z outputs = self.model( 2025-08-26T20:39:33.0568766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0568868Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0569165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0569245Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0569498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0569585Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0569871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0570967Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0571243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0571353Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0571675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0571802Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0571807Z 2025-08-26T20:39:33.0571930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0572150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0572231Z return mod(**inputs) 2025-08-26T20:39:33.0572517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0572601Z outputs = self.model( 2025-08-26T20:39:33.0572887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0572968Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0573265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0573355Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0573601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0573685Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0573963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0574072Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0574344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0574438Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0574442Z 2025-08-26T20:39:33.0574550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0574780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0574862Z return mod(**inputs) 2025-08-26T20:39:33.0575164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0575248Z outputs = self.model( 2025-08-26T20:39:33.0575521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0575604Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0575883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0575965Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0576215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0576301Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0576591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0576732Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0577018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0577194Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0577198Z 2025-08-26T20:39:33.0577317Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0577564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0577637Z return mod(**inputs) 2025-08-26T20:39:33.0577934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0578007Z outputs = self.model( 2025-08-26T20:39:33.0578284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0578374Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0578652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0578736Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0578973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0579058Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0579348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0579464Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0579747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0579835Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0579839Z 2025-08-26T20:39:33.0579959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0580171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0580243Z return mod(**inputs) 2025-08-26T20:39:33.0580537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0580611Z outputs = self.model( 2025-08-26T20:39:33.0580908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0580988Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0581271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0581375Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0581640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0581737Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0582019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0582134Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0582422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0582518Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0582522Z 2025-08-26T20:39:33.0582618Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0582706Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0582797Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0582880Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0582995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0583240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0583314Z return mod(**inputs) 2025-08-26T20:39:33.0583607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0583683Z outputs = self.model( 2025-08-26T20:39:33.0583986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0584096Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0584387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0584473Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0584720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0584809Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0585104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0585222Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0585514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0585621Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0585947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0586101Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0586106Z 2025-08-26T20:39:33.0586218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0586445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0586520Z return mod(**inputs) 2025-08-26T20:39:33.0586815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0586892Z outputs = self.model( 2025-08-26T20:39:33.0587177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0587264Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0587566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0587653Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0587898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0587985Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0588295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0588432Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0588726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0588833Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0589166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0589288Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0589292Z 2025-08-26T20:39:33.0589407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0589634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0589710Z return mod(**inputs) 2025-08-26T20:39:33.0590020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0590115Z outputs = self.model( 2025-08-26T20:39:33.0590400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0590488Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0590773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0590859Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0591125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0591220Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0591504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0591623Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0591918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0592008Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0592012Z 2025-08-26T20:39:33.0592133Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0592353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0592427Z return mod(**inputs) 2025-08-26T20:39:33.0592721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0592794Z outputs = self.model( 2025-08-26T20:39:33.0593083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0593166Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0593457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0593545Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0593790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0593886Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0594174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0594314Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0594318Z 2025-08-26T20:39:33.0594431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0594647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0594727Z return mod(**inputs) 2025-08-26T20:39:33.0595030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0595131Z outputs = self.model( 2025-08-26T20:39:33.0595419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0595500Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0595788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0595867Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0596120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0596511Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0596940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0597077Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0597384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0597470Z return self.act(input) 2025-08-26T20:39:33.0597474Z 2025-08-26T20:39:33.0597587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0597817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0597890Z return mod(**inputs) 2025-08-26T20:39:33.0598174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0598303Z outputs = self.model( 2025-08-26T20:39:33.0598593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0598682Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0598971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0599054Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0599311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0599398Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0599760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:33.0599857Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0599865Z 2025-08-26T20:39:33.0599984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0600205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0600278Z return mod(**inputs) 2025-08-26T20:39:33.0600574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0600651Z outputs = self.model( 2025-08-26T20:39:33.0600943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0601023Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0601309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0601395Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0601638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0601735Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0602018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0602177Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0602499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0602669Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0602673Z 2025-08-26T20:39:33.0602797Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0603015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0603095Z return mod(**inputs) 2025-08-26T20:39:33.0603380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0603457Z outputs = self.model( 2025-08-26T20:39:33.0603757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0603834Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0604119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0604220Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0604454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0604546Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0604822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0604936Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0605233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0605323Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0605327Z 2025-08-26T20:39:33.0605435Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0605648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0605728Z return mod(**inputs) 2025-08-26T20:39:33.0606003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0606080Z outputs = self.model( 2025-08-26T20:39:33.0606359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0606439Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0606727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0606803Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0607045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0607131Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0607414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0607518Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0607791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0607889Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0607893Z 2025-08-26T20:39:33.0607978Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0608073Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0608155Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0608236Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0608353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0608566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0608659Z return mod(**inputs) 2025-08-26T20:39:33.0608957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0609033Z outputs = self.model( 2025-08-26T20:39:33.0609315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0609394Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0609676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0609756Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0609993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0610085Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0610360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0610494Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0610771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0610882Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0611202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0611337Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0611356Z 2025-08-26T20:39:33.0611468Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0611667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0611745Z return mod(**inputs) 2025-08-26T20:39:33.0612022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0612098Z outputs = self.model( 2025-08-26T20:39:33.0612382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0612460Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0612742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0612819Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0613061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0613147Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0613421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0613533Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0613809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0613920Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0614230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0614347Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0614351Z 2025-08-26T20:39:33.0614469Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0614682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0614759Z return mod(**inputs) 2025-08-26T20:39:33.0615046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0615126Z outputs = self.model( 2025-08-26T20:39:33.0615429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0615526Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0615814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0615891Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0616132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0616215Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0616507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0616620Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0616904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0617017Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0617021Z 2025-08-26T20:39:33.0617132Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0617350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0617419Z return mod(**inputs) 2025-08-26T20:39:33.0617692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0617773Z outputs = self.model( 2025-08-26T20:39:33.0618108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0618207Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0618475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0618551Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0618789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0618869Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0619145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0619254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0619544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0619717Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0619721Z 2025-08-26T20:39:33.0619832Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0620054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0620127Z return mod(**inputs) 2025-08-26T20:39:33.0620430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0620506Z outputs = self.model( 2025-08-26T20:39:33.0620787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0620875Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0621171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0621257Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0621499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0621583Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0621908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0622026Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0622329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0622416Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0622420Z 2025-08-26T20:39:33.0622537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0622747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0622816Z return mod(**inputs) 2025-08-26T20:39:33.0623109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0623181Z outputs = self.model( 2025-08-26T20:39:33.0623467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0623542Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0623824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0623906Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0624128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0624214Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0624470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0624606Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0624893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0624987Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0624991Z 2025-08-26T20:39:33.0625089Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0625175Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0625267Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0625351Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0625461Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0625680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0625752Z return mod(**inputs) 2025-08-26T20:39:33.0626039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0626114Z outputs = self.model( 2025-08-26T20:39:33.0626391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0626478Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0626756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0626845Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0627084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0627171Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0627457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0627569Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0627852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0627957Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0628366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0628587Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0628596Z 2025-08-26T20:39:33.0628730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0628955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0629028Z return mod(**inputs) 2025-08-26T20:39:33.0629317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0629391Z outputs = self.model( 2025-08-26T20:39:33.0629667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0629758Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0630035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0630124Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0630363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0630474Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0630761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0630876Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0631159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0631284Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0631604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0631719Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0631723Z 2025-08-26T20:39:33.0631838Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0632067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0632140Z return mod(**inputs) 2025-08-26T20:39:33.0632441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0632515Z outputs = self.model( 2025-08-26T20:39:33.0632792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0632879Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0633154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0633240Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0633480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0633574Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0633860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0633977Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0634268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0634358Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0634362Z 2025-08-26T20:39:33.0634481Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0634700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0634772Z return mod(**inputs) 2025-08-26T20:39:33.0635083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0635175Z outputs = self.model( 2025-08-26T20:39:33.0635491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0635571Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0635855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0635941Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0636183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0636281Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0636565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0636701Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0636706Z 2025-08-26T20:39:33.0636818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0637055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0637134Z return mod(**inputs) 2025-08-26T20:39:33.0637417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0637499Z outputs = self.model( 2025-08-26T20:39:33.0637781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0637860Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0638194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0638272Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0638524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0638613Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0638910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0639042Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0639277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0639361Z return self.act(input) 2025-08-26T20:39:33.0639365Z 2025-08-26T20:39:33.0639530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0639765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0639839Z return mod(**inputs) 2025-08-26T20:39:33.0640125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0640209Z outputs = self.model( 2025-08-26T20:39:33.0640493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0640585Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0640875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0640950Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0641192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0641276Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0641560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:33.0641648Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0641652Z 2025-08-26T20:39:33.0641768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0642008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0642084Z return mod(**inputs) 2025-08-26T20:39:33.0642391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0642465Z outputs = self.model( 2025-08-26T20:39:33.0642749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0642828Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0643107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0643191Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0643424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0643520Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0643815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0643932Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0644206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0644368Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0644372Z 2025-08-26T20:39:33.0644489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0644720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0644801Z return mod(**inputs) 2025-08-26T20:39:33.0645079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0645151Z outputs = self.model( 2025-08-26T20:39:33.0645434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0645510Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0645777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0645849Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0646070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0646157Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0646416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0646521Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0646779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0646868Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0646872Z 2025-08-26T20:39:33.0646976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0647174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0647248Z return mod(**inputs) 2025-08-26T20:39:33.0647510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0647587Z outputs = self.model( 2025-08-26T20:39:33.0647847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0647920Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0648193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0648282Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0648533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0648614Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0648879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0648977Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0649238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0649336Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0649340Z 2025-08-26T20:39:33.0649420Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0649508Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0649586Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0649665Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0649777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0650005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0650079Z return mod(**inputs) 2025-08-26T20:39:33.0650340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0650410Z outputs = self.model( 2025-08-26T20:39:33.0650677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0650769Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0651039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0651116Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0651353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0651447Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0651727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0651832Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0652091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0652195Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0652492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0652625Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0652629Z 2025-08-26T20:39:33.0652740Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0652940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0653014Z return mod(**inputs) 2025-08-26T20:39:33.0653280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0653349Z outputs = self.model( 2025-08-26T20:39:33.0653619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0653693Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0653958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0654033Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0654262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0654346Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0654639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0654769Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0655045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0655147Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0655439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0655550Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0655553Z 2025-08-26T20:39:33.0655667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0655867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0655939Z return mod(**inputs) 2025-08-26T20:39:33.0656200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0656296Z outputs = self.model( 2025-08-26T20:39:33.0656556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0656630Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0656898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0656971Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0657218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0657299Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0657556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-26T20:39:33.0657661Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:39:33.0657922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0658014Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0658017Z 2025-08-26T20:39:33.0658119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0658324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0658390Z return mod(**inputs) 2025-08-26T20:39:33.0658654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0658731Z outputs = self.model( 2025-08-26T20:39:33.0658990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0659073Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0659345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0659422Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0659666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0659749Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0660030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0660147Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0660420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-26T20:39:33.0660589Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:39:33.0660593Z 2025-08-26T20:39:33.0660719Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0660964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0661036Z return mod(**inputs) 2025-08-26T20:39:33.0661320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0661393Z outputs = self.model( 2025-08-26T20:39:33.0661670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0661758Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0662033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0662116Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0662354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0662460Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0662746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0662860Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0663141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-26T20:39:33.0663227Z key_states = self.k_proj(current_states) 2025-08-26T20:39:33.0663231Z 2025-08-26T20:39:33.0663366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0663578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0663647Z return mod(**inputs) 2025-08-26T20:39:33.0663930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0664005Z outputs = self.model( 2025-08-26T20:39:33.0664292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0664368Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0664643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0664728Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0664963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0665057Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0665330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0665443Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0665723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-26T20:39:33.0665816Z value_states = self.v_proj(current_states) 2025-08-26T20:39:33.0665822Z 2025-08-26T20:39:33.0665915Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0666001Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0666090Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0666170Z cudagraph partition due to non gpu ops 2025-08-26T20:39:33.0666278Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0666496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0666567Z return mod(**inputs) 2025-08-26T20:39:33.0666851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0666923Z outputs = self.model( 2025-08-26T20:39:33.0667218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0667308Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0667602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0667688Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0667927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0668013Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0668299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0668418Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0668701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0668807Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0669152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:39:33.0669301Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:39:33.0669305Z 2025-08-26T20:39:33.0669416Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0669637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0669708Z return mod(**inputs) 2025-08-26T20:39:33.0670010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0670083Z outputs = self.model( 2025-08-26T20:39:33.0670355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0670441Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0670721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0670807Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0671042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0671125Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0671406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0671524Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0671807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-26T20:39:33.0671910Z attn_output, attn_weights = attention_interface( 2025-08-26T20:39:33.0672232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:39:33.0672351Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:39:33.0672355Z 2025-08-26T20:39:33.0672464Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0672684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0672754Z return mod(**inputs) 2025-08-26T20:39:33.0673036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0673111Z outputs = self.model( 2025-08-26T20:39:33.0673385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0673471Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0673769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0673859Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0674120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0674214Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0674489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-26T20:39:33.0674606Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:39:33.0674897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-26T20:39:33.0674987Z attn_output = self.out_proj(attn_output) 2025-08-26T20:39:33.0674991Z 2025-08-26T20:39:33.0675110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0675328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0675399Z return mod(**inputs) 2025-08-26T20:39:33.0675755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0675830Z outputs = self.model( 2025-08-26T20:39:33.0676120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0676200Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0676481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0676585Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0676831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0676927Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0677212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0677352Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0677358Z 2025-08-26T20:39:33.0677471Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0677689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0677770Z return mod(**inputs) 2025-08-26T20:39:33.0678057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0678142Z outputs = self.model( 2025-08-26T20:39:33.0678428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0678508Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0678807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0678889Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0679145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0679233Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0679615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-26T20:39:33.0679752Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:39:33.0679988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:39:33.0680077Z return self.act(input) 2025-08-26T20:39:33.0680082Z 2025-08-26T20:39:33.0680197Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0680425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0680520Z return mod(**inputs) 2025-08-26T20:39:33.0680830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-26T20:39:33.0680918Z outputs = self.model( 2025-08-26T20:39:33.0681206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-26T20:39:33.0681294Z decoder_outputs = self.decoder( 2025-08-26T20:39:33.0681581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-26T20:39:33.0681662Z layer_outputs = decoder_layer( 2025-08-26T20:39:33.0681916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:39:33.0682004Z return super().__call__(*args, **kwargs) 2025-08-26T20:39:33.0682297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-26T20:39:33.0682408Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:39:33.0682412Z 2025-08-26T20:39:33.0682531Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0682749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0682822Z return mod(**inputs) 2025-08-26T20:39:33.0683114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1377, in forward 2025-08-26T20:39:33.0683202Z lm_logits = self.lm_head(outputs[0]) 2025-08-26T20:39:33.0683225Z 2025-08-26T20:39:33.0683343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:39:33.0683560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:39:33.0683632Z return mod(**inputs) 2025-08-26T20:39:33.0683926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1383, in forward 2025-08-26T20:39:33.0684115Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:39:33.0684121Z 2025-08-26T20:39:42.8375028Z Compilation time (from dynamo_timed): 18.270969405 2025-08-26T20:39:42.8659876Z pass 2025-08-26T20:39:42.8661413Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:42.8662296Z TIMING: _recursive_pre_grad_passes:0.00961 _recursive_joint_graph_passes:0.48769 _recursive_post_grad_passes:0.10897 async_compile.wait:0.83572 code_gen:9.11386 inductor_compile:10.89507 backend_compile:15.15069 gc:0.00187 entire_frame_compile:18.27097 total_wall_time:18.27097 2025-08-26T20:39:42.8663321Z STATS: call_* op count: 517 | FakeTensorMode.__torch_dispatch__:17508 | FakeTensor.__torch_dispatch__:5831 | ProxyTorchDispatchMode.__torch_dispatch__:6406 2025-08-26T20:39:42.8663950Z Dynamo produced 1 graphs covering 517 ops with 0 graph breaks (0 unique) 2025-08-26T20:39:48.5139526Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:39:48.5140572Z from pkg_resources import resource_filename 2025-08-26T20:39:49.2871804Z 2025-08-26T20:39:52.9897288Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:39:52.9897800Z loading model: 0it [00:03, ?it/s] 2025-08-26T20:39:52.9911334Z cpu eval PegasusForCausalLM 2025-08-26T20:39:53.3895161Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:53.5546726Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:39:53.7116977Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:40:01.6628761Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6629084Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6629717Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6630086Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6630869Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6631221Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6631460Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6631691Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6631928Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6632187Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6632412Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6632647Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6632932Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6633389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6634291Z return mod(**inputs) 2025-08-26T20:40:01.6634918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6635477Z outputs = self.model.decoder( 2025-08-26T20:40:01.6635982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6636462Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6636869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6637389Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6637976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6638486Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6639060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6639838Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6640164Z 2025-08-26T20:40:01.6640310Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6640719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6641126Z return mod(**inputs) 2025-08-26T20:40:01.6641541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6641979Z outputs = self.model.decoder( 2025-08-26T20:40:01.6642407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6642829Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6643219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6643723Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6644330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6644798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6645253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.6645700Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.6645861Z 2025-08-26T20:40:01.6645978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6646375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6646729Z return mod(**inputs) 2025-08-26T20:40:01.6647225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6647685Z outputs = self.model.decoder( 2025-08-26T20:40:01.6648139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6648646Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6649158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6649649Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6650170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6650778Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6651458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.6652011Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.6652300Z 2025-08-26T20:40:01.6652403Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6652741Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6653067Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6653392Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6653750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6654160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6654536Z return mod(**inputs) 2025-08-26T20:40:01.6654990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6655437Z outputs = self.model.decoder( 2025-08-26T20:40:01.6655931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6656374Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6656769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6657259Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6657734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6658341Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6658807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6659274Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6659765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.6660309Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.6660524Z 2025-08-26T20:40:01.6660644Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6661049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6661420Z return mod(**inputs) 2025-08-26T20:40:01.6661873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6662520Z outputs = self.model.decoder( 2025-08-26T20:40:01.6663066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6663595Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6663984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6664389Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6664878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6665356Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6665838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6666303Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6666797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.6667311Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.6667496Z 2025-08-26T20:40:01.6667619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6668021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6668395Z return mod(**inputs) 2025-08-26T20:40:01.6668817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6670157Z outputs = self.model.decoder( 2025-08-26T20:40:01.6670582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6671009Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6671380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6671774Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6672207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6672698Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6673148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.6673594Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.6673754Z 2025-08-26T20:40:01.6673871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6674271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6674631Z return mod(**inputs) 2025-08-26T20:40:01.6675036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6675481Z outputs = self.model.decoder( 2025-08-26T20:40:01.6675919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6676371Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6676765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6677180Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6677621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6678124Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6678320Z 2025-08-26T20:40:01.6678442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6678838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6679203Z return mod(**inputs) 2025-08-26T20:40:01.6680017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6680504Z outputs = self.model.decoder( 2025-08-26T20:40:01.6680945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6681399Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6681831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6682232Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6682696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6683176Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6683593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.6683966Z return self.act(input) 2025-08-26T20:40:01.6684094Z 2025-08-26T20:40:01.6684211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6684608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6684960Z return mod(**inputs) 2025-08-26T20:40:01.6685366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6685825Z outputs = self.model.decoder( 2025-08-26T20:40:01.6686247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6686678Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6687049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6687438Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6687870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.6688332Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.6688522Z 2025-08-26T20:40:01.6688644Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6689028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6689381Z return mod(**inputs) 2025-08-26T20:40:01.6689786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6690219Z outputs = self.model.decoder( 2025-08-26T20:40:01.6690637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6691078Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6691467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6691890Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6692326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6692779Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6693233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6693749Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6693968Z 2025-08-26T20:40:01.6694088Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6694474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6694819Z return mod(**inputs) 2025-08-26T20:40:01.6695219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6695646Z outputs = self.model.decoder( 2025-08-26T20:40:01.6696066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6696811Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6697576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6697975Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6698450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6698909Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6699363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.6699809Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.6699964Z 2025-08-26T20:40:01.6700083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6700468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6700819Z return mod(**inputs) 2025-08-26T20:40:01.6701218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6701694Z outputs = self.model.decoder( 2025-08-26T20:40:01.6702120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6702551Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6702932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6703341Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6703801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6704304Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6704781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.6705208Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.6705367Z 2025-08-26T20:40:01.6705456Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6705693Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6705925Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6706146Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6706392Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6706783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6707136Z return mod(**inputs) 2025-08-26T20:40:01.6707540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6707969Z outputs = self.model.decoder( 2025-08-26T20:40:01.6708391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6708817Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6709195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6709587Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6710008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6710474Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6710942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6711391Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6711865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.6712383Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.6712590Z 2025-08-26T20:40:01.6712729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6713119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6713501Z return mod(**inputs) 2025-08-26T20:40:01.6713898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6714328Z outputs = self.model.decoder( 2025-08-26T20:40:01.6714751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6715180Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6715570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6715962Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6716418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6716881Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6717355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6717815Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6718286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.6718784Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.6718965Z 2025-08-26T20:40:01.6719103Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6719556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6719927Z return mod(**inputs) 2025-08-26T20:40:01.6720353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6720802Z outputs = self.model.decoder( 2025-08-26T20:40:01.6721243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6721697Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6722075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6722475Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6722919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6723396Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6723877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.6724321Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.6724502Z 2025-08-26T20:40:01.6724618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6725018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6725383Z return mod(**inputs) 2025-08-26T20:40:01.6725792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6726242Z outputs = self.model.decoder( 2025-08-26T20:40:01.6726682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6727129Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6727521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6727926Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6728420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6728914Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6729124Z 2025-08-26T20:40:01.6729251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6729659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6729981Z return mod(**inputs) 2025-08-26T20:40:01.6730361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6730791Z outputs = self.model.decoder( 2025-08-26T20:40:01.6731214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6731609Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6731968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6732364Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6732794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6733267Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6733677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.6734044Z return self.act(input) 2025-08-26T20:40:01.6734180Z 2025-08-26T20:40:01.6734285Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6734676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6735003Z return mod(**inputs) 2025-08-26T20:40:01.6735374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6735778Z outputs = self.model.decoder( 2025-08-26T20:40:01.6736177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6736582Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6736953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6737350Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6737780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.6738217Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.6738365Z 2025-08-26T20:40:01.6738486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6738863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6739213Z return mod(**inputs) 2025-08-26T20:40:01.6739618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6740051Z outputs = self.model.decoder( 2025-08-26T20:40:01.6740473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6740892Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6741268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6741656Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6742092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6742538Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6742994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6743539Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6743763Z 2025-08-26T20:40:01.6743899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6744295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6744616Z return mod(**inputs) 2025-08-26T20:40:01.6744993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6745398Z outputs = self.model.decoder( 2025-08-26T20:40:01.6745794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6746257Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6746610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6747004Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6747476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6747913Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6748392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.6748848Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.6749003Z 2025-08-26T20:40:01.6749114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6749522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6749868Z return mod(**inputs) 2025-08-26T20:40:01.6750261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6750695Z outputs = self.model.decoder( 2025-08-26T20:40:01.6751118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6751547Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6751919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6752303Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6752771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6753227Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6753705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.6754148Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.6754303Z 2025-08-26T20:40:01.6754394Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6754633Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6754865Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6755090Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6755336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6755729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6756080Z return mod(**inputs) 2025-08-26T20:40:01.6756487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6756923Z outputs = self.model.decoder( 2025-08-26T20:40:01.6757357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6757803Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6758226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6758619Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6759057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6759644Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6760192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6760668Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6761184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.6761705Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.6761915Z 2025-08-26T20:40:01.6762032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6762426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6762813Z return mod(**inputs) 2025-08-26T20:40:01.6763212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6763649Z outputs = self.model.decoder( 2025-08-26T20:40:01.6764070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6764503Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6764877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6765284Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6765719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6766190Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6766644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6767099Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6767565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.6768055Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.6768236Z 2025-08-26T20:40:01.6768349Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6768739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6769089Z return mod(**inputs) 2025-08-26T20:40:01.6769488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6769929Z outputs = self.model.decoder( 2025-08-26T20:40:01.6770367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6770798Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6771164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6771562Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6772036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6772509Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6772975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.6773413Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.6773583Z 2025-08-26T20:40:01.6773744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6774135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6774523Z return mod(**inputs) 2025-08-26T20:40:01.6774930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6775372Z outputs = self.model.decoder( 2025-08-26T20:40:01.6775790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6776192Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6776553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6776937Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6777371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6777850Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6778060Z 2025-08-26T20:40:01.6778175Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6778539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6778863Z return mod(**inputs) 2025-08-26T20:40:01.6779246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6779674Z outputs = self.model.decoder( 2025-08-26T20:40:01.6780123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6780569Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6780940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6781327Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6781740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6782192Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6782585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.6782945Z return self.act(input) 2025-08-26T20:40:01.6783066Z 2025-08-26T20:40:01.6783171Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6783538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6783870Z return mod(**inputs) 2025-08-26T20:40:01.6784243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6784657Z outputs = self.model.decoder( 2025-08-26T20:40:01.6785052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6785464Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6785815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6786188Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6786599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.6787016Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.6787161Z 2025-08-26T20:40:01.6787271Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6787631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6787964Z return mod(**inputs) 2025-08-26T20:40:01.6788366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6788772Z outputs = self.model.decoder( 2025-08-26T20:40:01.6789184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6789581Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6789934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6790303Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6790709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:01.6791112Z hidden_states = residual + hidden_states 2025-08-26T20:40:01.6791258Z 2025-08-26T20:40:01.6791362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6791727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6792076Z return mod(**inputs) 2025-08-26T20:40:01.6792476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6792897Z outputs = self.model.decoder( 2025-08-26T20:40:01.6793317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6793745Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6794119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6794530Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6794971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6795428Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6795882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6796564Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6796791Z 2025-08-26T20:40:01.6796913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6797305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6797667Z return mod(**inputs) 2025-08-26T20:40:01.6798102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6798543Z outputs = self.model.decoder( 2025-08-26T20:40:01.6798968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6799422Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6799902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6800317Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6800769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6801231Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6801684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.6802120Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.6802268Z 2025-08-26T20:40:01.6802389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6802778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6803118Z return mod(**inputs) 2025-08-26T20:40:01.6803594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6804030Z outputs = self.model.decoder( 2025-08-26T20:40:01.6804479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6804903Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6805278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6805670Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6806107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6806539Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6806995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.6807448Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.6807630Z 2025-08-26T20:40:01.6807714Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6807937Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6808147Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6808359Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6808601Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6808965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6809296Z return mod(**inputs) 2025-08-26T20:40:01.6809674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6810115Z outputs = self.model.decoder( 2025-08-26T20:40:01.6810521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6810915Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6811254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6811614Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6812018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6812451Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6812876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6813295Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6813747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.6814240Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.6814421Z 2025-08-26T20:40:01.6814533Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6814892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6815204Z return mod(**inputs) 2025-08-26T20:40:01.6815573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6815967Z outputs = self.model.decoder( 2025-08-26T20:40:01.6816351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6816745Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6817083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6817449Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6817877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6818320Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6818751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6819183Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6819634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.6820124Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.6820302Z 2025-08-26T20:40:01.6820429Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6820786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6821132Z return mod(**inputs) 2025-08-26T20:40:01.6821502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6821931Z outputs = self.model.decoder( 2025-08-26T20:40:01.6822331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6822724Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6823077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6823453Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6823849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6824297Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6824722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.6825134Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.6825277Z 2025-08-26T20:40:01.6825393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6825759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6826080Z return mod(**inputs) 2025-08-26T20:40:01.6826457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6826861Z outputs = self.model.decoder( 2025-08-26T20:40:01.6827421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6828010Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6828362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6828731Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6829143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6829595Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6829772Z 2025-08-26T20:40:01.6829886Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6830244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6830575Z return mod(**inputs) 2025-08-26T20:40:01.6830953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6831359Z outputs = self.model.decoder( 2025-08-26T20:40:01.6831748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6832148Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6832529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6832900Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6833324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6833764Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6834162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.6834506Z return self.act(input) 2025-08-26T20:40:01.6834618Z 2025-08-26T20:40:01.6834731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6835089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6835433Z return mod(**inputs) 2025-08-26T20:40:01.6835833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6836263Z outputs = self.model.decoder( 2025-08-26T20:40:01.6836703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6837118Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6837492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6837877Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6838518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.6839119Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.6839273Z 2025-08-26T20:40:01.6839388Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6839896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6840261Z return mod(**inputs) 2025-08-26T20:40:01.6840680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6841119Z outputs = self.model.decoder( 2025-08-26T20:40:01.6841558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6841999Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6842387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6842791Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6843231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6843699Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6844173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6844706Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6844936Z 2025-08-26T20:40:01.6845056Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6845452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6845811Z return mod(**inputs) 2025-08-26T20:40:01.6846223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6846664Z outputs = self.model.decoder( 2025-08-26T20:40:01.6847091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6847534Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6847960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6848371Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6848845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6849308Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6849787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.6850248Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.6850395Z 2025-08-26T20:40:01.6850517Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6850899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6851239Z return mod(**inputs) 2025-08-26T20:40:01.6851635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6852057Z outputs = self.model.decoder( 2025-08-26T20:40:01.6852499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6852915Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6853288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6853674Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6854143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6854652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6855089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.6855548Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.6855708Z 2025-08-26T20:40:01.6855796Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6856029Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6856256Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6856473Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6856722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6857117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6857470Z return mod(**inputs) 2025-08-26T20:40:01.6857863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6858293Z outputs = self.model.decoder( 2025-08-26T20:40:01.6858707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6859129Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6859502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6859885Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6860315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6860763Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6861207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6861646Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6862127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.6862643Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.6862837Z 2025-08-26T20:40:01.6862954Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6863359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6863727Z return mod(**inputs) 2025-08-26T20:40:01.6864131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6864560Z outputs = self.model.decoder( 2025-08-26T20:40:01.6864979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6865403Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6865775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6866162Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6866595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6867051Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6867513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6867960Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6868437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.6868930Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.6869106Z 2025-08-26T20:40:01.6869224Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6869626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6869978Z return mod(**inputs) 2025-08-26T20:40:01.6870385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6870817Z outputs = self.model.decoder( 2025-08-26T20:40:01.6871260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6871679Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6872055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6872447Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6872893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6873350Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6873810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.6874245Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.6874400Z 2025-08-26T20:40:01.6874516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6874907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6875251Z return mod(**inputs) 2025-08-26T20:40:01.6875652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6876085Z outputs = self.model.decoder( 2025-08-26T20:40:01.6876506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6876933Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6877301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6877702Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6878215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6878722Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6878912Z 2025-08-26T20:40:01.6879049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6879525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6879908Z return mod(**inputs) 2025-08-26T20:40:01.6880326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6880768Z outputs = self.model.decoder( 2025-08-26T20:40:01.6881198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6881636Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6882024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6882437Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6882955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6883446Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6883875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.6884263Z return self.act(input) 2025-08-26T20:40:01.6884386Z 2025-08-26T20:40:01.6884509Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6884940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6885302Z return mod(**inputs) 2025-08-26T20:40:01.6885712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6886153Z outputs = self.model.decoder( 2025-08-26T20:40:01.6886586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6887018Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6887404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6887807Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6888255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.6888713Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.6888866Z 2025-08-26T20:40:01.6888980Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6889385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6889739Z return mod(**inputs) 2025-08-26T20:40:01.6890141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6890569Z outputs = self.model.decoder( 2025-08-26T20:40:01.6890987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6891387Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6891743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6892114Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6892538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:01.6892985Z hidden_states = residual + hidden_states 2025-08-26T20:40:01.6893136Z 2025-08-26T20:40:01.6893249Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6893658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6894011Z return mod(**inputs) 2025-08-26T20:40:01.6894421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6894850Z outputs = self.model.decoder( 2025-08-26T20:40:01.6895273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6895698Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6896074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6896603Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6897015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6897478Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6897934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6898481Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6898705Z 2025-08-26T20:40:01.6898817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6899202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6899549Z return mod(**inputs) 2025-08-26T20:40:01.6899946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6900398Z outputs = self.model.decoder( 2025-08-26T20:40:01.6900820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6901251Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6901628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6902021Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6902447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6902903Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6903359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.6903797Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.6903942Z 2025-08-26T20:40:01.6904052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6904443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6904791Z return mod(**inputs) 2025-08-26T20:40:01.6905195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6905626Z outputs = self.model.decoder( 2025-08-26T20:40:01.6906043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6906476Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6906853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6907243Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6907678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6908125Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6908577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.6909068Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.6909228Z 2025-08-26T20:40:01.6909326Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6909586Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6909837Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6910062Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6910316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6910705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6911048Z return mod(**inputs) 2025-08-26T20:40:01.6911452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6911881Z outputs = self.model.decoder( 2025-08-26T20:40:01.6912304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6912724Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6913125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6913512Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6913938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6914396Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6914837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6915310Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6915786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.6916302Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.6916504Z 2025-08-26T20:40:01.6916623Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6917003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6917351Z return mod(**inputs) 2025-08-26T20:40:01.6917753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6918176Z outputs = self.model.decoder( 2025-08-26T20:40:01.6918587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6919015Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6919397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6919973Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6920424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6920890Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6921363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6921836Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6922313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.6922810Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.6922988Z 2025-08-26T20:40:01.6923100Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6923496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6923846Z return mod(**inputs) 2025-08-26T20:40:01.6924282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6924707Z outputs = self.model.decoder( 2025-08-26T20:40:01.6925149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6925576Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6925954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6926341Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6926769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6927223Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6927667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.6928108Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.6928287Z 2025-08-26T20:40:01.6928407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6928803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6929153Z return mod(**inputs) 2025-08-26T20:40:01.6929542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6929943Z outputs = self.model.decoder( 2025-08-26T20:40:01.6930328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6930763Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6931140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6931540Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6931994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6932467Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6932663Z 2025-08-26T20:40:01.6932768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6933135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6933462Z return mod(**inputs) 2025-08-26T20:40:01.6933842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6934247Z outputs = self.model.decoder( 2025-08-26T20:40:01.6934643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6935044Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6935400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6935767Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6936171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6936617Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6937013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.6937361Z return self.act(input) 2025-08-26T20:40:01.6937475Z 2025-08-26T20:40:01.6937582Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6937949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6938278Z return mod(**inputs) 2025-08-26T20:40:01.6938678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6939084Z outputs = self.model.decoder( 2025-08-26T20:40:01.6939499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6939905Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6940257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6940626Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6941030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.6941460Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.6941615Z 2025-08-26T20:40:01.6941726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6942120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6942472Z return mod(**inputs) 2025-08-26T20:40:01.6942890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6943320Z outputs = self.model.decoder( 2025-08-26T20:40:01.6943740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6944168Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6944543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6944963Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6945392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6945845Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6946301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6946809Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6947036Z 2025-08-26T20:40:01.6947147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6947535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6947882Z return mod(**inputs) 2025-08-26T20:40:01.6948281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6948707Z outputs = self.model.decoder( 2025-08-26T20:40:01.6949124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6949553Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6949931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6950322Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6950748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6951202Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6951649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.6952084Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.6952230Z 2025-08-26T20:40:01.6952347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6952724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6953069Z return mod(**inputs) 2025-08-26T20:40:01.6953575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6954011Z outputs = self.model.decoder( 2025-08-26T20:40:01.6954453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6954874Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6955232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6955610Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6956020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6956444Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6956872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.6957290Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.6957438Z 2025-08-26T20:40:01.6957548Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6957774Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6957997Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6958223Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.6958479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6958869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6959215Z return mod(**inputs) 2025-08-26T20:40:01.6959842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6960394Z outputs = self.model.decoder( 2025-08-26T20:40:01.6960831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6961267Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6961628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6962005Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6962411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6962842Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6963264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6963694Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6964153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.6964643Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.6964828Z 2025-08-26T20:40:01.6964944Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6965307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6965644Z return mod(**inputs) 2025-08-26T20:40:01.6966028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6966437Z outputs = self.model.decoder( 2025-08-26T20:40:01.6966827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6967233Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6967589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6967956Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6968382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6968808Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6969250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.6969680Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.6970154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.6970818Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.6971038Z 2025-08-26T20:40:01.6971150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6971536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6971884Z return mod(**inputs) 2025-08-26T20:40:01.6972263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6972688Z outputs = self.model.decoder( 2025-08-26T20:40:01.6973073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6973474Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6973827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6974194Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6974590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6975044Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6975564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.6975979Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.6976119Z 2025-08-26T20:40:01.6976232Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6976591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6976926Z return mod(**inputs) 2025-08-26T20:40:01.6977307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6977713Z outputs = self.model.decoder( 2025-08-26T20:40:01.6978108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6978512Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6978878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6979274Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6979719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6980193Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6980389Z 2025-08-26T20:40:01.6980504Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6980889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6981239Z return mod(**inputs) 2025-08-26T20:40:01.6981640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6982068Z outputs = self.model.decoder( 2025-08-26T20:40:01.6982490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6982911Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6983309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6983701Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6984149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.6984631Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.6985050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.6985418Z return self.act(input) 2025-08-26T20:40:01.6985538Z 2025-08-26T20:40:01.6985650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6986038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6986387Z return mod(**inputs) 2025-08-26T20:40:01.6986790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6987220Z outputs = self.model.decoder( 2025-08-26T20:40:01.6987654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6988077Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6988447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6988836Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6989265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.6989708Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.6989864Z 2025-08-26T20:40:01.6989975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6990362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6990708Z return mod(**inputs) 2025-08-26T20:40:01.6991099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6991531Z outputs = self.model.decoder( 2025-08-26T20:40:01.6991944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6992365Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6992737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6993118Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6993549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:01.6993990Z hidden_states = residual + hidden_states 2025-08-26T20:40:01.6994134Z 2025-08-26T20:40:01.6994252Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.6994637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.6994985Z return mod(**inputs) 2025-08-26T20:40:01.6995388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.6995820Z outputs = self.model.decoder( 2025-08-26T20:40:01.6996362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.6996789Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.6997170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.6997558Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.6997992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.6998511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.6998982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.6999550Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.6999785Z 2025-08-26T20:40:01.6999905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7000303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7000664Z return mod(**inputs) 2025-08-26T20:40:01.7001059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7001483Z outputs = self.model.decoder( 2025-08-26T20:40:01.7001898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7002325Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7002733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7003137Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7003574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7004042Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7004507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.7004964Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.7005109Z 2025-08-26T20:40:01.7005216Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7005581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7005912Z return mod(**inputs) 2025-08-26T20:40:01.7006293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7006692Z outputs = self.model.decoder( 2025-08-26T20:40:01.7007083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7007480Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7007832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7008192Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7008597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7009020Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7009447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.7009876Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.7010027Z 2025-08-26T20:40:01.7010117Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7010348Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7010575Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7010797Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7011040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7011430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7011779Z return mod(**inputs) 2025-08-26T20:40:01.7014797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7015226Z outputs = self.model.decoder( 2025-08-26T20:40:01.7015653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7016068Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7016427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7016804Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7017207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7017637Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7018065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7018494Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7018976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.7019469Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.7019678Z 2025-08-26T20:40:01.7019811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7020190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7020529Z return mod(**inputs) 2025-08-26T20:40:01.7020912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7021365Z outputs = self.model.decoder( 2025-08-26T20:40:01.7021789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7022222Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7022572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7022968Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7023416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7023880Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7024336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7024833Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7025282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.7025749Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.7025914Z 2025-08-26T20:40:01.7026027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7026395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7026719Z return mod(**inputs) 2025-08-26T20:40:01.7027099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7027506Z outputs = self.model.decoder( 2025-08-26T20:40:01.7027911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7028301Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7028645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7041786Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7042468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7043122Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7043642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.7044074Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.7044221Z 2025-08-26T20:40:01.7044346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7044724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7045071Z return mod(**inputs) 2025-08-26T20:40:01.7045469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7045889Z outputs = self.model.decoder( 2025-08-26T20:40:01.7046295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7046715Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7047083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7047462Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7047937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7048408Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7048608Z 2025-08-26T20:40:01.7048726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7049122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7049481Z return mod(**inputs) 2025-08-26T20:40:01.7049893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7050348Z outputs = self.model.decoder( 2025-08-26T20:40:01.7050753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7051158Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7051517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7051887Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7052296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7052740Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7053134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.7053486Z return self.act(input) 2025-08-26T20:40:01.7053600Z 2025-08-26T20:40:01.7053709Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7054091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7054424Z return mod(**inputs) 2025-08-26T20:40:01.7054803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7055213Z outputs = self.model.decoder( 2025-08-26T20:40:01.7055611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7056011Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7056356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7056724Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7057133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.7057561Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.7057752Z 2025-08-26T20:40:01.7057876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7058270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7058605Z return mod(**inputs) 2025-08-26T20:40:01.7058989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7059398Z outputs = self.model.decoder( 2025-08-26T20:40:01.7059794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7060199Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7060555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7060931Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7061345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7061766Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7062195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.7062377Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.7062382Z 2025-08-26T20:40:01.7062490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7062698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7062767Z return mod(**inputs) 2025-08-26T20:40:01.7063039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7063144Z outputs = self.model.decoder( 2025-08-26T20:40:01.7063417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7063499Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7063733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7063820Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7064096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7064198Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7064472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.7064557Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.7064562Z 2025-08-26T20:40:01.7064670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7064882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7064952Z return mod(**inputs) 2025-08-26T20:40:01.7065229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7065307Z outputs = self.model.decoder( 2025-08-26T20:40:01.7065585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7065661Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7065889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7065991Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7066251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7066378Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7066645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.7066748Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.7066754Z 2025-08-26T20:40:01.7066848Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7066931Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7067018Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7067095Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7067200Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7067411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7067478Z return mod(**inputs) 2025-08-26T20:40:01.7067759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7067835Z outputs = self.model.decoder( 2025-08-26T20:40:01.7068106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7068181Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7068420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7068509Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7068773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7068880Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7069145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7069274Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7069595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.7069742Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.7069747Z 2025-08-26T20:40:01.7069867Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7070080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7070157Z return mod(**inputs) 2025-08-26T20:40:01.7070441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7070520Z outputs = self.model.decoder( 2025-08-26T20:40:01.7070809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7070889Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7071130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7071215Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7071499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7071611Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7071892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7072003Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7072316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.7072434Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.7072446Z 2025-08-26T20:40:01.7072558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7072797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7072879Z return mod(**inputs) 2025-08-26T20:40:01.7073220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7073311Z outputs = self.model.decoder( 2025-08-26T20:40:01.7073595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7073673Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7073922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7074008Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7074311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7074419Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7074713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.7074810Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.7074832Z 2025-08-26T20:40:01.7074946Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7075167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7075239Z return mod(**inputs) 2025-08-26T20:40:01.7075528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7075607Z outputs = self.model.decoder( 2025-08-26T20:40:01.7075891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7075998Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7076245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7076339Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7076631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7076762Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7076773Z 2025-08-26T20:40:01.7076886Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7077103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7077182Z return mod(**inputs) 2025-08-26T20:40:01.7077473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7077562Z outputs = self.model.decoder( 2025-08-26T20:40:01.7077853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7077932Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7078186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7078273Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7078570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7078714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7078943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.7079032Z return self.act(input) 2025-08-26T20:40:01.7079036Z 2025-08-26T20:40:01.7079150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7079391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7079556Z return mod(**inputs) 2025-08-26T20:40:01.7079889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7079979Z outputs = self.model.decoder( 2025-08-26T20:40:01.7080335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7080447Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7080823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7080959Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7081315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.7081426Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.7081430Z 2025-08-26T20:40:01.7081562Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7081756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7081830Z return mod(**inputs) 2025-08-26T20:40:01.7082116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7082189Z outputs = self.model.decoder( 2025-08-26T20:40:01.7082457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7082527Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7082758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7082863Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7083154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:01.7083240Z hidden_states = residual + hidden_states 2025-08-26T20:40:01.7083244Z 2025-08-26T20:40:01.7083358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7083585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7083657Z return mod(**inputs) 2025-08-26T20:40:01.7083953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7084033Z outputs = self.model.decoder( 2025-08-26T20:40:01.7084320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7084406Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7084649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7084742Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7085031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7085140Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7085443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.7085605Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.7085609Z 2025-08-26T20:40:01.7085724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7085934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7086010Z return mod(**inputs) 2025-08-26T20:40:01.7086292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7086390Z outputs = self.model.decoder( 2025-08-26T20:40:01.7086681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7086777Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7087025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7087113Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7087402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7087518Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7087807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.7087906Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.7087910Z 2025-08-26T20:40:01.7088025Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7088249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7088323Z return mod(**inputs) 2025-08-26T20:40:01.7088632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7088719Z outputs = self.model.decoder( 2025-08-26T20:40:01.7089008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7089093Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7089333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7089440Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7089783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7089886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7090180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.7090275Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.7090279Z 2025-08-26T20:40:01.7090375Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7090461Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7090544Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7090635Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7090745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7090967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7091039Z return mod(**inputs) 2025-08-26T20:40:01.7091322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7091409Z outputs = self.model.decoder( 2025-08-26T20:40:01.7091689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7091775Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7092014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7092100Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7092397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7092503Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7092794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7092923Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7093336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.7093563Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.7093572Z 2025-08-26T20:40:01.7093740Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7094111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7094188Z return mod(**inputs) 2025-08-26T20:40:01.7094486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7094566Z outputs = self.model.decoder( 2025-08-26T20:40:01.7094923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7095020Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7095265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7095362Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7095653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7095792Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7096082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7096353Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7096794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.7097001Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.7097007Z 2025-08-26T20:40:01.7097129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7097341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7097413Z return mod(**inputs) 2025-08-26T20:40:01.7097707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7097787Z outputs = self.model.decoder( 2025-08-26T20:40:01.7098076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7098154Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7098397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7098486Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7098770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7098883Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7099162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.7099261Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.7099264Z 2025-08-26T20:40:01.7099375Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7099583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7099664Z return mod(**inputs) 2025-08-26T20:40:01.7099948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7100037Z outputs = self.model.decoder( 2025-08-26T20:40:01.7100321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7100441Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7100691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7100808Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7101086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7101208Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7101212Z 2025-08-26T20:40:01.7101322Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7101521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7101586Z return mod(**inputs) 2025-08-26T20:40:01.7101862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7101937Z outputs = self.model.decoder( 2025-08-26T20:40:01.7102211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7102286Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7102541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7102632Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7102897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7103023Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7103248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.7103350Z return self.act(input) 2025-08-26T20:40:01.7103354Z 2025-08-26T20:40:01.7103464Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7103675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7103752Z return mod(**inputs) 2025-08-26T20:40:01.7104028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7104110Z outputs = self.model.decoder( 2025-08-26T20:40:01.7104381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7104457Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7104703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7104787Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7105076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.7105165Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.7105169Z 2025-08-26T20:40:01.7105287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7105499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7105571Z return mod(**inputs) 2025-08-26T20:40:01.7105863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7105938Z outputs = self.model.decoder( 2025-08-26T20:40:01.7106212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7106285Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7106508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7106599Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7106894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7107020Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7107285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.7107438Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.7107450Z 2025-08-26T20:40:01.7107557Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7107754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7107830Z return mod(**inputs) 2025-08-26T20:40:01.7108097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7108179Z outputs = self.model.decoder( 2025-08-26T20:40:01.7108462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7108535Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7108767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7108867Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7109137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7109238Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7109501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.7109612Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.7109615Z 2025-08-26T20:40:01.7109717Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7109922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7109990Z return mod(**inputs) 2025-08-26T20:40:01.7110267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7110342Z outputs = self.model.decoder( 2025-08-26T20:40:01.7110607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7110689Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7110911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7110998Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7111271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7111371Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7111645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.7111734Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.7111738Z 2025-08-26T20:40:01.7111826Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7111907Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7111987Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7112071Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7112177Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7112385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7112450Z return mod(**inputs) 2025-08-26T20:40:01.7112716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7112821Z outputs = self.model.decoder( 2025-08-26T20:40:01.7113089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7113184Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7113413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7113501Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7113767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7113864Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7114140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7114240Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7114539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.7114674Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.7114692Z 2025-08-26T20:40:01.7114798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7115005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7115073Z return mod(**inputs) 2025-08-26T20:40:01.7115354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7115433Z outputs = self.model.decoder( 2025-08-26T20:40:01.7115722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7115818Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7116056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7116147Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7116423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7116535Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7116815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7116918Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7117232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.7117353Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.7117357Z 2025-08-26T20:40:01.7117477Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7117689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7117768Z return mod(**inputs) 2025-08-26T20:40:01.7118058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7118141Z outputs = self.model.decoder( 2025-08-26T20:40:01.7118436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7118514Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7118764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7118851Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7119153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7119289Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7119719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.7119851Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.7119858Z 2025-08-26T20:40:01.7119974Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7120200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7120274Z return mod(**inputs) 2025-08-26T20:40:01.7120567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7120657Z outputs = self.model.decoder( 2025-08-26T20:40:01.7120950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7121040Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7121292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7121415Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7121857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7122017Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7122021Z 2025-08-26T20:40:01.7122139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7122353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7122424Z return mod(**inputs) 2025-08-26T20:40:01.7122720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7122816Z outputs = self.model.decoder( 2025-08-26T20:40:01.7123114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7123192Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7123440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7123526Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7123813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7123946Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7124178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.7124261Z return self.act(input) 2025-08-26T20:40:01.7124267Z 2025-08-26T20:40:01.7124380Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7124595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7124673Z return mod(**inputs) 2025-08-26T20:40:01.7124962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7125048Z outputs = self.model.decoder( 2025-08-26T20:40:01.7125336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7125419Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7125660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7125744Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7126041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.7126130Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.7126156Z 2025-08-26T20:40:01.7126277Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7126503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7126578Z return mod(**inputs) 2025-08-26T20:40:01.7126866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7126946Z outputs = self.model.decoder( 2025-08-26T20:40:01.7127235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7127312Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7127549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7127642Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7127923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:01.7128018Z hidden_states = residual + hidden_states 2025-08-26T20:40:01.7128021Z 2025-08-26T20:40:01.7128130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7128364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7128434Z return mod(**inputs) 2025-08-26T20:40:01.7128712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7128797Z outputs = self.model.decoder( 2025-08-26T20:40:01.7129075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7129194Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7129431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7129516Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7129807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7129913Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7130206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:01.7130369Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:01.7130373Z 2025-08-26T20:40:01.7130488Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7130699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7130772Z return mod(**inputs) 2025-08-26T20:40:01.7131062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7131138Z outputs = self.model.decoder( 2025-08-26T20:40:01.7131425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7131505Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7131750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7131834Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7132096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7132247Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7132621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:01.7132743Z key_states = self.k_proj(current_states) 2025-08-26T20:40:01.7132778Z 2025-08-26T20:40:01.7132901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7133129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7133204Z return mod(**inputs) 2025-08-26T20:40:01.7133463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7133540Z outputs = self.model.decoder( 2025-08-26T20:40:01.7133801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7133874Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7134105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7134187Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7134464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7134564Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7134838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:01.7134945Z value_states = self.v_proj(current_states) 2025-08-26T20:40:01.7134949Z 2025-08-26T20:40:01.7135031Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7135118Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7135198Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7135283Z cudagraph partition due to non gpu ops 2025-08-26T20:40:01.7135385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7135586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7135684Z return mod(**inputs) 2025-08-26T20:40:01.7135974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7136060Z outputs = self.model.decoder( 2025-08-26T20:40:01.7136354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7136430Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7136669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7136749Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7137049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7137148Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7137421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7137530Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7137829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:01.7137974Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:01.7137978Z 2025-08-26T20:40:01.7138083Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7138295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7138363Z return mod(**inputs) 2025-08-26T20:40:01.7138636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7138720Z outputs = self.model.decoder( 2025-08-26T20:40:01.7138993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7139093Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7139316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7139412Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7139687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7139784Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7140056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:01.7140152Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:01.7140449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:01.7140562Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:01.7140566Z 2025-08-26T20:40:01.7140670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7140878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7140946Z return mod(**inputs) 2025-08-26T20:40:01.7141238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7141313Z outputs = self.model.decoder( 2025-08-26T20:40:01.7141579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7141660Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7141883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7141987Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7142260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:01.7142364Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:01.7142654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:01.7142741Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:01.7142744Z 2025-08-26T20:40:01.7142860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7143069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7143147Z return mod(**inputs) 2025-08-26T20:40:01.7143428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7143508Z outputs = self.model.decoder( 2025-08-26T20:40:01.7143798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7143874Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7144117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7144203Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7144492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7144611Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7144615Z 2025-08-26T20:40:01.7144717Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7144921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7144989Z return mod(**inputs) 2025-08-26T20:40:01.7145258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7145347Z outputs = self.model.decoder( 2025-08-26T20:40:01.7145644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7145728Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7145945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7146031Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7146289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:01.7146403Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:01.7146618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:01.7146688Z return self.act(input) 2025-08-26T20:40:01.7146691Z 2025-08-26T20:40:01.7146802Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7147003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7147076Z return mod(**inputs) 2025-08-26T20:40:01.7147343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-26T20:40:01.7147435Z outputs = self.model.decoder( 2025-08-26T20:40:01.7147723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:01.7147800Z layer_outputs = decoder_layer( 2025-08-26T20:40:01.7148045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:01.7148147Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:01.7148439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:01.7148533Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:01.7148537Z 2025-08-26T20:40:01.7148647Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7148865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7148932Z return mod(**inputs) 2025-08-26T20:40:01.7149204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1650, in forward 2025-08-26T20:40:01.7149284Z logits = self.lm_head(outputs[0]) 2025-08-26T20:40:01.7149288Z 2025-08-26T20:40:01.7149391Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:01.7149597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:01.7149666Z return mod(**inputs) 2025-08-26T20:40:01.7149939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1656, in forward 2025-08-26T20:40:01.7150090Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:40:01.7150094Z 2025-08-26T20:40:11.1589342Z Compilation time (from dynamo_timed): 16.125468945 2025-08-26T20:40:11.1607040Z pass 2025-08-26T20:40:11.1607979Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:40:11.1616020Z TIMING: _recursive_pre_grad_passes:0.00815 _recursive_joint_graph_passes:0.65941 _recursive_post_grad_passes:0.0798 async_compile.wait:0.72202 code_gen:9.03504 inductor_compile:10.40383 backend_compile:13.771 gc:0.00125 entire_frame_compile:16.12547 total_wall_time:16.12547 2025-08-26T20:40:11.1617527Z STATS: call_* op count: 369 | FakeTensorMode.__torch_dispatch__:13164 | FakeTensor.__torch_dispatch__:4526 | ProxyTorchDispatchMode.__torch_dispatch__:4803 2025-08-26T20:40:11.1618165Z Dynamo produced 1 graphs covering 369 ops with 0 graph breaks (0 unique) 2025-08-26T20:40:16.7962450Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:40:16.7963476Z from pkg_resources import resource_filename 2025-08-26T20:40:17.4050356Z 2025-08-26T20:40:23.4649494Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:40:23.4654065Z loading model: 0it [00:06, ?it/s] 2025-08-26T20:40:23.4680242Z cpu eval PegasusForConditionalGeneration 2025-08-26T20:40:24.1423978Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:40:24.4347400Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:40:24.7149323Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:40:42.6745246Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6745719Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6745978Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6746269Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6747095Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6747354Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6747577Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6747807Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6748159Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6748411Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6748658Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6748993Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6749390Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6749949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6750440Z return mod(**inputs) 2025-08-26T20:40:42.6750937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6751518Z outputs = self.model( 2025-08-26T20:40:42.6751977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6752453Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6753073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6753529Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6753982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6754414Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6754923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6755403Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6755941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.6756481Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.6756718Z 2025-08-26T20:40:42.6756839Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6757245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6757609Z return mod(**inputs) 2025-08-26T20:40:42.6758024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6758453Z outputs = self.model( 2025-08-26T20:40:42.6758989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6759630Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6760145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6760591Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6760979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6761385Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6761831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6762300Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6762759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.6763255Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.6763401Z 2025-08-26T20:40:42.6763509Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6763899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6764272Z return mod(**inputs) 2025-08-26T20:40:42.6764665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6765087Z outputs = self.model( 2025-08-26T20:40:42.6765489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6765915Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6766310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6766726Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6767105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6767495Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6767929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6768389Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6768834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.6769273Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.6769432Z 2025-08-26T20:40:42.6769521Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6769752Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6769975Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6770198Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6770451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6770846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6771201Z return mod(**inputs) 2025-08-26T20:40:42.6771605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6772051Z outputs = self.model( 2025-08-26T20:40:42.6772433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6772838Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6773232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6773658Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6774037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6774449Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6774878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6775329Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6775777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6776233Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6776713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.6777232Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.6777436Z 2025-08-26T20:40:42.6777552Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6777941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6778291Z return mod(**inputs) 2025-08-26T20:40:42.6778706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6779142Z outputs = self.model( 2025-08-26T20:40:42.6779566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6779994Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6780411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6780840Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6781224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6781648Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6782109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6782567Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6783009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6783454Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6783930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.6784417Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.6784589Z 2025-08-26T20:40:42.6784706Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6785093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6785434Z return mod(**inputs) 2025-08-26T20:40:42.6785838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6786285Z outputs = self.model( 2025-08-26T20:40:42.6786706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6787128Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6787546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6787971Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6788340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6788728Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6789158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6789614Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6790085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.6790546Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.6790710Z 2025-08-26T20:40:42.6790825Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6791225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6791590Z return mod(**inputs) 2025-08-26T20:40:42.6791998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6792434Z outputs = self.model( 2025-08-26T20:40:42.6792842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6793278Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6793706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6794132Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6794519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6794941Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6795377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6795878Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6796078Z 2025-08-26T20:40:42.6796532Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6796946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6797366Z return mod(**inputs) 2025-08-26T20:40:42.6797796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6798241Z outputs = self.model( 2025-08-26T20:40:42.6798666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6799117Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6799620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6800080Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6800463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6800881Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6801342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6801850Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6802287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.6802666Z return self.act(input) 2025-08-26T20:40:42.6802799Z 2025-08-26T20:40:42.6802916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6803320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6803682Z return mod(**inputs) 2025-08-26T20:40:42.6804097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6804533Z outputs = self.model( 2025-08-26T20:40:42.6804950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6805393Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6805897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6806331Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6806751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6807155Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6807615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.6808073Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.6808229Z 2025-08-26T20:40:42.6808344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6808744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6809111Z return mod(**inputs) 2025-08-26T20:40:42.6809523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6809948Z outputs = self.model( 2025-08-26T20:40:42.6810416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6810895Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6811325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6811757Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6812134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6812530Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6812963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6813442Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6813901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.6814425Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.6814655Z 2025-08-26T20:40:42.6814767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6815155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6815501Z return mod(**inputs) 2025-08-26T20:40:42.6815896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6816319Z outputs = self.model( 2025-08-26T20:40:42.6816723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6817173Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6817608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6818051Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6818438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6818835Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6819278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6819741Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6820187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.6820619Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.6820775Z 2025-08-26T20:40:42.6820885Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6821295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6821654Z return mod(**inputs) 2025-08-26T20:40:42.6822081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6822512Z outputs = self.model( 2025-08-26T20:40:42.6822919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6823345Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6823754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6824176Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6824549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6824941Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6825400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6825850Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6826326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.6826793Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.6826957Z 2025-08-26T20:40:42.6827051Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6827274Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6827501Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6827721Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6827967Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6828369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6828711Z return mod(**inputs) 2025-08-26T20:40:42.6829127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6829563Z outputs = self.model( 2025-08-26T20:40:42.6829969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6830398Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6830837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6831262Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6831652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6832105Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6832592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6833066Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6833534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6834014Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6834509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.6835033Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.6835249Z 2025-08-26T20:40:42.6835366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6835785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6836161Z return mod(**inputs) 2025-08-26T20:40:42.6836593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6837046Z outputs = self.model( 2025-08-26T20:40:42.6837479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6837927Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6838358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6838791Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6839179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6839664Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6840115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6840579Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6841026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6841495Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6841982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.6842516Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.6842698Z 2025-08-26T20:40:42.6842823Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6843216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6843576Z return mod(**inputs) 2025-08-26T20:40:42.6843988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6844442Z outputs = self.model( 2025-08-26T20:40:42.6844855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6845291Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6845723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6846186Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6846573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6846962Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6847411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6847853Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6848297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.6848742Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.6848889Z 2025-08-26T20:40:42.6849000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6849389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6849742Z return mod(**inputs) 2025-08-26T20:40:42.6850147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6850562Z outputs = self.model( 2025-08-26T20:40:42.6850970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6851395Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6851811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6852237Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6852620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6853021Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6853477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6853962Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6854147Z 2025-08-26T20:40:42.6854267Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6854646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6855001Z return mod(**inputs) 2025-08-26T20:40:42.6855412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6855838Z outputs = self.model( 2025-08-26T20:40:42.6856238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6856662Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6857083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6857532Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6857903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6858284Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6858713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6859182Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6859645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.6860024Z return self.act(input) 2025-08-26T20:40:42.6860142Z 2025-08-26T20:40:42.6860252Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6860644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6860994Z return mod(**inputs) 2025-08-26T20:40:42.6861395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6861811Z outputs = self.model( 2025-08-26T20:40:42.6862214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6862638Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6863057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6863485Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6863851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6864239Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6864681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.6865118Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.6865265Z 2025-08-26T20:40:42.6865385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6865764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6866111Z return mod(**inputs) 2025-08-26T20:40:42.6866509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6866935Z outputs = self.model( 2025-08-26T20:40:42.6867349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6867782Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6868238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6868687Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6869058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6869437Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6869863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6870306Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6870748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.6871257Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.6871474Z 2025-08-26T20:40:42.6871587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6871975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6872341Z return mod(**inputs) 2025-08-26T20:40:42.6872751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6873170Z outputs = self.model( 2025-08-26T20:40:42.6873584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6874039Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6874476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6874936Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6875315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6875722Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6876179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6876648Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6877103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.6877538Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.6877696Z 2025-08-26T20:40:42.6878455Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6878854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6879210Z return mod(**inputs) 2025-08-26T20:40:42.6880246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6880698Z outputs = self.model( 2025-08-26T20:40:42.6881122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6881598Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6882036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6882522Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6882910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6883309Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6883785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6884299Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6884791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.6885266Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.6885434Z 2025-08-26T20:40:42.6885541Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6885777Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6886018Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6886250Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6886515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6886909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6887267Z return mod(**inputs) 2025-08-26T20:40:42.6887685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6888119Z outputs = self.model( 2025-08-26T20:40:42.6888536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6888968Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6890180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6890641Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6891031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6891434Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6891907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6892392Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6892894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6893369Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6893848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.6894378Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.6894589Z 2025-08-26T20:40:42.6894704Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6895101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6895459Z return mod(**inputs) 2025-08-26T20:40:42.6895868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6896446Z outputs = self.model( 2025-08-26T20:40:42.6896879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6897313Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6897747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6898182Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6898569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6898971Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6899413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6899866Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6900318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6900776Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6901444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.6901992Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.6902178Z 2025-08-26T20:40:42.6902295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6902701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6903045Z return mod(**inputs) 2025-08-26T20:40:42.6903449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6903870Z outputs = self.model( 2025-08-26T20:40:42.6904262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6904693Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6905116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6905544Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6905919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6906337Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6906785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6907236Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6907689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.6908124Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.6908310Z 2025-08-26T20:40:42.6908422Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6908815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6909173Z return mod(**inputs) 2025-08-26T20:40:42.6909593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6910006Z outputs = self.model( 2025-08-26T20:40:42.6910406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6910836Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6911276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6911707Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6912081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6912483Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6912944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6913431Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6913622Z 2025-08-26T20:40:42.6913737Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6914132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6914485Z return mod(**inputs) 2025-08-26T20:40:42.6914894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6915323Z outputs = self.model( 2025-08-26T20:40:42.6915726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6916166Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6916615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6917058Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6917453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6917858Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6918302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6918787Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6919216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.6919651Z return self.act(input) 2025-08-26T20:40:42.6919789Z 2025-08-26T20:40:42.6919907Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6920313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6920676Z return mod(**inputs) 2025-08-26T20:40:42.6921094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6921564Z outputs = self.model( 2025-08-26T20:40:42.6921980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6922420Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6922855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6923294Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6923679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6924095Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6924551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.6924973Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.6925113Z 2025-08-26T20:40:42.6925221Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6925587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6925917Z return mod(**inputs) 2025-08-26T20:40:42.6926291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6926686Z outputs = self.model( 2025-08-26T20:40:42.6927053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6927458Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6927858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6928278Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6928645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6929034Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6929482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-26T20:40:42.6929908Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.6930043Z 2025-08-26T20:40:42.6930154Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6930512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6930841Z return mod(**inputs) 2025-08-26T20:40:42.6931239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6931638Z outputs = self.model( 2025-08-26T20:40:42.6932033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6932433Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6932829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6933228Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6933580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6933939Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6934343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6934765Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6935190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.6935669Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.6935895Z 2025-08-26T20:40:42.6936000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6936363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6936694Z return mod(**inputs) 2025-08-26T20:40:42.6937074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6937475Z outputs = self.model( 2025-08-26T20:40:42.6937848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6938290Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6938719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6939151Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6939533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6939946Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6940393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6940861Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6941310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.6941742Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.6941902Z 2025-08-26T20:40:42.6942015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6942419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6942785Z return mod(**inputs) 2025-08-26T20:40:42.6943209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6943649Z outputs = self.model( 2025-08-26T20:40:42.6944070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6944513Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6944950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6945390Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6945783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6946199Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6946667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6947125Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6947588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.6948035Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.6948197Z 2025-08-26T20:40:42.6948287Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6948524Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6948755Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6948978Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.6949238Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6949633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6949987Z return mod(**inputs) 2025-08-26T20:40:42.6950393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6950825Z outputs = self.model( 2025-08-26T20:40:42.6951237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6951701Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6952125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6952562Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6952937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6953330Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6953780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6954216Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6954657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6955103Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6955576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.6956095Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.6956292Z 2025-08-26T20:40:42.6956402Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6956783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6957131Z return mod(**inputs) 2025-08-26T20:40:42.6957530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6957949Z outputs = self.model( 2025-08-26T20:40:42.6958345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6958773Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6959184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6959684Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6960060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6960462Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6960908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6961369Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6961857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.6962301Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.6962797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.6963301Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.6963477Z 2025-08-26T20:40:42.6963597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6963984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6964326Z return mod(**inputs) 2025-08-26T20:40:42.6964736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6965137Z outputs = self.model( 2025-08-26T20:40:42.6965525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6965945Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6966359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6966807Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6967171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6967555Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6967963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6968392Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6968844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.6969291Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.6969439Z 2025-08-26T20:40:42.6969557Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6969941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6970288Z return mod(**inputs) 2025-08-26T20:40:42.6970690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6971109Z outputs = self.model( 2025-08-26T20:40:42.6971506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6971933Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6972349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6972777Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6973164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6973525Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6973933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6974384Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6974562Z 2025-08-26T20:40:42.6974678Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6975044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6975364Z return mod(**inputs) 2025-08-26T20:40:42.6975745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6976141Z outputs = self.model( 2025-08-26T20:40:42.6976543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6976947Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6977373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6977799Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6978172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6978567Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6978997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.6979482Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.6979898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.6980280Z return self.act(input) 2025-08-26T20:40:42.6980399Z 2025-08-26T20:40:42.6980523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6980884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6981234Z return mod(**inputs) 2025-08-26T20:40:42.6981617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6982040Z outputs = self.model( 2025-08-26T20:40:42.6982434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6982863Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6983294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6983784Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6984159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6984548Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6984987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.6985431Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.6985579Z 2025-08-26T20:40:42.6985697Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6986088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6986437Z return mod(**inputs) 2025-08-26T20:40:42.6986832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6987253Z outputs = self.model( 2025-08-26T20:40:42.6987657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6988078Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6988496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6988920Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6989293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6989688Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6990125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6990579Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6991046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.6991578Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.6991800Z 2025-08-26T20:40:42.6991920Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6992329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6992684Z return mod(**inputs) 2025-08-26T20:40:42.6993085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6993506Z outputs = self.model( 2025-08-26T20:40:42.6993901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.6994318Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.6994720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.6995146Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.6995519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.6995899Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.6996565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.6997079Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.6997538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.6997995Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.6998143Z 2025-08-26T20:40:42.6998256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.6998642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.6999024Z return mod(**inputs) 2025-08-26T20:40:42.6999484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.6999923Z outputs = self.model( 2025-08-26T20:40:42.7000358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7000800Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7001229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7001675Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7002043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7002411Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7002819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7003245Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7003668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7004077Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7004235Z 2025-08-26T20:40:42.7004324Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7004555Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7004781Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7004996Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7005249Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7005632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7005978Z return mod(**inputs) 2025-08-26T20:40:42.7006374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7006780Z outputs = self.model( 2025-08-26T20:40:42.7007235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7007692Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7008119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7008546Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7008923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7009306Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7009749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7010196Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7010640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7011069Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7011551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7012089Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7012291Z 2025-08-26T20:40:42.7012411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7012788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7013133Z return mod(**inputs) 2025-08-26T20:40:42.7013541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7013984Z outputs = self.model( 2025-08-26T20:40:42.7014388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7014819Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7015241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7015671Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7016043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7016432Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7016882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7017340Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7017790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7018259Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7018732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7019236Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7019420Z 2025-08-26T20:40:42.7019533Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7019928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7020281Z return mod(**inputs) 2025-08-26T20:40:42.7020688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7021111Z outputs = self.model( 2025-08-26T20:40:42.7021515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7021941Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7022383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7022810Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7023199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7023600Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7024043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7024491Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7024945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7025391Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7025540Z 2025-08-26T20:40:42.7025658Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7026051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7026399Z return mod(**inputs) 2025-08-26T20:40:42.7026799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7027242Z outputs = self.model( 2025-08-26T20:40:42.7027645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7028070Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7028509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7028933Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7029308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7029726Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7030171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7030654Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7030840Z 2025-08-26T20:40:42.7030945Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7031312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7031633Z return mod(**inputs) 2025-08-26T20:40:42.7032016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7032417Z outputs = self.model( 2025-08-26T20:40:42.7032794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7033200Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7033588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7033987Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7034355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7034744Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7035184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7035667Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7036080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7036443Z return self.act(input) 2025-08-26T20:40:42.7036563Z 2025-08-26T20:40:42.7036681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7037094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7037447Z return mod(**inputs) 2025-08-26T20:40:42.7037867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7038304Z outputs = self.model( 2025-08-26T20:40:42.7038718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7039149Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7039667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7040117Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7040507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7040904Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7041351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7041805Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7041948Z 2025-08-26T20:40:42.7042089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7042466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7042798Z return mod(**inputs) 2025-08-26T20:40:42.7043188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7043599Z outputs = self.model( 2025-08-26T20:40:42.7043989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7044427Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7044841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7045269Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7045646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7046049Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7046448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-26T20:40:42.7046872Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7047029Z 2025-08-26T20:40:42.7047177Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7047564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7047918Z return mod(**inputs) 2025-08-26T20:40:42.7048311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7048746Z outputs = self.model( 2025-08-26T20:40:42.7049125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7049532Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7049928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7050321Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7050677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7051045Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7051447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7051864Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7052311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7052792Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7053013Z 2025-08-26T20:40:42.7053127Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7053499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7053810Z return mod(**inputs) 2025-08-26T20:40:42.7054182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7054579Z outputs = self.model( 2025-08-26T20:40:42.7054958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7055365Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7055755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7056161Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7056505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7056911Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7057296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7057708Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7058148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7058575Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7058740Z 2025-08-26T20:40:42.7058858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7059243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7059600Z return mod(**inputs) 2025-08-26T20:40:42.7060012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7060430Z outputs = self.model( 2025-08-26T20:40:42.7060805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7061196Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7061587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7061982Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7062333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7062693Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7063111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7063540Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7063967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7064388Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7064534Z 2025-08-26T20:40:42.7064620Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7064848Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7065072Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7065289Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7065527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7065910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7066248Z return mod(**inputs) 2025-08-26T20:40:42.7066663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7067064Z outputs = self.model( 2025-08-26T20:40:42.7067456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7067868Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7068266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7068673Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7069026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7069396Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7069806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7070226Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7070644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7071091Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7071543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7072033Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7072221Z 2025-08-26T20:40:42.7072335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7072703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7073059Z return mod(**inputs) 2025-08-26T20:40:42.7073453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7073861Z outputs = self.model( 2025-08-26T20:40:42.7074250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7074661Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7075069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7075501Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7075882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7076279Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7076716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7077169Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7077643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7078110Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7078605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7079109Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7079296Z 2025-08-26T20:40:42.7079416Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7079897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7080284Z return mod(**inputs) 2025-08-26T20:40:42.7080704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7081124Z outputs = self.model( 2025-08-26T20:40:42.7081533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7081943Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7082357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7082759Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7083125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7083493Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7083901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7084325Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7084745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7085158Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7085307Z 2025-08-26T20:40:42.7085412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7085776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7086131Z return mod(**inputs) 2025-08-26T20:40:42.7086502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7086901Z outputs = self.model( 2025-08-26T20:40:42.7087278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7087677Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7088067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7088500Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7088860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7089255Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7089682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7090130Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7090315Z 2025-08-26T20:40:42.7090421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7090785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7091117Z return mod(**inputs) 2025-08-26T20:40:42.7091494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7091888Z outputs = self.model( 2025-08-26T20:40:42.7092270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7092671Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7093066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7093461Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7093822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7094181Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7094573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7095008Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7095395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7095736Z return self.act(input) 2025-08-26T20:40:42.7095881Z 2025-08-26T20:40:42.7095991Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7096613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7096954Z return mod(**inputs) 2025-08-26T20:40:42.7097330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7097730Z outputs = self.model( 2025-08-26T20:40:42.7098111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7098514Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7098903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7099312Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7099671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7100042Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7100458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7100904Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7101054Z 2025-08-26T20:40:42.7101170Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7101523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7101846Z return mod(**inputs) 2025-08-26T20:40:42.7102204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7102629Z outputs = self.model( 2025-08-26T20:40:42.7103012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7103413Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7103808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7104201Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7104552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7104918Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7105316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7105733Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7106141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7106622Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7106835Z 2025-08-26T20:40:42.7106942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7107314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7107648Z return mod(**inputs) 2025-08-26T20:40:42.7108048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7108490Z outputs = self.model( 2025-08-26T20:40:42.7108876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7109286Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7109680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7110090Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7110494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7110891Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7111316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7111730Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7112150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7112555Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7112691Z 2025-08-26T20:40:42.7112801Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7113165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7113489Z return mod(**inputs) 2025-08-26T20:40:42.7113864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7114262Z outputs = self.model( 2025-08-26T20:40:42.7114640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7115055Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7115450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7115892Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7116246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7116613Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7117044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7117532Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7117998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7118458Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7118617Z 2025-08-26T20:40:42.7118718Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7118955Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7119192Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7119427Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7119750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7120174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7120539Z return mod(**inputs) 2025-08-26T20:40:42.7120948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7121349Z outputs = self.model( 2025-08-26T20:40:42.7121721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7122130Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7122532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7122940Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7123296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7123653Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7124057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7124475Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7124921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7125352Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7125815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7126312Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7126507Z 2025-08-26T20:40:42.7126614Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7126987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7127311Z return mod(**inputs) 2025-08-26T20:40:42.7127678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7128076Z outputs = self.model( 2025-08-26T20:40:42.7128451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7128845Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7129269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7129716Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7130091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7130486Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7130924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7131424Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7131839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7132290Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7132741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7133206Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7133383Z 2025-08-26T20:40:42.7133487Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7133844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7134165Z return mod(**inputs) 2025-08-26T20:40:42.7134537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7134929Z outputs = self.model( 2025-08-26T20:40:42.7135316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7135714Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7136112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7136524Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7136895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7137286Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7137737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7138193Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7138653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7139086Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7139236Z 2025-08-26T20:40:42.7139343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7139736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7140060Z return mod(**inputs) 2025-08-26T20:40:42.7140438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7140853Z outputs = self.model( 2025-08-26T20:40:42.7141264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7141703Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7142144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7142567Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7142943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7143318Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7143730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7144213Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7144429Z 2025-08-26T20:40:42.7144539Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7144924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7145277Z return mod(**inputs) 2025-08-26T20:40:42.7145675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7146086Z outputs = self.model( 2025-08-26T20:40:42.7146484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7146926Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7147342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7147763Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7148132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7148518Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7148943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7149413Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7149824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7150181Z return self.act(input) 2025-08-26T20:40:42.7150309Z 2025-08-26T20:40:42.7150420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7150805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7151150Z return mod(**inputs) 2025-08-26T20:40:42.7151542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7151969Z outputs = self.model( 2025-08-26T20:40:42.7152343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7152747Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7153140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7153537Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7153887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7154255Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7154694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7155123Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7155284Z 2025-08-26T20:40:42.7155418Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7155818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7156178Z return mod(**inputs) 2025-08-26T20:40:42.7156590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7157013Z outputs = self.model( 2025-08-26T20:40:42.7157429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7157873Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7158307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7158744Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7159125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7159626Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7160075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-26T20:40:42.7160519Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7160671Z 2025-08-26T20:40:42.7160787Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7161020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7161119Z return mod(**inputs) 2025-08-26T20:40:42.7161396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7161468Z outputs = self.model( 2025-08-26T20:40:42.7161742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7161823Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7162093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7162176Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7162402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7162490Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7162754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7162850Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7163124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7163277Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7163283Z 2025-08-26T20:40:42.7163396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7163609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7163687Z return mod(**inputs) 2025-08-26T20:40:42.7163970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7164043Z outputs = self.model( 2025-08-26T20:40:42.7164336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7164416Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7164729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7164805Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7165042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7165134Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7165398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7165498Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7165760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7165848Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7165853Z 2025-08-26T20:40:42.7165958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7166160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7166234Z return mod(**inputs) 2025-08-26T20:40:42.7166500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7166596Z outputs = self.model( 2025-08-26T20:40:42.7166864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7166939Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7167215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7167294Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7167535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7167637Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7167915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7168020Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7168299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7168399Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7168403Z 2025-08-26T20:40:42.7168490Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7168587Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7168669Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7168751Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7168868Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7169093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7169167Z return mod(**inputs) 2025-08-26T20:40:42.7169436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7169506Z outputs = self.model( 2025-08-26T20:40:42.7169786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7169867Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7170154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7170232Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7170465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7170557Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7170838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7170961Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7171241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7171372Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7171688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7171827Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7171831Z 2025-08-26T20:40:42.7171950Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7172161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7172238Z return mod(**inputs) 2025-08-26T20:40:42.7172521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7172597Z outputs = self.model( 2025-08-26T20:40:42.7172888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7172969Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7173281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7173358Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7173596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7173688Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7173967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7174089Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7174368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7174480Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7174788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7174907Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7174911Z 2025-08-26T20:40:42.7175027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7175238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7175315Z return mod(**inputs) 2025-08-26T20:40:42.7175597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7175672Z outputs = self.model( 2025-08-26T20:40:42.7175961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7176039Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7176324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7176403Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7176646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7176730Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7177010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7177113Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7177393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7177504Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7177508Z 2025-08-26T20:40:42.7177619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7177876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7177957Z return mod(**inputs) 2025-08-26T20:40:42.7178239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7178329Z outputs = self.model( 2025-08-26T20:40:42.7178596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7178678Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7178943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7179019Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7179260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7179344Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7179629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7179780Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7179784Z 2025-08-26T20:40:42.7179893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7180111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7180180Z return mod(**inputs) 2025-08-26T20:40:42.7180464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7180555Z outputs = self.model( 2025-08-26T20:40:42.7180837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7180930Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7181193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7181273Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7181496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7181582Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7181843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7181962Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7182186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7182261Z return self.act(input) 2025-08-26T20:40:42.7182266Z 2025-08-26T20:40:42.7182382Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7182593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7182666Z return mod(**inputs) 2025-08-26T20:40:42.7182955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7183028Z outputs = self.model( 2025-08-26T20:40:42.7183315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7183392Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7183681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7183759Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7184015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7184109Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7184404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7184501Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7184504Z 2025-08-26T20:40:42.7184614Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7184825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7184899Z return mod(**inputs) 2025-08-26T20:40:42.7185165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7185244Z outputs = self.model( 2025-08-26T20:40:42.7185511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7185584Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7185856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7185962Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7186202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7186286Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7186571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7186668Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7186945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7187135Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7187141Z 2025-08-26T20:40:42.7187251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7187471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7187543Z return mod(**inputs) 2025-08-26T20:40:42.7187822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7187903Z outputs = self.model( 2025-08-26T20:40:42.7188183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7188270Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7188545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7188626Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7188848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7188927Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7189203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7189302Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7189586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7189671Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7189674Z 2025-08-26T20:40:42.7189782Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7189998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7190069Z return mod(**inputs) 2025-08-26T20:40:42.7190377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7190451Z outputs = self.model( 2025-08-26T20:40:42.7190765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7190847Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7191125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7191211Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7191447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7191538Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7191822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7191920Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7192211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7192305Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7192328Z 2025-08-26T20:40:42.7192423Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7192510Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7192595Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7192685Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7192797Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7193014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7193084Z return mod(**inputs) 2025-08-26T20:40:42.7193369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7193479Z outputs = self.model( 2025-08-26T20:40:42.7193764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7193852Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7194135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7194219Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7194457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7194543Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7194832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7194932Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7195234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7195344Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7195665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7195820Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7195824Z 2025-08-26T20:40:42.7195937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7196161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7196369Z return mod(**inputs) 2025-08-26T20:40:42.7196682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7196762Z outputs = self.model( 2025-08-26T20:40:42.7197060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7197220Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7197535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7197627Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7197869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7197955Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7198272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7198370Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7198678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7198788Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7199116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7199241Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7199279Z 2025-08-26T20:40:42.7199396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7199673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7199751Z return mod(**inputs) 2025-08-26T20:40:42.7200051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7200125Z outputs = self.model( 2025-08-26T20:40:42.7200416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7200544Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7200838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7200937Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7201179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7201265Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7201558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7201655Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7201961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7202049Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7202055Z 2025-08-26T20:40:42.7202174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7202389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7202459Z return mod(**inputs) 2025-08-26T20:40:42.7202755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7202829Z outputs = self.model( 2025-08-26T20:40:42.7203122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7203199Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7203484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7203571Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7203811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7203906Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7204220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7204374Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7204380Z 2025-08-26T20:40:42.7204491Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7204702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7204781Z return mod(**inputs) 2025-08-26T20:40:42.7205060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7205140Z outputs = self.model( 2025-08-26T20:40:42.7205421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7205501Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7205790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7205870Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7206115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7206220Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7206517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7206643Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7206870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7206952Z return self.act(input) 2025-08-26T20:40:42.7206975Z 2025-08-26T20:40:42.7207085Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7207306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7207376Z return mod(**inputs) 2025-08-26T20:40:42.7207660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7207741Z outputs = self.model( 2025-08-26T20:40:42.7208022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7208108Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7208388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7208465Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7208707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7208791Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7209080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7209167Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7209171Z 2025-08-26T20:40:42.7209289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7209503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7209574Z return mod(**inputs) 2025-08-26T20:40:42.7209863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7209934Z outputs = self.model( 2025-08-26T20:40:42.7210223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7210304Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7210608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7210694Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7210948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7211043Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7211323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-26T20:40:42.7211414Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7211418Z 2025-08-26T20:40:42.7211527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7211736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7211814Z return mod(**inputs) 2025-08-26T20:40:42.7212101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7212181Z outputs = self.model( 2025-08-26T20:40:42.7212465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7212544Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7212852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7212928Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7213174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7213257Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7213535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7213662Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7213941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7214113Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7214118Z 2025-08-26T20:40:42.7214231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7214449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7214519Z return mod(**inputs) 2025-08-26T20:40:42.7214819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7214898Z outputs = self.model( 2025-08-26T20:40:42.7215198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7215283Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7215565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7215642Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7215889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7215973Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7216262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7216360Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7216666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7216751Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7216756Z 2025-08-26T20:40:42.7216864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7217103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7217176Z return mod(**inputs) 2025-08-26T20:40:42.7217490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7217561Z outputs = self.model( 2025-08-26T20:40:42.7217833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7217916Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7218180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7218260Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7218485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7218574Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7218838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7218930Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7219210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7219323Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7219327Z 2025-08-26T20:40:42.7219421Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7219508Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7219590Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7219679Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7219788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7220005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7220097Z return mod(**inputs) 2025-08-26T20:40:42.7220380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7220460Z outputs = self.model( 2025-08-26T20:40:42.7220741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7220829Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7221107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7221183Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7221423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7221506Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7221792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7221884Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7222150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7222249Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7222542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7222682Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7222685Z 2025-08-26T20:40:42.7222787Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7222999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7223068Z return mod(**inputs) 2025-08-26T20:40:42.7223351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7223449Z outputs = self.model( 2025-08-26T20:40:42.7223733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7223831Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7224097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7224178Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7224401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7224480Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7224753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7224852Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7225143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7225247Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7225557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7225694Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7225698Z 2025-08-26T20:40:42.7225801Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7226007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7226074Z return mod(**inputs) 2025-08-26T20:40:42.7226349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7226444Z outputs = self.model( 2025-08-26T20:40:42.7226710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7226792Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7227056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7227137Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7227359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7227440Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7227709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7227800Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7228066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7228149Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7228154Z 2025-08-26T20:40:42.7228267Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7228466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7228538Z return mod(**inputs) 2025-08-26T20:40:42.7228828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7228899Z outputs = self.model( 2025-08-26T20:40:42.7229184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7229263Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7229537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7229620Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7229872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7229963Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7230258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7230390Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7230401Z 2025-08-26T20:40:42.7230510Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7230719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7230796Z return mod(**inputs) 2025-08-26T20:40:42.7231084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7231160Z outputs = self.model( 2025-08-26T20:40:42.7231425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7231498Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7231769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7231860Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7232090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7232169Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7232432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7232559Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7232790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7232871Z return self.act(input) 2025-08-26T20:40:42.7232874Z 2025-08-26T20:40:42.7232977Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7233180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7233249Z return mod(**inputs) 2025-08-26T20:40:42.7233514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7233589Z outputs = self.model( 2025-08-26T20:40:42.7233855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7233936Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7234196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7234271Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7234501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7234581Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7234859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7234947Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7234951Z 2025-08-26T20:40:42.7235059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7235272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7235342Z return mod(**inputs) 2025-08-26T20:40:42.7235630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7235706Z outputs = self.model( 2025-08-26T20:40:42.7236000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7236091Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7236369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7236452Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7236680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7236770Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7237071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7237168Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7237480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7237642Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7237648Z 2025-08-26T20:40:42.7237765Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7237981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7238078Z return mod(**inputs) 2025-08-26T20:40:42.7238361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7238432Z outputs = self.model( 2025-08-26T20:40:42.7238721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7238798Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7239085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7239179Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7239417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7239583Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7239875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7239983Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7240265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7240349Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7240360Z 2025-08-26T20:40:42.7240471Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7240684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7240765Z return mod(**inputs) 2025-08-26T20:40:42.7241055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7241135Z outputs = self.model( 2025-08-26T20:40:42.7241425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7241504Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7241797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7241873Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7242121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7242204Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7242507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7242614Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7242922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7243022Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7243044Z 2025-08-26T20:40:42.7243134Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7243227Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7243311Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7243393Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7243510Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7243721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7243799Z return mod(**inputs) 2025-08-26T20:40:42.7244080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7244154Z outputs = self.model( 2025-08-26T20:40:42.7244448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7244527Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7244816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7244912Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7245147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7245238Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7245516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7245620Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7245916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7246022Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7246335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7246479Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7246483Z 2025-08-26T20:40:42.7246599Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7246809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7246886Z return mod(**inputs) 2025-08-26T20:40:42.7247168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7247243Z outputs = self.model( 2025-08-26T20:40:42.7247531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7247611Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7247895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7247967Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7248180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7248263Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7248522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7248619Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7248882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7248989Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7249299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7249427Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7249431Z 2025-08-26T20:40:42.7249547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7249752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7249830Z return mod(**inputs) 2025-08-26T20:40:42.7250112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7250190Z outputs = self.model( 2025-08-26T20:40:42.7250482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7250559Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7250831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7250904Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7251135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7251242Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7251507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7251608Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7251872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7251963Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7251984Z 2025-08-26T20:40:42.7252089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7252289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7252362Z return mod(**inputs) 2025-08-26T20:40:42.7252631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7252707Z outputs = self.model( 2025-08-26T20:40:42.7252973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7253053Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7253317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7253389Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7253627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7253705Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7253968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7254085Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7254091Z 2025-08-26T20:40:42.7254194Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7254395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7254460Z return mod(**inputs) 2025-08-26T20:40:42.7254725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7254794Z outputs = self.model( 2025-08-26T20:40:42.7255057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7255140Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7255422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7255509Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7255748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7255838Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7256101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7256219Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7256441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7256510Z return self.act(input) 2025-08-26T20:40:42.7256516Z 2025-08-26T20:40:42.7256627Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7256829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7256895Z return mod(**inputs) 2025-08-26T20:40:42.7257173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7257260Z outputs = self.model( 2025-08-26T20:40:42.7257533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7257607Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7257876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7257949Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7258173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7258278Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7258547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7258636Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7258639Z 2025-08-26T20:40:42.7258744Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7258945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7259019Z return mod(**inputs) 2025-08-26T20:40:42.7259302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7259381Z outputs = self.model( 2025-08-26T20:40:42.7259666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7259745Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7260036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7260111Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7260356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7260441Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7260730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-26T20:40:42.7260815Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7260818Z 2025-08-26T20:40:42.7260928Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7261146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7261215Z return mod(**inputs) 2025-08-26T20:40:42.7261509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7261599Z outputs = self.model( 2025-08-26T20:40:42.7261882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7261986Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7262278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7262357Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7262580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7262667Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7262932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7263028Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7263298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7263449Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7263454Z 2025-08-26T20:40:42.7263571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7263800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7263869Z return mod(**inputs) 2025-08-26T20:40:42.7264162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7264235Z outputs = self.model( 2025-08-26T20:40:42.7264537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7264636Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7264933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7265011Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7265254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7265347Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7265630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7265731Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7265998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7266076Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7266080Z 2025-08-26T20:40:42.7266192Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7266399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7266472Z return mod(**inputs) 2025-08-26T20:40:42.7266746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7266815Z outputs = self.model( 2025-08-26T20:40:42.7267095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7267168Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7267446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7267519Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7267760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7267841Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7268126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7268227Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7268539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7268638Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7268642Z 2025-08-26T20:40:42.7268724Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7268804Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7268891Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7268971Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7269087Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7269296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7269367Z return mod(**inputs) 2025-08-26T20:40:42.7269656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7269728Z outputs = self.model( 2025-08-26T20:40:42.7270020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7270120Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7270411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7270487Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7270723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7270815Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7271114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7271216Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7271482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7271581Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7271887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7272024Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7272028Z 2025-08-26T20:40:42.7272138Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7272337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7272409Z return mod(**inputs) 2025-08-26T20:40:42.7272679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7272751Z outputs = self.model( 2025-08-26T20:40:42.7273023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7273099Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7273369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7273441Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7273675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7273767Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7274058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7274162Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7274470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7274576Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7274918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7275040Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7275044Z 2025-08-26T20:40:42.7275163Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7275378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7275457Z return mod(**inputs) 2025-08-26T20:40:42.7275741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7275821Z outputs = self.model( 2025-08-26T20:40:42.7276119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7276201Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7276489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7276585Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7276828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7276920Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7277223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-26T20:40:42.7277328Z hidden_states, attn_weights = self.self_attn( 2025-08-26T20:40:42.7277631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7277741Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7277746Z 2025-08-26T20:40:42.7277857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7278074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7278154Z return mod(**inputs) 2025-08-26T20:40:42.7278441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7278524Z outputs = self.model( 2025-08-26T20:40:42.7278807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7278884Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7279179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7279262Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7279594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7279685Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7279977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7280120Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7280124Z 2025-08-26T20:40:42.7280237Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7280461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7280534Z return mod(**inputs) 2025-08-26T20:40:42.7280831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7280918Z outputs = self.model( 2025-08-26T20:40:42.7281250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7281345Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7281647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7281739Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7281978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7282065Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7282376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-26T20:40:42.7282506Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7282749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7282826Z return self.act(input) 2025-08-26T20:40:42.7282830Z 2025-08-26T20:40:42.7282952Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7283169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7283240Z return mod(**inputs) 2025-08-26T20:40:42.7283564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7283637Z outputs = self.model( 2025-08-26T20:40:42.7283936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-26T20:40:42.7284015Z encoder_outputs = self.encoder( 2025-08-26T20:40:42.7284304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-26T20:40:42.7284410Z layer_outputs = encoder_layer( 2025-08-26T20:40:42.7284653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7284748Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7285040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-26T20:40:42.7285138Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7285143Z 2025-08-26T20:40:42.7285255Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7285472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7285551Z return mod(**inputs) 2025-08-26T20:40:42.7285844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7285927Z outputs = self.model( 2025-08-26T20:40:42.7286218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7286300Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7286602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7286684Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7286930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7287018Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7287305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7287428Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7287717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7287896Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7287916Z 2025-08-26T20:40:42.7288034Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7288278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7288355Z return mod(**inputs) 2025-08-26T20:40:42.7288646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7288728Z outputs = self.model( 2025-08-26T20:40:42.7289016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7289104Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7289394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7289474Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7289723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7289808Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7290101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7290230Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7290523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7290610Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7290614Z 2025-08-26T20:40:42.7290725Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7290947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7291037Z return mod(**inputs) 2025-08-26T20:40:42.7291336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7291411Z outputs = self.model( 2025-08-26T20:40:42.7291700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7291790Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7292081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7292167Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7292411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7292498Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7292797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7292918Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7293204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7293298Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7293303Z 2025-08-26T20:40:42.7293397Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7293483Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7293566Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7293654Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7293764Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7293981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7294053Z return mod(**inputs) 2025-08-26T20:40:42.7294341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7294424Z outputs = self.model( 2025-08-26T20:40:42.7294730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7294817Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7295653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7295739Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7295989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7296075Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7296520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7296635Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7296936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7297044Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7297360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7297577Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7297582Z 2025-08-26T20:40:42.7297694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7297916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7297987Z return mod(**inputs) 2025-08-26T20:40:42.7298272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7298385Z outputs = self.model( 2025-08-26T20:40:42.7298668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7298756Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7299040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7299127Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7299363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7299448Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7299740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7299846Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7300133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7300240Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7300550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7300679Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7300684Z 2025-08-26T20:40:42.7300796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7301011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7301081Z return mod(**inputs) 2025-08-26T20:40:42.7301367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7301439Z outputs = self.model( 2025-08-26T20:40:42.7301718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7301805Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7302152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7302239Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7302501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7302590Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7302878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7302982Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7303274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7303362Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7303368Z 2025-08-26T20:40:42.7303486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7303699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7303770Z return mod(**inputs) 2025-08-26T20:40:42.7304065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7304159Z outputs = self.model( 2025-08-26T20:40:42.7304448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7304527Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7304806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7304892Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7305128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7305244Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7305537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7305660Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7305965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7306123Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7306127Z 2025-08-26T20:40:42.7306244Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7306461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7306538Z return mod(**inputs) 2025-08-26T20:40:42.7306827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7306901Z outputs = self.model( 2025-08-26T20:40:42.7307205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7307282Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7307569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7307645Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7307886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7307979Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7308266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7308394Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7308703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7308798Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7308802Z 2025-08-26T20:40:42.7308930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7309152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7309230Z return mod(**inputs) 2025-08-26T20:40:42.7309520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7309603Z outputs = self.model( 2025-08-26T20:40:42.7309898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7309976Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7310267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7310344Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7310586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7310670Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7310981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7311101Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7311392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7311495Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7311499Z 2025-08-26T20:40:42.7311587Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7311700Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7311786Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7311873Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7311999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7312218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7312298Z return mod(**inputs) 2025-08-26T20:40:42.7312587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7312662Z outputs = self.model( 2025-08-26T20:40:42.7312957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7313039Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7313332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7313414Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7313659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7313753Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7314041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7314165Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7314453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7314570Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7314891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7315036Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7315042Z 2025-08-26T20:40:42.7315163Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7315399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7315482Z return mod(**inputs) 2025-08-26T20:40:42.7315789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7315867Z outputs = self.model( 2025-08-26T20:40:42.7316164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7316243Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7316540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7316620Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7316876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7316964Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7317253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7317380Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7317694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7317807Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7318127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7318246Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7318258Z 2025-08-26T20:40:42.7318371Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7318612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7318693Z return mod(**inputs) 2025-08-26T20:40:42.7318987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7319070Z outputs = self.model( 2025-08-26T20:40:42.7319359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7319692Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7320001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7320082Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7320333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7320424Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7320711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7320838Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7321126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7321228Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7321233Z 2025-08-26T20:40:42.7321344Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7321565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7321637Z return mod(**inputs) 2025-08-26T20:40:42.7321925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7322010Z outputs = self.model( 2025-08-26T20:40:42.7322331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7322424Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7322731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7322814Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7323072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7323159Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7323454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7323584Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7323588Z 2025-08-26T20:40:42.7323710Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7323928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7324001Z return mod(**inputs) 2025-08-26T20:40:42.7324301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7324377Z outputs = self.model( 2025-08-26T20:40:42.7324688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7324768Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7325052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7325138Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7325379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7325500Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7325797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7325926Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7326169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7326256Z return self.act(input) 2025-08-26T20:40:42.7326260Z 2025-08-26T20:40:42.7326370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7326572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7326643Z return mod(**inputs) 2025-08-26T20:40:42.7326908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7326976Z outputs = self.model( 2025-08-26T20:40:42.7327266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7327346Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7327636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7327713Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7327956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7328049Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7328351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7328444Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7328448Z 2025-08-26T20:40:42.7328558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7328770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7328849Z return mod(**inputs) 2025-08-26T20:40:42.7329146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7329228Z outputs = self.model( 2025-08-26T20:40:42.7329521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7329611Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7329896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7329973Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7330218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7330305Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7330598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7330705Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7330989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7331167Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7331171Z 2025-08-26T20:40:42.7331275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7331482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7331550Z return mod(**inputs) 2025-08-26T20:40:42.7331823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7331911Z outputs = self.model( 2025-08-26T20:40:42.7332191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7332277Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7332561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7332646Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7332890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7332974Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7333264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7333371Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7333660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7333747Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7333751Z 2025-08-26T20:40:42.7333879Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7334079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7334147Z return mod(**inputs) 2025-08-26T20:40:42.7334419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7334486Z outputs = self.model( 2025-08-26T20:40:42.7334758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7334831Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7335100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7335181Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7335419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7335507Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7335789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7335892Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7336165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7336253Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7336256Z 2025-08-26T20:40:42.7336346Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7336428Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7336513Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7336591Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7336694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7336903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7336969Z return mod(**inputs) 2025-08-26T20:40:42.7337245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7337329Z outputs = self.model( 2025-08-26T20:40:42.7337595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7337678Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7337943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7338023Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7338262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7338344Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7338620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7338724Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7339000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7339102Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7339404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7339539Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7339543Z 2025-08-26T20:40:42.7339649Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7339861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7339932Z return mod(**inputs) 2025-08-26T20:40:42.7340210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7340282Z outputs = self.model( 2025-08-26T20:40:42.7340555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7340640Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7340908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7340991Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7341215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7341300Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7341611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7341719Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7342032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7342134Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7342436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7342546Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7342550Z 2025-08-26T20:40:42.7342655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7342863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7342932Z return mod(**inputs) 2025-08-26T20:40:42.7343211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7343280Z outputs = self.model( 2025-08-26T20:40:42.7343545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7343647Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7343911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7343994Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7344219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7344305Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7344573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7344688Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7344963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7345052Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7345056Z 2025-08-26T20:40:42.7345172Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7345385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7345458Z return mod(**inputs) 2025-08-26T20:40:42.7345750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7345818Z outputs = self.model( 2025-08-26T20:40:42.7346093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7346169Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7346443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7346517Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7346742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7346832Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7347111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7347234Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7347513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7347670Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7347684Z 2025-08-26T20:40:42.7347793Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7348027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7348107Z return mod(**inputs) 2025-08-26T20:40:42.7348404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7348489Z outputs = self.model( 2025-08-26T20:40:42.7348770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7348848Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7349135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7349213Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7349456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7349541Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7349837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7349962Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7350265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7350352Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7350356Z 2025-08-26T20:40:42.7350459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7350666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7350732Z return mod(**inputs) 2025-08-26T20:40:42.7350996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7351091Z outputs = self.model( 2025-08-26T20:40:42.7351372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7351458Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7351744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7351821Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7352067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7352150Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7352486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7352595Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7352866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7352959Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7352963Z 2025-08-26T20:40:42.7353043Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7353134Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7353213Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7353297Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7353402Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7353601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7353674Z return mod(**inputs) 2025-08-26T20:40:42.7353954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7354033Z outputs = self.model( 2025-08-26T20:40:42.7354333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7354418Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7354723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7354802Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7355052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7355135Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7355416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7355536Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7355827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7355940Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7356252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7356398Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7356423Z 2025-08-26T20:40:42.7356536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7356750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7356828Z return mod(**inputs) 2025-08-26T20:40:42.7357126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7357206Z outputs = self.model( 2025-08-26T20:40:42.7357503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7357603Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7357906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7357983Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7358230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7358315Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7358613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7358728Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7359065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7359180Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7359566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7359699Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7359703Z 2025-08-26T20:40:42.7359815Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7360027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7360108Z return mod(**inputs) 2025-08-26T20:40:42.7360396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7360480Z outputs = self.model( 2025-08-26T20:40:42.7360772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7360861Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7361156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7361265Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7361530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7361633Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7361929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7362045Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7362325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7362421Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7362425Z 2025-08-26T20:40:42.7362534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7362757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7362828Z return mod(**inputs) 2025-08-26T20:40:42.7363110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7363190Z outputs = self.model( 2025-08-26T20:40:42.7363490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7363575Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7363855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7363940Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7364174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7364278Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7364571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7364701Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7364706Z 2025-08-26T20:40:42.7364827Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7365045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7365119Z return mod(**inputs) 2025-08-26T20:40:42.7365414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7365491Z outputs = self.model( 2025-08-26T20:40:42.7365785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7365866Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7366161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7366242Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7366482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7366578Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7366865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7367005Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7367236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7367313Z return self.act(input) 2025-08-26T20:40:42.7367317Z 2025-08-26T20:40:42.7367437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7367651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7367733Z return mod(**inputs) 2025-08-26T20:40:42.7368036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7368113Z outputs = self.model( 2025-08-26T20:40:42.7368424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7368505Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7368795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7368872Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7369116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7369200Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7369482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7369585Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7369589Z 2025-08-26T20:40:42.7369690Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7369898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7369980Z return mod(**inputs) 2025-08-26T20:40:42.7370250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7370336Z outputs = self.model( 2025-08-26T20:40:42.7370597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7370675Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7370952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7371029Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7371243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7371320Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7371585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:42.7371663Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7371667Z 2025-08-26T20:40:42.7371772Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7371962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7372025Z return mod(**inputs) 2025-08-26T20:40:42.7372305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7372378Z outputs = self.model( 2025-08-26T20:40:42.7372662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7372738Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7373017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7373101Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7373337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7373427Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7373703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7373817Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7374095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7374272Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7374276Z 2025-08-26T20:40:42.7374422Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7374637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7374713Z return mod(**inputs) 2025-08-26T20:40:42.7374997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7375071Z outputs = self.model( 2025-08-26T20:40:42.7375359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7375436Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7375723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7375796Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7376028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7376109Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7376397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7376506Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7376784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7376876Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7376880Z 2025-08-26T20:40:42.7376990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7377222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7377311Z return mod(**inputs) 2025-08-26T20:40:42.7377578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7377655Z outputs = self.model( 2025-08-26T20:40:42.7377922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7378008Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7378289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7378365Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7378619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7378699Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7378975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7379073Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7379347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7379441Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7379445Z 2025-08-26T20:40:42.7379525Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7379612Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7379688Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7379763Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7379872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7380066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7380139Z return mod(**inputs) 2025-08-26T20:40:42.7380421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7380496Z outputs = self.model( 2025-08-26T20:40:42.7380803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7380884Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7381175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7381248Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7381483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7381564Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7381829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7381938Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7382203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7382313Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7382627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7382787Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7382798Z 2025-08-26T20:40:42.7382908Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7383118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7383192Z return mod(**inputs) 2025-08-26T20:40:42.7383457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7383577Z outputs = self.model( 2025-08-26T20:40:42.7383845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7383918Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7384193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7384267Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7384498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7384577Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7384841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7384949Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7385217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7385324Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7385633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7385758Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7385761Z 2025-08-26T20:40:42.7385872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7386084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7386161Z return mod(**inputs) 2025-08-26T20:40:42.7386442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7386524Z outputs = self.model( 2025-08-26T20:40:42.7386813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7386905Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7387198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7387274Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7387504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7387584Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7387847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7387953Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7388224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7388313Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7388317Z 2025-08-26T20:40:42.7388423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7388631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7388698Z return mod(**inputs) 2025-08-26T20:40:42.7388980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7389056Z outputs = self.model( 2025-08-26T20:40:42.7389324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7389408Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7389679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7389771Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7390019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7390103Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7390392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7390511Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7390797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7390958Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7390962Z 2025-08-26T20:40:42.7391081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7391289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7391359Z return mod(**inputs) 2025-08-26T20:40:42.7391635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7391704Z outputs = self.model( 2025-08-26T20:40:42.7391973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7392055Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7392322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7392403Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7392626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7392713Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7392992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7393111Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7393419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7393535Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7393541Z 2025-08-26T20:40:42.7393652Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7393852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7393919Z return mod(**inputs) 2025-08-26T20:40:42.7394195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7394263Z outputs = self.model( 2025-08-26T20:40:42.7394534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7394611Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7394893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7394968Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7395190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7395298Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7395581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7395702Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7395980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7396072Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7396096Z 2025-08-26T20:40:42.7396346Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7396439Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7396532Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7396613Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7396724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7396946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7397015Z return mod(**inputs) 2025-08-26T20:40:42.7397308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7397382Z outputs = self.model( 2025-08-26T20:40:42.7397663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7397750Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7398029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7398118Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7398351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7398445Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7398726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7398841Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7399127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7399233Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7399603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7399752Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7399816Z 2025-08-26T20:40:42.7399937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7400184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7400260Z return mod(**inputs) 2025-08-26T20:40:42.7400563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7400638Z outputs = self.model( 2025-08-26T20:40:42.7400939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7401019Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7401309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7401400Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7401645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7401735Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7402000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7402140Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7402412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7402509Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7402807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7402915Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7402942Z 2025-08-26T20:40:42.7403052Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7403257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7403324Z return mod(**inputs) 2025-08-26T20:40:42.7403613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7403686Z outputs = self.model( 2025-08-26T20:40:42.7403973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7404050Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7404331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7404417Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7404658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7404753Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7405039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7405161Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7405453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7405537Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7405541Z 2025-08-26T20:40:42.7405653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7405852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7405928Z return mod(**inputs) 2025-08-26T20:40:42.7406212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7406286Z outputs = self.model( 2025-08-26T20:40:42.7406602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7406684Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7406983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7407063Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7407304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7407396Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7407680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7407818Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7407824Z 2025-08-26T20:40:42.7407935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7408158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7408229Z return mod(**inputs) 2025-08-26T20:40:42.7408515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7408612Z outputs = self.model( 2025-08-26T20:40:42.7408890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7408975Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7409259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7409337Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7409602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7409690Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7409976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7410103Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7410338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7410412Z return self.act(input) 2025-08-26T20:40:42.7410416Z 2025-08-26T20:40:42.7410525Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7410742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7410811Z return mod(**inputs) 2025-08-26T20:40:42.7411103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7411178Z outputs = self.model( 2025-08-26T20:40:42.7411460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7411544Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7411833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7411918Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7412159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7412243Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7412532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7412620Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7412625Z 2025-08-26T20:40:42.7412742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7412969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7413047Z return mod(**inputs) 2025-08-26T20:40:42.7413344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7413421Z outputs = self.model( 2025-08-26T20:40:42.7413711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7413788Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7414075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7414153Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7414390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7414488Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7414771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7414884Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7415166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7415355Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7415359Z 2025-08-26T20:40:42.7415469Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7415682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7415759Z return mod(**inputs) 2025-08-26T20:40:42.7416039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7416137Z outputs = self.model( 2025-08-26T20:40:42.7416425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7416503Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7416805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7416883Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7417131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7417216Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7417511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7417618Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7417911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7418006Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7418010Z 2025-08-26T20:40:42.7418120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7418343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7418414Z return mod(**inputs) 2025-08-26T20:40:42.7418699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7418779Z outputs = self.model( 2025-08-26T20:40:42.7419065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7419151Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7419445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7419542Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7419790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7419889Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7420180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7420286Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7420575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7420669Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7420673Z 2025-08-26T20:40:42.7420759Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7420859Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7420939Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7421024Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7421129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7421331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7421407Z return mod(**inputs) 2025-08-26T20:40:42.7421697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7421775Z outputs = self.model( 2025-08-26T20:40:42.7422040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7422118Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7422408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7422515Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7422762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7422850Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7423142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7423249Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7423542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7423649Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7423943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7424084Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7424089Z 2025-08-26T20:40:42.7424194Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7424397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7424472Z return mod(**inputs) 2025-08-26T20:40:42.7424742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7424821Z outputs = self.model( 2025-08-26T20:40:42.7425087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7425168Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7425433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7425506Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7425736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7425817Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7426109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7426226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7426494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7426602Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7426896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7427014Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7427018Z 2025-08-26T20:40:42.7427130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7427351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7427421Z return mod(**inputs) 2025-08-26T20:40:42.7427713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7427788Z outputs = self.model( 2025-08-26T20:40:42.7428060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7428160Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7428437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7428514Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7428755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7428857Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7429151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7429255Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7429536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7429633Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7429637Z 2025-08-26T20:40:42.7429748Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7429967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7430037Z return mod(**inputs) 2025-08-26T20:40:42.7430327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7430408Z outputs = self.model( 2025-08-26T20:40:42.7430676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7430758Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7431022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7431103Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7431327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7431405Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7431711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-26T20:40:42.7431796Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7431800Z 2025-08-26T20:40:42.7431917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7432131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7432208Z return mod(**inputs) 2025-08-26T20:40:42.7432509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7432583Z outputs = self.model( 2025-08-26T20:40:42.7432888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7432969Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7433271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7433346Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7433581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7433671Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7433967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7434092Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7434387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7434565Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7434574Z 2025-08-26T20:40:42.7434684Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7434895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7434971Z return mod(**inputs) 2025-08-26T20:40:42.7435294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7435391Z outputs = self.model( 2025-08-26T20:40:42.7435684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7435763Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7436061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7436140Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7436384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7436467Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7436766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7436892Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7437215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7437311Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7437315Z 2025-08-26T20:40:42.7437430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7437654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7437727Z return mod(**inputs) 2025-08-26T20:40:42.7438017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7438098Z outputs = self.model( 2025-08-26T20:40:42.7438385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7438472Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7438761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7438843Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7439113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7439201Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7439585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7439714Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7440024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7440120Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7440124Z 2025-08-26T20:40:42.7440212Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7440307Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7440392Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7440486Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7440600Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7440817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7440898Z return mod(**inputs) 2025-08-26T20:40:42.7441189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7441300Z outputs = self.model( 2025-08-26T20:40:42.7441596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7441675Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7441962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7442039Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7442284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7442386Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7442678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7442800Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7443077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7443190Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7443498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7443647Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7443651Z 2025-08-26T20:40:42.7443761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7443974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7444055Z return mod(**inputs) 2025-08-26T20:40:42.7444336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7444417Z outputs = self.model( 2025-08-26T20:40:42.7444699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7444777Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7445084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7445160Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7445402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7445487Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7445792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7445908Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7446204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7446321Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7446629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7446751Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7446755Z 2025-08-26T20:40:42.7446864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7447077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7447157Z return mod(**inputs) 2025-08-26T20:40:42.7447426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7447498Z outputs = self.model( 2025-08-26T20:40:42.7447749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7447853Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7448111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7448183Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7448414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7448491Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7448760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7448886Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7449149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7449240Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7449245Z 2025-08-26T20:40:42.7449348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7449553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7449619Z return mod(**inputs) 2025-08-26T20:40:42.7449886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7449953Z outputs = self.model( 2025-08-26T20:40:42.7450216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7450299Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7450585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7450662Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7450878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7450956Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7451218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7451334Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7451339Z 2025-08-26T20:40:42.7451447Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7451641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7451714Z return mod(**inputs) 2025-08-26T20:40:42.7451998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7452070Z outputs = self.model( 2025-08-26T20:40:42.7452367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7452441Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7452706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7452778Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7452995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7453079Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7453337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7453464Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7453678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7453749Z return self.act(input) 2025-08-26T20:40:42.7453761Z 2025-08-26T20:40:42.7453886Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7454086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7454159Z return mod(**inputs) 2025-08-26T20:40:42.7454430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7454506Z outputs = self.model( 2025-08-26T20:40:42.7454765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7454857Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7455126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7455198Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7455425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7455506Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7455764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7455852Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7455856Z 2025-08-26T20:40:42.7455957Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7456157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7456222Z return mod(**inputs) 2025-08-26T20:40:42.7456479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7456556Z outputs = self.model( 2025-08-26T20:40:42.7456816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7456897Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7457155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7457242Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7457471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7457554Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7457842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7457950Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7458257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7458438Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7458444Z 2025-08-26T20:40:42.7458560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7458771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7458841Z return mod(**inputs) 2025-08-26T20:40:42.7459129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7459201Z outputs = self.model( 2025-08-26T20:40:42.7459489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7459564Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7459829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7459910Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7460133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7460239Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7460502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7460605Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7460890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7460977Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7460998Z 2025-08-26T20:40:42.7461118Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7461331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7461410Z return mod(**inputs) 2025-08-26T20:40:42.7461696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7461770Z outputs = self.model( 2025-08-26T20:40:42.7462060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7462139Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7462430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7462504Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7462739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7462832Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7463115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7463227Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7463510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7463613Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7463617Z 2025-08-26T20:40:42.7463703Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7463788Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7463876Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7463957Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7464074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7464287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7464356Z return mod(**inputs) 2025-08-26T20:40:42.7464664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7464739Z outputs = self.model( 2025-08-26T20:40:42.7485490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7485758Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7486148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7486235Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7486508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7486605Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7486927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7487040Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7487313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7487462Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7487766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7487912Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7487921Z 2025-08-26T20:40:42.7488040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7488253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7488389Z return mod(**inputs) 2025-08-26T20:40:42.7488669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7488753Z outputs = self.model( 2025-08-26T20:40:42.7489026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7489114Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7489388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7489466Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7489707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7489791Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7490064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7490169Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7490434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7490544Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7490843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7490968Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7490972Z 2025-08-26T20:40:42.7491082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7491304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7491373Z return mod(**inputs) 2025-08-26T20:40:42.7491637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7491720Z outputs = self.model( 2025-08-26T20:40:42.7492011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7492097Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7492373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7492454Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7492702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7492791Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7493082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7493189Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7493479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7493570Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7493575Z 2025-08-26T20:40:42.7493689Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7493914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7494004Z return mod(**inputs) 2025-08-26T20:40:42.7494291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7494365Z outputs = self.model( 2025-08-26T20:40:42.7494648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7494736Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7495023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7495122Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7495357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7495440Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7495717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7495832Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7496106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7496523Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7496531Z 2025-08-26T20:40:42.7496646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7496875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7496949Z return mod(**inputs) 2025-08-26T20:40:42.7497236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7497322Z outputs = self.model( 2025-08-26T20:40:42.7497607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7497693Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7497974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7498051Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7498304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7498387Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7498733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7498846Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7499147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7499233Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7499237Z 2025-08-26T20:40:42.7499343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7499552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7499619Z return mod(**inputs) 2025-08-26T20:40:42.7499891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7499962Z outputs = self.model( 2025-08-26T20:40:42.7500229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7500316Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7500584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7500667Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7500936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7501020Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7501317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7501425Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7501697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7501812Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7501816Z 2025-08-26T20:40:42.7501909Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7501992Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7502070Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7502157Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7502263Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7502472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7502539Z return mod(**inputs) 2025-08-26T20:40:42.7502806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7502882Z outputs = self.model( 2025-08-26T20:40:42.7503152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7503244Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7503507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7503578Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7503808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7503891Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7504164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7504273Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7504547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7504646Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7504945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7505124Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7505128Z 2025-08-26T20:40:42.7505233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7505460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7505530Z return mod(**inputs) 2025-08-26T20:40:42.7505800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7505877Z outputs = self.model( 2025-08-26T20:40:42.7506146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7506228Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7506498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7506581Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7506809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7506890Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7507181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7507290Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7507559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7507658Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7507950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7508087Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7508091Z 2025-08-26T20:40:42.7508199Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7508410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7508479Z return mod(**inputs) 2025-08-26T20:40:42.7508761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7508830Z outputs = self.model( 2025-08-26T20:40:42.7509098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7509180Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7509447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7509528Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7509757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7509836Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7510113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7510224Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7510501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7510586Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7510590Z 2025-08-26T20:40:42.7510693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7510902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7510970Z return mod(**inputs) 2025-08-26T20:40:42.7511260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7511332Z outputs = self.model( 2025-08-26T20:40:42.7511618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7511697Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7511962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7512042Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7512265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7512352Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7512619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-08-26T20:40:42.7512702Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7512706Z 2025-08-26T20:40:42.7512818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7513020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7513098Z return mod(**inputs) 2025-08-26T20:40:42.7513405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7513487Z outputs = self.model( 2025-08-26T20:40:42.7513767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7513845Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7514132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7514227Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7514474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7514559Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7514842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7514983Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7514987Z 2025-08-26T20:40:42.7515097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7515316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7515385Z return mod(**inputs) 2025-08-26T20:40:42.7515666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7515749Z outputs = self.model( 2025-08-26T20:40:42.7516031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7516119Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7516418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7516504Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7516748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7516834Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7517132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7517267Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7517510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7517589Z return self.act(input) 2025-08-26T20:40:42.7517593Z 2025-08-26T20:40:42.7517728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7517954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7518043Z return mod(**inputs) 2025-08-26T20:40:42.7518345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7518420Z outputs = self.model( 2025-08-26T20:40:42.7518719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7518798Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7519089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7519177Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7519422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7519604Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7519907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7520024Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7520029Z 2025-08-26T20:40:42.7520153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7520373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7520453Z return mod(**inputs) 2025-08-26T20:40:42.7520749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7520824Z outputs = self.model( 2025-08-26T20:40:42.7521179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7521263Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7521558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7521639Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7521890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7521976Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7522265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7522385Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7522677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7522857Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7522861Z 2025-08-26T20:40:42.7522974Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7523195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7523277Z return mod(**inputs) 2025-08-26T20:40:42.7523567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7523649Z outputs = self.model( 2025-08-26T20:40:42.7523940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7524028Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7524314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7524397Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7524666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7524755Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7525071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7525183Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7525473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7525570Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7525574Z 2025-08-26T20:40:42.7525685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7525909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7525981Z return mod(**inputs) 2025-08-26T20:40:42.7526279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7526353Z outputs = self.model( 2025-08-26T20:40:42.7526646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7526753Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7527044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7527129Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7527373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7527459Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7527754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7527882Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7528181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7528278Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7528284Z 2025-08-26T20:40:42.7528384Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7528476Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7528563Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7528655Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7528771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7528994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7529078Z return mod(**inputs) 2025-08-26T20:40:42.7529375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7529462Z outputs = self.model( 2025-08-26T20:40:42.7529759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7529850Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7530145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7530230Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7530484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7530575Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7530874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7530985Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7531283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7531441Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7531786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7531945Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7531949Z 2025-08-26T20:40:42.7532072Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7532294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7532364Z return mod(**inputs) 2025-08-26T20:40:42.7532648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7532738Z outputs = self.model( 2025-08-26T20:40:42.7533008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7533090Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7533361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7533436Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7533685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7533765Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7534037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7534136Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7534409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7534526Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7534817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7534934Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7534940Z 2025-08-26T20:40:42.7535046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7535251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7535318Z return mod(**inputs) 2025-08-26T20:40:42.7535585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7535661Z outputs = self.model( 2025-08-26T20:40:42.7535927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7536009Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7536275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7536347Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7536576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7536658Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7536928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7537027Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7537297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7537381Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7537386Z 2025-08-26T20:40:42.7537489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7537713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7537786Z return mod(**inputs) 2025-08-26T20:40:42.7538098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7538174Z outputs = self.model( 2025-08-26T20:40:42.7538455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7538540Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7538818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7538903Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7539139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7539232Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7539520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7539635Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7539923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7540094Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7540099Z 2025-08-26T20:40:42.7540211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7540412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7540477Z return mod(**inputs) 2025-08-26T20:40:42.7540749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7540836Z outputs = self.model( 2025-08-26T20:40:42.7541114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7541187Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7541466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7541540Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7541769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7541857Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7542130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7542246Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7542517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7542599Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7542603Z 2025-08-26T20:40:42.7542714Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7542916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7542990Z return mod(**inputs) 2025-08-26T20:40:42.7543260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7543336Z outputs = self.model( 2025-08-26T20:40:42.7543606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7543680Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7543958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7544030Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7544280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7544377Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7544648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7544763Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7545032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7545125Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7545129Z 2025-08-26T20:40:42.7545212Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7545295Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7545381Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7545461Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7545572Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7545775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7545843Z return mod(**inputs) 2025-08-26T20:40:42.7546134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7546215Z outputs = self.model( 2025-08-26T20:40:42.7546481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7546555Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7546817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7546908Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7547150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7547246Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7547543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7547669Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7547953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7548057Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7548377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7548521Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7548526Z 2025-08-26T20:40:42.7548655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7548872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7548951Z return mod(**inputs) 2025-08-26T20:40:42.7549238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7549314Z outputs = self.model( 2025-08-26T20:40:42.7549613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7549688Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7549962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7550034Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7550258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7550347Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7550633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7550764Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7551031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7551129Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7551427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7551536Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7551540Z 2025-08-26T20:40:42.7551651Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7551854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7551927Z return mod(**inputs) 2025-08-26T20:40:42.7552196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7552268Z outputs = self.model( 2025-08-26T20:40:42.7552543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7552663Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7552939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7553013Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7553237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7553342Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7553608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7553721Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7553998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7554093Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7554097Z 2025-08-26T20:40:42.7554206Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7554417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7554495Z return mod(**inputs) 2025-08-26T20:40:42.7554779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7554859Z outputs = self.model( 2025-08-26T20:40:42.7555140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7555221Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7555515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7555594Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7555843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7555929Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7556240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7556374Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7556378Z 2025-08-26T20:40:42.7556490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7556719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7556792Z return mod(**inputs) 2025-08-26T20:40:42.7557107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7557202Z outputs = self.model( 2025-08-26T20:40:42.7557496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7557585Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7557875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7557962Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7558207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7558296Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7558597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7558728Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7558972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7559073Z return self.act(input) 2025-08-26T20:40:42.7559077Z 2025-08-26T20:40:42.7559196Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7559417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7559577Z return mod(**inputs) 2025-08-26T20:40:42.7559886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7559962Z outputs = self.model( 2025-08-26T20:40:42.7560291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7560376Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7560668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7560759Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7561015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7561103Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7561360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7561450Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7561454Z 2025-08-26T20:40:42.7561558Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7561756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7561830Z return mod(**inputs) 2025-08-26T20:40:42.7562093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7562170Z outputs = self.model( 2025-08-26T20:40:42.7562435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7562510Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7562785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7562864Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7563113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7563202Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7563492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:42.7563619Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7563624Z 2025-08-26T20:40:42.7563740Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7563980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7564056Z return mod(**inputs) 2025-08-26T20:40:42.7564354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7564428Z outputs = self.model( 2025-08-26T20:40:42.7564716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7564804Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7565092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7565182Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7565427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7565513Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7565813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7565941Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7566238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7566404Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7566409Z 2025-08-26T20:40:42.7566528Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7566762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7566836Z return mod(**inputs) 2025-08-26T20:40:42.7567132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7567208Z outputs = self.model( 2025-08-26T20:40:42.7567505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7567588Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7567875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7567963Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7568204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7568297Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7568586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7568705Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7568999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7569087Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7569091Z 2025-08-26T20:40:42.7569207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7569415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7569491Z return mod(**inputs) 2025-08-26T20:40:42.7569775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7569849Z outputs = self.model( 2025-08-26T20:40:42.7570147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7570553Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7570844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7570938Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7571183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7571276Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7571561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7571674Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7571956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7572058Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7572062Z 2025-08-26T20:40:42.7572149Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7572234Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7572327Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7572409Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7572545Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7572758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7572828Z return mod(**inputs) 2025-08-26T20:40:42.7573120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7573192Z outputs = self.model( 2025-08-26T20:40:42.7573476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7573575Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7573859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7573944Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7574184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7574278Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7574559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7574674Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7574955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7575059Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7575382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7575526Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7575530Z 2025-08-26T20:40:42.7575648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7575863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7575936Z return mod(**inputs) 2025-08-26T20:40:42.7576227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7576302Z outputs = self.model( 2025-08-26T20:40:42.7576588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7576665Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7576957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7577033Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7577285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7577400Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7577686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7577799Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7578088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7578196Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7578524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7578655Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7578659Z 2025-08-26T20:40:42.7578780Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7578990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7579070Z return mod(**inputs) 2025-08-26T20:40:42.7579370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7579443Z outputs = self.model( 2025-08-26T20:40:42.7579731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7579808Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7580097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7580195Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7580432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7580524Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7580806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7580919Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7581198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7581284Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7581295Z 2025-08-26T20:40:42.7581405Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7581614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7581693Z return mod(**inputs) 2025-08-26T20:40:42.7581973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7582054Z outputs = self.model( 2025-08-26T20:40:42.7582333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7582411Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7582702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7582779Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7583024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7583107Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7583386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7583514Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7583808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7583978Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7583998Z 2025-08-26T20:40:42.7584111Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7584331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7584402Z return mod(**inputs) 2025-08-26T20:40:42.7584683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7584764Z outputs = self.model( 2025-08-26T20:40:42.7585048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7585134Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7585417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7585494Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7585738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7585844Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7586134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7586250Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7586536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7586621Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7586649Z 2025-08-26T20:40:42.7586760Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7586978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7587048Z return mod(**inputs) 2025-08-26T20:40:42.7587334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7587408Z outputs = self.model( 2025-08-26T20:40:42.7587689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7587773Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7588057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7588140Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7588382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7588470Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7588768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7588882Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7589164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7589255Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7589259Z 2025-08-26T20:40:42.7589351Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7589435Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7589516Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7589602Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7589708Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7589924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7589994Z return mod(**inputs) 2025-08-26T20:40:42.7590289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7590371Z outputs = self.model( 2025-08-26T20:40:42.7590670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7590758Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7591041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7591118Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7591364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7591447Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7591740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7591856Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7592148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7592273Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7592595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7592749Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7592753Z 2025-08-26T20:40:42.7592867Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7593092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7593185Z return mod(**inputs) 2025-08-26T20:40:42.7593479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7593563Z outputs = self.model( 2025-08-26T20:40:42.7593856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7593945Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7594235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7594320Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7594561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7594648Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7594957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7595078Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7595374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7595482Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7595804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7595933Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7595937Z 2025-08-26T20:40:42.7596051Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7596438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7596516Z return mod(**inputs) 2025-08-26T20:40:42.7596818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7596898Z outputs = self.model( 2025-08-26T20:40:42.7597245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7597338Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7597661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7597753Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7598000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7598087Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7598385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7598503Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7598798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7598891Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7598895Z 2025-08-26T20:40:42.7599010Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7599237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7599344Z return mod(**inputs) 2025-08-26T20:40:42.7599691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7599771Z outputs = self.model( 2025-08-26T20:40:42.7600071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7600154Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7600474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7600561Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7600817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7600912Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7601205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7601335Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7601347Z 2025-08-26T20:40:42.7601459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7601671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7601749Z return mod(**inputs) 2025-08-26T20:40:42.7602031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7602114Z outputs = self.model( 2025-08-26T20:40:42.7602401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7602478Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7602776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7602850Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7603080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7603161Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7603425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7603552Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7603770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7603865Z return self.act(input) 2025-08-26T20:40:42.7603869Z 2025-08-26T20:40:42.7603975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7604189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7604265Z return mod(**inputs) 2025-08-26T20:40:42.7604531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7604608Z outputs = self.model( 2025-08-26T20:40:42.7604873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7604951Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7605217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7605295Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7605541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7605625Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7605936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7606040Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7606045Z 2025-08-26T20:40:42.7606153Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7606378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7606448Z return mod(**inputs) 2025-08-26T20:40:42.7606735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7606844Z outputs = self.model( 2025-08-26T20:40:42.7607132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7607210Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7607494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7607580Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7607818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7607911Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7608203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7608310Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7608615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7608781Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7608785Z 2025-08-26T20:40:42.7608904Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7609127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7609204Z return mod(**inputs) 2025-08-26T20:40:42.7609471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7609540Z outputs = self.model( 2025-08-26T20:40:42.7609818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7609893Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7610169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7610242Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7610482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7610588Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7610855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7610964Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7611226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7611307Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7611318Z 2025-08-26T20:40:42.7611423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7611625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7611699Z return mod(**inputs) 2025-08-26T20:40:42.7611967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7612042Z outputs = self.model( 2025-08-26T20:40:42.7612308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7612406Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7612679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7612755Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7612983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7613063Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7613347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7613457Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7613727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7613823Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7613826Z 2025-08-26T20:40:42.7613908Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7613995Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7614073Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7614151Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7614261Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7614466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7614543Z return mod(**inputs) 2025-08-26T20:40:42.7614812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7614884Z outputs = self.model( 2025-08-26T20:40:42.7615162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7615237Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7615510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7615584Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7615821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7615916Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7616217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7616331Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7616631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7616737Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7617073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7617219Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7617223Z 2025-08-26T20:40:42.7617341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7617556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7617632Z return mod(**inputs) 2025-08-26T20:40:42.7617913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7617983Z outputs = self.model( 2025-08-26T20:40:42.7618257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7618332Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7618617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7618713Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7618960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7619049Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7619313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7619418Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7619700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7619804Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7620098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7620210Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7620213Z 2025-08-26T20:40:42.7620323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7620524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7620597Z return mod(**inputs) 2025-08-26T20:40:42.7620869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7620941Z outputs = self.model( 2025-08-26T20:40:42.7621229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7621307Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7621592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7621671Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7621912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7621996Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7622276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7622389Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7622670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7622766Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7622770Z 2025-08-26T20:40:42.7622898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7623112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7623189Z return mod(**inputs) 2025-08-26T20:40:42.7623489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7623571Z outputs = self.model( 2025-08-26T20:40:42.7623856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7623942Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7624228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7624306Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7624553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7624639Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7624930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-26T20:40:42.7625046Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7625050Z 2025-08-26T20:40:42.7625159Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7625380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7625450Z return mod(**inputs) 2025-08-26T20:40:42.7625740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7625812Z outputs = self.model( 2025-08-26T20:40:42.7626110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7626196Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7626480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7626565Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7626806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7626897Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7627178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7627294Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7627582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7627744Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7627748Z 2025-08-26T20:40:42.7627867Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7628081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7628152Z return mod(**inputs) 2025-08-26T20:40:42.7628441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7628514Z outputs = self.model( 2025-08-26T20:40:42.7628803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7628879Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7629166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7629245Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7629498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7629590Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7629889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7630014Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7630295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7630380Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7630384Z 2025-08-26T20:40:42.7630501Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7630714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7630794Z return mod(**inputs) 2025-08-26T20:40:42.7631076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7631158Z outputs = self.model( 2025-08-26T20:40:42.7631440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7631539Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7631830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7631906Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7632148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7632232Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7632509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7632651Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7632931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7633031Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7633035Z 2025-08-26T20:40:42.7633122Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7633210Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7633300Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7633381Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7633496Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7633710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7633780Z return mod(**inputs) 2025-08-26T20:40:42.7634066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7634141Z outputs = self.model( 2025-08-26T20:40:42.7634431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7634509Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7634796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7634876Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7635113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7635203Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7635480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7635600Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7635878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7636013Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7636372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7636521Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7636526Z 2025-08-26T20:40:42.7636646Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7636868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7636946Z return mod(**inputs) 2025-08-26T20:40:42.7637240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7637317Z outputs = self.model( 2025-08-26T20:40:42.7637618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7637701Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7637999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7638079Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7638340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7638433Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7638756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7638882Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7639207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7639341Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7639737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7639861Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7639867Z 2025-08-26T20:40:42.7639992Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7640209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7640290Z return mod(**inputs) 2025-08-26T20:40:42.7640583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7640658Z outputs = self.model( 2025-08-26T20:40:42.7640961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7641045Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7641345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7641426Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7641680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7641769Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7642061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7642186Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7642474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7642573Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7642579Z 2025-08-26T20:40:42.7642693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7642939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7643021Z return mod(**inputs) 2025-08-26T20:40:42.7643336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7643422Z outputs = self.model( 2025-08-26T20:40:42.7643716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7643794Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7644096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7644175Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7644444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7644533Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7644856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7644992Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7644995Z 2025-08-26T20:40:42.7645127Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7645354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7645426Z return mod(**inputs) 2025-08-26T20:40:42.7645717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7645792Z outputs = self.model( 2025-08-26T20:40:42.7646082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7646193Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7646483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7646568Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7646813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7646908Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7647274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7647409Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7647630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7647702Z return self.act(input) 2025-08-26T20:40:42.7647707Z 2025-08-26T20:40:42.7647817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7648019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7648086Z return mod(**inputs) 2025-08-26T20:40:42.7648362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7648432Z outputs = self.model( 2025-08-26T20:40:42.7648708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7648781Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7649043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7649122Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7649348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7649437Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7649718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7649809Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7649813Z 2025-08-26T20:40:42.7649931Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7650133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7650208Z return mod(**inputs) 2025-08-26T20:40:42.7650475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7650549Z outputs = self.model( 2025-08-26T20:40:42.7650813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7650887Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7651168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7651244Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7651491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7651599Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7651889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7651991Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7652257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7652416Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7652437Z 2025-08-26T20:40:42.7652541Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7652750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7652816Z return mod(**inputs) 2025-08-26T20:40:42.7653100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7653182Z outputs = self.model( 2025-08-26T20:40:42.7653468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7653565Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7653846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7653928Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7654164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7654251Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7654543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7654648Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7654936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7655023Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7655027Z 2025-08-26T20:40:42.7655138Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7655357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7655427Z return mod(**inputs) 2025-08-26T20:40:42.7655724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7655796Z outputs = self.model( 2025-08-26T20:40:42.7656080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7656163Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7656443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7656527Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7656748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7656836Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7657115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7657221Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7657523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7657611Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7657614Z 2025-08-26T20:40:42.7657704Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7657786Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7657866Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7657970Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7658078Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7658298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7658372Z return mod(**inputs) 2025-08-26T20:40:42.7658656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7658741Z outputs = self.model( 2025-08-26T20:40:42.7659041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7659122Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7659391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7659470Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7659698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7659778Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7660052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7660150Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7660423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7660524Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7660822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7660962Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7660967Z 2025-08-26T20:40:42.7661070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7661279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7661344Z return mod(**inputs) 2025-08-26T20:40:42.7661621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7661688Z outputs = self.model( 2025-08-26T20:40:42.7661961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7662048Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7662357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7662442Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7662701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7662788Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7663078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7663182Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7663465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7663568Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7663884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7664009Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7664012Z 2025-08-26T20:40:42.7664121Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7664338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7664430Z return mod(**inputs) 2025-08-26T20:40:42.7664719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7664792Z outputs = self.model( 2025-08-26T20:40:42.7665074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7665160Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7665440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7665543Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7665781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7665864Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7666155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7666262Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7666552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7666639Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7666642Z 2025-08-26T20:40:42.7666757Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7666970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7667046Z return mod(**inputs) 2025-08-26T20:40:42.7667338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7667411Z outputs = self.model( 2025-08-26T20:40:42.7667699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7667776Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7668057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7668142Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7668379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7668469Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7668755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7668895Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7669175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7669349Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7669355Z 2025-08-26T20:40:42.7669478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7669691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7669769Z return mod(**inputs) 2025-08-26T20:40:42.7670051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7670125Z outputs = self.model( 2025-08-26T20:40:42.7670414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7670495Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7670781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7670859Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7671125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7671209Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7671493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7671618Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7671902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7672014Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7672018Z 2025-08-26T20:40:42.7672130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7672341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7672422Z return mod(**inputs) 2025-08-26T20:40:42.7672705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7672785Z outputs = self.model( 2025-08-26T20:40:42.7673062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7673140Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7673428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7673506Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7673749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7673836Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7674121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7674237Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7674515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7674614Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7674618Z 2025-08-26T20:40:42.7674706Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7674800Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7674884Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7674963Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7675082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7675312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7675398Z return mod(**inputs) 2025-08-26T20:40:42.7675695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7675774Z outputs = self.model( 2025-08-26T20:40:42.7676063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7676141Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7676431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7676508Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7676753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7676840Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7677120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7677241Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7677521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7677652Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7677967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7678109Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7678113Z 2025-08-26T20:40:42.7678231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7678461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7678539Z return mod(**inputs) 2025-08-26T20:40:42.7678828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7678911Z outputs = self.model( 2025-08-26T20:40:42.7679207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7679290Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7679671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7679755Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7680009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7680097Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7680391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7680519Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7680823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7680938Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7681253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7681376Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7681380Z 2025-08-26T20:40:42.7681490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7681702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7681783Z return mod(**inputs) 2025-08-26T20:40:42.7682101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7682188Z outputs = self.model( 2025-08-26T20:40:42.7682494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7682578Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7682874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7682953Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7683206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7683294Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7683600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7683725Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7684014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7684110Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7684115Z 2025-08-26T20:40:42.7684251Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7684479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7684557Z return mod(**inputs) 2025-08-26T20:40:42.7684853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7684941Z outputs = self.model( 2025-08-26T20:40:42.7685234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7685348Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7685637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7685717Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7685969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7686055Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7686356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-08-26T20:40:42.7686444Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7686448Z 2025-08-26T20:40:42.7686567Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7686787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7686862Z return mod(**inputs) 2025-08-26T20:40:42.7687159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7687235Z outputs = self.model( 2025-08-26T20:40:42.7687532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7687615Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7687906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7687994Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7688246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7688339Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7688639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7688777Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7688789Z 2025-08-26T20:40:42.7688917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7689138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7689235Z return mod(**inputs) 2025-08-26T20:40:42.7689543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7689624Z outputs = self.model( 2025-08-26T20:40:42.7689910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7689987Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7690277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7690350Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7690579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7690654Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7690914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7691057Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7691268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7691347Z return self.act(input) 2025-08-26T20:40:42.7691351Z 2025-08-26T20:40:42.7691455Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7691663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7691730Z return mod(**inputs) 2025-08-26T20:40:42.7692005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7692086Z outputs = self.model( 2025-08-26T20:40:42.7692353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7692433Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7692699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7692769Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7692999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7693078Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7693363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7693452Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7693456Z 2025-08-26T20:40:42.7693573Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7693785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7693854Z return mod(**inputs) 2025-08-26T20:40:42.7694136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7694205Z outputs = self.model( 2025-08-26T20:40:42.7694474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7694548Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7694810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7694891Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7695119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7695230Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7695540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7695648Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7695950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7696110Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7696114Z 2025-08-26T20:40:42.7696370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7696593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7696673Z return mod(**inputs) 2025-08-26T20:40:42.7696940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7697012Z outputs = self.model( 2025-08-26T20:40:42.7697288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7697364Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7697696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7697767Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7697991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7698079Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7698344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7698481Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7698749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7698835Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7698840Z 2025-08-26T20:40:42.7698952Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7699169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7699247Z return mod(**inputs) 2025-08-26T20:40:42.7699541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7699615Z outputs = self.model( 2025-08-26T20:40:42.7699871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7699945Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7700211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7700282Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7700507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7700586Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7700842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7700944Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7701203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7701295Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7701299Z 2025-08-26T20:40:42.7701379Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7701463Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7701539Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7701636Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7701746Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7701964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7702045Z return mod(**inputs) 2025-08-26T20:40:42.7702330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7702404Z outputs = self.model( 2025-08-26T20:40:42.7702696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7702774Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7703066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7703147Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7703389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7703475Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7703743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7703868Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7704130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7704238Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7704533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7704688Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7704692Z 2025-08-26T20:40:42.7704806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7705007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7705081Z return mod(**inputs) 2025-08-26T20:40:42.7705363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7705438Z outputs = self.model( 2025-08-26T20:40:42.7705726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7705811Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7706082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7706156Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7706384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7706465Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7706730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7706837Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7707101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7707204Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7707497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7707607Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7707613Z 2025-08-26T20:40:42.7707724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7707939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7708014Z return mod(**inputs) 2025-08-26T20:40:42.7708311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7708389Z outputs = self.model( 2025-08-26T20:40:42.7708657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7708730Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7709004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7709079Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7709307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7709388Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7709654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7709761Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7710035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7710182Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7710185Z 2025-08-26T20:40:42.7710295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7710515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7710584Z return mod(**inputs) 2025-08-26T20:40:42.7710867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7710961Z outputs = self.model( 2025-08-26T20:40:42.7711236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7711317Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7711591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7711666Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7711906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7711985Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7712269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7712378Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7712651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7712812Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7712816Z 2025-08-26T20:40:42.7712920Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7713133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7713201Z return mod(**inputs) 2025-08-26T20:40:42.7713489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7713558Z outputs = self.model( 2025-08-26T20:40:42.7713832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7713912Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7714188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7714270Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7714524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7714608Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7714893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7715005Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7715275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7715357Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7715360Z 2025-08-26T20:40:42.7715469Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7715671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7715738Z return mod(**inputs) 2025-08-26T20:40:42.7716012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7716081Z outputs = self.model( 2025-08-26T20:40:42.7716355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7716445Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7716718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7716798Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7717030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7717120Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7717420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7717535Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7717826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7717915Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7717919Z 2025-08-26T20:40:42.7718010Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7718262Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7718352Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7718435Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7718547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7718791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7718863Z return mod(**inputs) 2025-08-26T20:40:42.7719156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7719231Z outputs = self.model( 2025-08-26T20:40:42.7719580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7719674Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7719960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7720046Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7720293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7720382Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7720699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7720828Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7721124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7721227Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7721546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7721685Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7721689Z 2025-08-26T20:40:42.7721796Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7722005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7722074Z return mod(**inputs) 2025-08-26T20:40:42.7722349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7722420Z outputs = self.model( 2025-08-26T20:40:42.7722687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7722774Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7723041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7723141Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7723365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7723450Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7723718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7723827Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7724123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7724219Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7724512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7724620Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7724624Z 2025-08-26T20:40:42.7724731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7724925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7724989Z return mod(**inputs) 2025-08-26T20:40:42.7725255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7725324Z outputs = self.model( 2025-08-26T20:40:42.7725598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7725672Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7725937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7726019Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7726243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7726331Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7726596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7726701Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7726974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7727059Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7727063Z 2025-08-26T20:40:42.7727191Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7727393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7727482Z return mod(**inputs) 2025-08-26T20:40:42.7727757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7727825Z outputs = self.model( 2025-08-26T20:40:42.7728088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7728159Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7728424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7728497Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7728714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7728801Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7729058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7729208Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7729211Z 2025-08-26T20:40:42.7729313Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7729513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7729577Z return mod(**inputs) 2025-08-26T20:40:42.7729833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7729906Z outputs = self.model( 2025-08-26T20:40:42.7730182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7730265Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7730528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7730601Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7730829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7730908Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7731178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7731297Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7731510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7731589Z return self.act(input) 2025-08-26T20:40:42.7731593Z 2025-08-26T20:40:42.7731697Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7731906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7731972Z return mod(**inputs) 2025-08-26T20:40:42.7732246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7732315Z outputs = self.model( 2025-08-26T20:40:42.7732581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7732660Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7732929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7733007Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7733232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7733325Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7733600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7733706Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7733711Z 2025-08-26T20:40:42.7733824Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7734024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7734101Z return mod(**inputs) 2025-08-26T20:40:42.7734373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7734446Z outputs = self.model( 2025-08-26T20:40:42.7734735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7734819Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7735113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7735202Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7735427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7735534Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7735797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-26T20:40:42.7735886Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7735889Z 2025-08-26T20:40:42.7735991Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7736190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7736279Z return mod(**inputs) 2025-08-26T20:40:42.7736558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7736638Z outputs = self.model( 2025-08-26T20:40:42.7736918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7737003Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7737281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7737360Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7737592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7737670Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7737942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7738046Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7738311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7738473Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7738477Z 2025-08-26T20:40:42.7738580Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7738785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7738851Z return mod(**inputs) 2025-08-26T20:40:42.7739127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7739195Z outputs = self.model( 2025-08-26T20:40:42.7739465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7739618Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7739900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7740000Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7740243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7740322Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7740598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7740698Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7740971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7741053Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7741057Z 2025-08-26T20:40:42.7741169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7741371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7741437Z return mod(**inputs) 2025-08-26T20:40:42.7741712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7741800Z outputs = self.model( 2025-08-26T20:40:42.7742090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7742167Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7742448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7742533Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7742786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7742879Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7743160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7743267Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7743554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7743646Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7743650Z 2025-08-26T20:40:42.7743744Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7743828Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7743917Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7743998Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7744107Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7744326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7744396Z return mod(**inputs) 2025-08-26T20:40:42.7744685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7744760Z outputs = self.model( 2025-08-26T20:40:42.7745039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7745122Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7745401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7745485Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7745722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7745808Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7746115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7746222Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7746530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7746639Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7746955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7747094Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7747099Z 2025-08-26T20:40:42.7747207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7747427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7747499Z return mod(**inputs) 2025-08-26T20:40:42.7747790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7747861Z outputs = self.model( 2025-08-26T20:40:42.7748143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7748247Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7748532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7748617Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7748853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7748938Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7749247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7749354Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7749642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7749749Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7750067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7750185Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7750189Z 2025-08-26T20:40:42.7750299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7750520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7750591Z return mod(**inputs) 2025-08-26T20:40:42.7750881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7750954Z outputs = self.model( 2025-08-26T20:40:42.7751234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7751322Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7751602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7751685Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7751920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7752012Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7752292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7752398Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7752698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7752783Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7752787Z 2025-08-26T20:40:42.7752913Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7753118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7753184Z return mod(**inputs) 2025-08-26T20:40:42.7753458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7753526Z outputs = self.model( 2025-08-26T20:40:42.7753801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7753875Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7754153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7754227Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7754451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7754556Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7754820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7754942Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7755220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7755380Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7755407Z 2025-08-26T20:40:42.7755519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7755735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7755813Z return mod(**inputs) 2025-08-26T20:40:42.7756098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7756179Z outputs = self.model( 2025-08-26T20:40:42.7756464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7756541Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7756829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7756906Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7757148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7757234Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7757514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7757639Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7757919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7758015Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7758019Z 2025-08-26T20:40:42.7758130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7758347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7758418Z return mod(**inputs) 2025-08-26T20:40:42.7758705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7758789Z outputs = self.model( 2025-08-26T20:40:42.7759099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7759182Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7759561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7759648Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7759895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7759981Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7760285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7760400Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7760687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7760784Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7760788Z 2025-08-26T20:40:42.7760870Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7760958Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7761037Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7761142Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7761246Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7761444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7761521Z return mod(**inputs) 2025-08-26T20:40:42.7761786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7761863Z outputs = self.model( 2025-08-26T20:40:42.7762144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7762220Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7762493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7762567Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7762796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7762877Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7763139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7763254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7763517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7763625Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7763920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7764061Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7764065Z 2025-08-26T20:40:42.7764171Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7764371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7764446Z return mod(**inputs) 2025-08-26T20:40:42.7764710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7764787Z outputs = self.model( 2025-08-26T20:40:42.7765049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7765125Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7765411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7765488Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7765733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7765818Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7766086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7766195Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7766457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7766565Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7766857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7766975Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7766978Z 2025-08-26T20:40:42.7767081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7767279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7767370Z return mod(**inputs) 2025-08-26T20:40:42.7767640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7767716Z outputs = self.model( 2025-08-26T20:40:42.7767985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7768066Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7768338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7768430Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7768667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7768747Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7769023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7769133Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7769399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7769489Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7769492Z 2025-08-26T20:40:42.7769596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7769803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7769870Z return mod(**inputs) 2025-08-26T20:40:42.7770142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7770217Z outputs = self.model( 2025-08-26T20:40:42.7770486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7770567Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7770832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7770912Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7771138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7771218Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7771493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7771632Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7771637Z 2025-08-26T20:40:42.7771751Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7771968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7772040Z return mod(**inputs) 2025-08-26T20:40:42.7772317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7772386Z outputs = self.model( 2025-08-26T20:40:42.7772659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7772733Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7773009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7773084Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7773308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7773397Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7773663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7773805Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7774020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7774090Z return self.act(input) 2025-08-26T20:40:42.7774094Z 2025-08-26T20:40:42.7774204Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7774403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7774493Z return mod(**inputs) 2025-08-26T20:40:42.7774759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7774830Z outputs = self.model( 2025-08-26T20:40:42.7775110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7775186Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7775473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7775550Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7775794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7775878Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7776172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7776265Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7776269Z 2025-08-26T20:40:42.7776378Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7776604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7776672Z return mod(**inputs) 2025-08-26T20:40:42.7776938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7777013Z outputs = self.model( 2025-08-26T20:40:42.7777277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7777356Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7777622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7777710Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7777965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7778051Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7778363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7778472Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7778772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7778932Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7778936Z 2025-08-26T20:40:42.7779045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7779266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7779337Z return mod(**inputs) 2025-08-26T20:40:42.7779626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7779699Z outputs = self.model( 2025-08-26T20:40:42.7779988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7780082Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7780369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7780454Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7780695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7780786Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7781100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7781209Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7781497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7781585Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7781591Z 2025-08-26T20:40:42.7781708Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7781921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7781991Z return mod(**inputs) 2025-08-26T20:40:42.7782282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7782354Z outputs = self.model( 2025-08-26T20:40:42.7782642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7782721Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7783009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7783087Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7783323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7783415Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7783716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7783827Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7784131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7784223Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7784235Z 2025-08-26T20:40:42.7784320Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7784421Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7784514Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7784594Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7784720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7784945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7785015Z return mod(**inputs) 2025-08-26T20:40:42.7785307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7785381Z outputs = self.model( 2025-08-26T20:40:42.7785667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7785746Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7786029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7786113Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7786350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7786460Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7786744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7786852Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7787166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7787272Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7787595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7787763Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7787766Z 2025-08-26T20:40:42.7787878Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7788085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7788154Z return mod(**inputs) 2025-08-26T20:40:42.7788431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7788499Z outputs = self.model( 2025-08-26T20:40:42.7788780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7788853Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7789122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7789204Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7789433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7789520Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7789790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7789892Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7790184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7790288Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7790609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7790729Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7790733Z 2025-08-26T20:40:42.7790871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7791086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7791156Z return mod(**inputs) 2025-08-26T20:40:42.7791458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7791529Z outputs = self.model( 2025-08-26T20:40:42.7791803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7791876Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7792142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7792220Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7792445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7792534Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7792799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-26T20:40:42.7792902Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:40:42.7793185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7793268Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7793272Z 2025-08-26T20:40:42.7793381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7793580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7793653Z return mod(**inputs) 2025-08-26T20:40:42.7793940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7794011Z outputs = self.model( 2025-08-26T20:40:42.7794285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7794360Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7794635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7794708Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7794937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7795017Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7795282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-26T20:40:42.7795374Z hidden_states = residual + hidden_states 2025-08-26T20:40:42.7795378Z 2025-08-26T20:40:42.7795482Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7795691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7795757Z return mod(**inputs) 2025-08-26T20:40:42.7796025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7796109Z outputs = self.model( 2025-08-26T20:40:42.7796545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7796637Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7796916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7796995Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7797243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7797375Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7797672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7797816Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7798109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-26T20:40:42.7798270Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-26T20:40:42.7798275Z 2025-08-26T20:40:42.7798386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7798608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7798677Z return mod(**inputs) 2025-08-26T20:40:42.7798967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7799041Z outputs = self.model( 2025-08-26T20:40:42.7799319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7799413Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7799798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7799888Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7800133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7800231Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7800525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7800681Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7800982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-26T20:40:42.7801073Z key_states = self.k_proj(current_states) 2025-08-26T20:40:42.7801077Z 2025-08-26T20:40:42.7801211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7801414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7801482Z return mod(**inputs) 2025-08-26T20:40:42.7801754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7801825Z outputs = self.model( 2025-08-26T20:40:42.7802097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7802174Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7802464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7802543Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7802788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7802882Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7803171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7803295Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7803602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-26T20:40:42.7803695Z value_states = self.v_proj(current_states) 2025-08-26T20:40:42.7803699Z 2025-08-26T20:40:42.7803797Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7803885Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7803979Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7804088Z cudagraph partition due to non gpu ops 2025-08-26T20:40:42.7804203Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7804449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7804527Z return mod(**inputs) 2025-08-26T20:40:42.7804829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7804904Z outputs = self.model( 2025-08-26T20:40:42.7805196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7805283Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7805574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7805661Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7805904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7805996Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7806286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7806430Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7806723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7806831Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7807153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-26T20:40:42.7807316Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:40:42.7807320Z 2025-08-26T20:40:42.7807442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7807658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7807732Z return mod(**inputs) 2025-08-26T20:40:42.7808032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7808108Z outputs = self.model( 2025-08-26T20:40:42.7808407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7808488Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7808780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7808870Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7809115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7809210Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7809499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7809617Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7809912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-26T20:40:42.7810018Z attn_output, attn_weights = attention_interface( 2025-08-26T20:40:42.7810342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-26T20:40:42.7810468Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:40:42.7810473Z 2025-08-26T20:40:42.7810592Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7810824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7810897Z return mod(**inputs) 2025-08-26T20:40:42.7811223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7811299Z outputs = self.model( 2025-08-26T20:40:42.7811588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7811665Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7811945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7812029Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7812267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7812363Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7812654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-26T20:40:42.7812775Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-26T20:40:42.7813066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-26T20:40:42.7813171Z attn_output = self.out_proj(attn_output) 2025-08-26T20:40:42.7813175Z 2025-08-26T20:40:42.7813293Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7813505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7813582Z return mod(**inputs) 2025-08-26T20:40:42.7813866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7813957Z outputs = self.model( 2025-08-26T20:40:42.7814247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7814324Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7814616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7814694Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7814942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7815032Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7815336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7815470Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7815475Z 2025-08-26T20:40:42.7815585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7815803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7815873Z return mod(**inputs) 2025-08-26T20:40:42.7816154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7816236Z outputs = self.model( 2025-08-26T20:40:42.7816515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7816601Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7816880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7816956Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7817215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7817301Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7817624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-26T20:40:42.7817751Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:40:42.7818004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:40:42.7818083Z return self.act(input) 2025-08-26T20:40:42.7818088Z 2025-08-26T20:40:42.7818198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7818416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7818486Z return mod(**inputs) 2025-08-26T20:40:42.7818774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-26T20:40:42.7818850Z outputs = self.model( 2025-08-26T20:40:42.7819133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-26T20:40:42.7819220Z decoder_outputs = self.decoder( 2025-08-26T20:40:42.7819503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-26T20:40:42.7819608Z layer_outputs = decoder_layer( 2025-08-26T20:40:42.7819843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:40:42.7819928Z return super().__call__(*args, **kwargs) 2025-08-26T20:40:42.7820239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-26T20:40:42.7820327Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:40:42.7820331Z 2025-08-26T20:40:42.7820476Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7820687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7820766Z return mod(**inputs) 2025-08-26T20:40:42.7821048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1489, in forward 2025-08-26T20:40:42.7821179Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-26T20:40:42.7821184Z 2025-08-26T20:40:42.7821304Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:40:42.7821514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:40:42.7821591Z return mod(**inputs) 2025-08-26T20:40:42.7821873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1494, in forward 2025-08-26T20:40:42.7822060Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:40:42.7822066Z 2025-08-26T20:40:55.1575658Z Compilation time (from dynamo_timed): 28.873710985 2025-08-26T20:40:55.1588514Z pass 2025-08-26T20:40:55.1588997Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:40:55.1589904Z TIMING: _recursive_pre_grad_passes:0.01555 _recursive_joint_graph_passes:1.17476 _recursive_post_grad_passes:0.16877 async_compile.wait:0.79976 code_gen:11.82876 inductor_compile:15.1151 backend_compile:22.72696 gc:0.00043 entire_frame_compile:28.87371 total_wall_time:28.87371 2025-08-26T20:40:55.1590920Z STATS: call_* op count: 965 | FakeTensorMode.__torch_dispatch__:33293 | FakeTensor.__torch_dispatch__:11091 | ProxyTorchDispatchMode.__torch_dispatch__:12299 2025-08-26T20:40:55.1591469Z Dynamo produced 1 graphs covering 965 ops with 0 graph breaks (0 unique) 2025-08-26T20:41:01.3190976Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:41:01.3193711Z from pkg_resources import resource_filename 2025-08-26T20:41:01.9551663Z 2025-08-26T20:41:01.9680481Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-08-26T20:41:01.9681191Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-08-26T20:41:03.4400292Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-26T20:41:03.4401266Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-26T20:41:03.4402274Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-26T20:41:03.4403300Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-26T20:41:03.6236161Z 2025-08-26T20:41:03.6237003Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:41:03.6251368Z cpu eval RobertaForCausalLM 2025-08-26T20:41:04.5194689Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:04.8893820Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:05.1793136Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:13.2667906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2671298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2671692Z return mod(**inputs) 2025-08-26T20:41:13.2675728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2676289Z outputs = self.roberta( 2025-08-26T20:41:13.2676760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-26T20:41:13.2677237Z embedding_output = self.embeddings( 2025-08-26T20:41:13.2677715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-26T20:41:13.2678318Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:41:13.2679037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-08-26T20:41:13.2679851Z mask = input_ids.ne(padding_idx).int() 2025-08-26T20:41:13.2680024Z 2025-08-26T20:41:13.2680117Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2680486Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2680775Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2681024Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2681274Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2681505Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2681721Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2681927Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2682141Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2682358Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2682587Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2682810Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2683407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2683819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2684186Z return mod(**inputs) 2025-08-26T20:41:13.2684691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2685136Z outputs = self.roberta( 2025-08-26T20:41:13.2685552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-26T20:41:13.2686010Z embedding_output = self.embeddings( 2025-08-26T20:41:13.2686446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-26T20:41:13.2687043Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:41:13.2687690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-26T20:41:13.2688315Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:41:13.2688655Z 2025-08-26T20:41:13.2688774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2689171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2689525Z return mod(**inputs) 2025-08-26T20:41:13.2689921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2690364Z outputs = self.roberta( 2025-08-26T20:41:13.2690815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-26T20:41:13.2691310Z embedding_output = self.embeddings( 2025-08-26T20:41:13.2691749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-26T20:41:13.2692313Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:41:13.2692951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-26T20:41:13.2693563Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:41:13.2693818Z 2025-08-26T20:41:13.2693933Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2694349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2694697Z return mod(**inputs) 2025-08-26T20:41:13.2695098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2695530Z outputs = self.roberta( 2025-08-26T20:41:13.2695926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2696563Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2696995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2697427Z layer_outputs = layer_module( 2025-08-26T20:41:13.2697816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2698232Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2698681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2699134Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2699605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2700004Z return func(*args, **kwargs) 2025-08-26T20:41:13.2700458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2700899Z self_outputs = self.self( 2025-08-26T20:41:13.2701292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2701690Z return func(*args, **kwargs) 2025-08-26T20:41:13.2702103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.2702693Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.2702989Z 2025-08-26T20:41:13.2703101Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2703499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2703857Z return mod(**inputs) 2025-08-26T20:41:13.2704247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2704698Z outputs = self.roberta( 2025-08-26T20:41:13.2705125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2705551Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2705985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2706412Z layer_outputs = layer_module( 2025-08-26T20:41:13.2706793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2707216Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2707673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2708125Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2708566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2708973Z return func(*args, **kwargs) 2025-08-26T20:41:13.2709400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2709840Z self_outputs = self.self( 2025-08-26T20:41:13.2710235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2710655Z return func(*args, **kwargs) 2025-08-26T20:41:13.2711082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.2711520Z self.key(current_states) 2025-08-26T20:41:13.2711650Z 2025-08-26T20:41:13.2711768Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2712172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2712535Z return mod(**inputs) 2025-08-26T20:41:13.2712946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2713395Z outputs = self.roberta( 2025-08-26T20:41:13.2713818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2714253Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2714690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2715132Z layer_outputs = layer_module( 2025-08-26T20:41:13.2715538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2715937Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2716397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2716845Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2717260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2717666Z return func(*args, **kwargs) 2025-08-26T20:41:13.2718089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2718525Z self_outputs = self.self( 2025-08-26T20:41:13.2718926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2719339Z return func(*args, **kwargs) 2025-08-26T20:41:13.2719868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.2720322Z self.value(current_states) 2025-08-26T20:41:13.2720481Z 2025-08-26T20:41:13.2720574Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2720847Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2721245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2721609Z return mod(**inputs) 2025-08-26T20:41:13.2722030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2722469Z outputs = self.roberta( 2025-08-26T20:41:13.2722911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2723338Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2723769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2724191Z layer_outputs = layer_module( 2025-08-26T20:41:13.2724568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2724963Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2725384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2725837Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2726246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2726649Z return func(*args, **kwargs) 2025-08-26T20:41:13.2727055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2727483Z self_outputs = self.self( 2025-08-26T20:41:13.2727880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2728287Z return func(*args, **kwargs) 2025-08-26T20:41:13.2728696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.2729188Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.2729398Z 2025-08-26T20:41:13.2729513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2729905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2730256Z return mod(**inputs) 2025-08-26T20:41:13.2730696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2731160Z outputs = self.roberta( 2025-08-26T20:41:13.2731582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2732013Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2732464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2732893Z layer_outputs = layer_module( 2025-08-26T20:41:13.2733270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2733665Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2734126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2734561Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2734973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2735370Z return func(*args, **kwargs) 2025-08-26T20:41:13.2735779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.2736274Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.2736723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.2737134Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2737286Z 2025-08-26T20:41:13.2737393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2737760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2738113Z return mod(**inputs) 2025-08-26T20:41:13.2738481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2738879Z outputs = self.roberta( 2025-08-26T20:41:13.2739286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2739742Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2740159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2740583Z layer_outputs = layer_module( 2025-08-26T20:41:13.2740967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2741338Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2741741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2742152Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2742595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2743027Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2743488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2743994Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2744462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.2744898Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2745056Z 2025-08-26T20:41:13.2745168Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2745555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2745908Z return mod(**inputs) 2025-08-26T20:41:13.2746316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2746740Z outputs = self.roberta( 2025-08-26T20:41:13.2747167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2747598Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2748016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2748411Z layer_outputs = layer_module( 2025-08-26T20:41:13.2748769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2749147Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2749580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2750013Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2750448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2750903Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2751363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2751871Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2752360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.2752839Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.2753282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.2753665Z return self.act(input) 2025-08-26T20:41:13.2753790Z 2025-08-26T20:41:13.2753915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2754319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2754681Z return mod(**inputs) 2025-08-26T20:41:13.2755093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2755525Z outputs = self.roberta( 2025-08-26T20:41:13.2755929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2756365Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2756796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2757245Z layer_outputs = layer_module( 2025-08-26T20:41:13.2757629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2758025Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2758472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2758932Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2759377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2759912Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2760388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.2760937Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.2761442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.2761903Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2762056Z 2025-08-26T20:41:13.2762183Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2762630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2763014Z return mod(**inputs) 2025-08-26T20:41:13.2763416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2763850Z outputs = self.roberta( 2025-08-26T20:41:13.2764245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2764675Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2765138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2765567Z layer_outputs = layer_module( 2025-08-26T20:41:13.2765949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2766334Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2766765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2767220Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2767635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2768030Z return func(*args, **kwargs) 2025-08-26T20:41:13.2768449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2768889Z self_outputs = self.self( 2025-08-26T20:41:13.2769277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2769689Z return func(*args, **kwargs) 2025-08-26T20:41:13.2770106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.2770688Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.2770968Z 2025-08-26T20:41:13.2771079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2771455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2771788Z return mod(**inputs) 2025-08-26T20:41:13.2772166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2772569Z outputs = self.roberta( 2025-08-26T20:41:13.2772961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2773371Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2773769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2774180Z layer_outputs = layer_module( 2025-08-26T20:41:13.2774544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2774927Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2775343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2775783Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2776200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2776619Z return func(*args, **kwargs) 2025-08-26T20:41:13.2777068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2777500Z self_outputs = self.self( 2025-08-26T20:41:13.2777886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2778265Z return func(*args, **kwargs) 2025-08-26T20:41:13.2778657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.2779080Z self.key(current_states) 2025-08-26T20:41:13.2779202Z 2025-08-26T20:41:13.2779316Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2779706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2780058Z return mod(**inputs) 2025-08-26T20:41:13.2780459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2780877Z outputs = self.roberta( 2025-08-26T20:41:13.2781272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2781690Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2782092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2782506Z layer_outputs = layer_module( 2025-08-26T20:41:13.2782879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2783261Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2783690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2784152Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2784562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2784958Z return func(*args, **kwargs) 2025-08-26T20:41:13.2785352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2785760Z self_outputs = self.self( 2025-08-26T20:41:13.2786132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2786534Z return func(*args, **kwargs) 2025-08-26T20:41:13.2786932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.2787418Z self.value(current_states) 2025-08-26T20:41:13.2787550Z 2025-08-26T20:41:13.2787646Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2787906Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2788285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2788635Z return mod(**inputs) 2025-08-26T20:41:13.2789052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2789469Z outputs = self.roberta( 2025-08-26T20:41:13.2789868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2790282Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2790738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2791168Z layer_outputs = layer_module( 2025-08-26T20:41:13.2791538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2791922Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2793141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2793643Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2794058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2794453Z return func(*args, **kwargs) 2025-08-26T20:41:13.2795055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2795487Z self_outputs = self.self( 2025-08-26T20:41:13.2795879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2796386Z return func(*args, **kwargs) 2025-08-26T20:41:13.2796801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.2797285Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.2797498Z 2025-08-26T20:41:13.2797616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2798021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2798468Z return mod(**inputs) 2025-08-26T20:41:13.2798881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2799320Z outputs = self.roberta( 2025-08-26T20:41:13.2799840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2800289Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2800759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2801171Z layer_outputs = layer_module( 2025-08-26T20:41:13.2801546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2801941Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2802374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2802807Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2803212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2803611Z return func(*args, **kwargs) 2025-08-26T20:41:13.2804021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.2804504Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.2804979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.2805412Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2805571Z 2025-08-26T20:41:13.2805686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2806076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2806426Z return mod(**inputs) 2025-08-26T20:41:13.2806813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2807224Z outputs = self.roberta( 2025-08-26T20:41:13.2807627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2808053Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2808495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2808921Z layer_outputs = layer_module( 2025-08-26T20:41:13.2809321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2809710Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2810138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2810566Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2810997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2811417Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2811877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2812391Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2812857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.2813293Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2813473Z 2025-08-26T20:41:13.2813587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2813974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2814321Z return mod(**inputs) 2025-08-26T20:41:13.2814714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2815136Z outputs = self.roberta( 2025-08-26T20:41:13.2815539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2815985Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2816405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2816821Z layer_outputs = layer_module( 2025-08-26T20:41:13.2817201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2817595Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2818024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2818448Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2818877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2819295Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2819752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2820259Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2820724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.2821189Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.2821599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.2821968Z return self.act(input) 2025-08-26T20:41:13.2822090Z 2025-08-26T20:41:13.2822213Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2822607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2822971Z return mod(**inputs) 2025-08-26T20:41:13.2823370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2823787Z outputs = self.roberta( 2025-08-26T20:41:13.2824199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2824644Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2825062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2825488Z layer_outputs = layer_module( 2025-08-26T20:41:13.2825860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2826241Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2826668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2827107Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2827538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2827957Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2828405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.2828941Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.2829442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.2829872Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2830019Z 2025-08-26T20:41:13.2830138Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2830517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2830888Z return mod(**inputs) 2025-08-26T20:41:13.2831289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2831706Z outputs = self.roberta( 2025-08-26T20:41:13.2832101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2832527Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2832941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2833361Z layer_outputs = layer_module( 2025-08-26T20:41:13.2833733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2834120Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2834568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2835005Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2835421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2835815Z return func(*args, **kwargs) 2025-08-26T20:41:13.2836231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2836651Z self_outputs = self.self( 2025-08-26T20:41:13.2837086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2837483Z return func(*args, **kwargs) 2025-08-26T20:41:13.2837881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.2838461Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.2838759Z 2025-08-26T20:41:13.2838874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2839312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2839815Z return mod(**inputs) 2025-08-26T20:41:13.2840236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2840659Z outputs = self.roberta( 2025-08-26T20:41:13.2841063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2841485Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2841897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2842326Z layer_outputs = layer_module( 2025-08-26T20:41:13.2842704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2843105Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2843545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2843984Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2844425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2844825Z return func(*args, **kwargs) 2025-08-26T20:41:13.2845231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2845643Z self_outputs = self.self( 2025-08-26T20:41:13.2846027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2846451Z return func(*args, **kwargs) 2025-08-26T20:41:13.2846863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.2847284Z self.key(current_states) 2025-08-26T20:41:13.2847405Z 2025-08-26T20:41:13.2847516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2847907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2848254Z return mod(**inputs) 2025-08-26T20:41:13.2848647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2849065Z outputs = self.roberta( 2025-08-26T20:41:13.2849463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2849889Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2850309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2850734Z layer_outputs = layer_module( 2025-08-26T20:41:13.2851106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2851492Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2851925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2852360Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2852749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2853130Z return func(*args, **kwargs) 2025-08-26T20:41:13.2853539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2853961Z self_outputs = self.self( 2025-08-26T20:41:13.2854391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2854775Z return func(*args, **kwargs) 2025-08-26T20:41:13.2855177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.2855582Z self.value(current_states) 2025-08-26T20:41:13.2855713Z 2025-08-26T20:41:13.2855796Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2856039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2856397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2856728Z return mod(**inputs) 2025-08-26T20:41:13.2857103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2857516Z outputs = self.roberta( 2025-08-26T20:41:13.2857921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2858343Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2858778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2859254Z layer_outputs = layer_module( 2025-08-26T20:41:13.2859626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2860027Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2860459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2860895Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2861314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2861746Z return func(*args, **kwargs) 2025-08-26T20:41:13.2862141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2862540Z self_outputs = self.self( 2025-08-26T20:41:13.2862905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2863284Z return func(*args, **kwargs) 2025-08-26T20:41:13.2863679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.2864156Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.2864363Z 2025-08-26T20:41:13.2864476Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2864877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2865208Z return mod(**inputs) 2025-08-26T20:41:13.2865577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2865972Z outputs = self.roberta( 2025-08-26T20:41:13.2866353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2866753Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2867144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2867563Z layer_outputs = layer_module( 2025-08-26T20:41:13.2867939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2868333Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2868757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2869176Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2869597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2870002Z return func(*args, **kwargs) 2025-08-26T20:41:13.2870429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.2870917Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.2871391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.2871826Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2871984Z 2025-08-26T20:41:13.2872095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2872486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2872839Z return mod(**inputs) 2025-08-26T20:41:13.2873232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2873656Z outputs = self.roberta( 2025-08-26T20:41:13.2874064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2874509Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2874922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2875333Z layer_outputs = layer_module( 2025-08-26T20:41:13.2875707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2876093Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2876539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2876976Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2877405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2877826Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2878286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2878791Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2879250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.2879777Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2879941Z 2025-08-26T20:41:13.2880057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2880455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2880821Z return mod(**inputs) 2025-08-26T20:41:13.2881228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2881653Z outputs = self.roberta( 2025-08-26T20:41:13.2882062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2882495Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2882915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2883335Z layer_outputs = layer_module( 2025-08-26T20:41:13.2883713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2884105Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2884571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2885003Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2885454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2885877Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2886329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2886830Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2887289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.2887750Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.2888158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.2888530Z return self.act(input) 2025-08-26T20:41:13.2888650Z 2025-08-26T20:41:13.2888772Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2889155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2889533Z return mod(**inputs) 2025-08-26T20:41:13.2889930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2890349Z outputs = self.roberta( 2025-08-26T20:41:13.2890740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2891163Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2891580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2892024Z layer_outputs = layer_module( 2025-08-26T20:41:13.2892400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2892782Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2893210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2893653Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2894060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2894455Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2894879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.2895373Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.2895832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.2896364Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2896512Z 2025-08-26T20:41:13.2896631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2896997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2897334Z return mod(**inputs) 2025-08-26T20:41:13.2897713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2898130Z outputs = self.roberta( 2025-08-26T20:41:13.2898530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2898958Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2899386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2899862Z layer_outputs = layer_module( 2025-08-26T20:41:13.2900238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2900657Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2901093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2901520Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2901914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2902288Z return func(*args, **kwargs) 2025-08-26T20:41:13.2902699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2903124Z self_outputs = self.self( 2025-08-26T20:41:13.2903514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2903908Z return func(*args, **kwargs) 2025-08-26T20:41:13.2904309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.2904908Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.2905197Z 2025-08-26T20:41:13.2905308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2905697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2906043Z return mod(**inputs) 2025-08-26T20:41:13.2906433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2906890Z outputs = self.roberta( 2025-08-26T20:41:13.2907295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2907719Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2908127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2908554Z layer_outputs = layer_module( 2025-08-26T20:41:13.2908928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2909318Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2909746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2910173Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2910587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2910984Z return func(*args, **kwargs) 2025-08-26T20:41:13.2911397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2911819Z self_outputs = self.self( 2025-08-26T20:41:13.2912198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2912594Z return func(*args, **kwargs) 2025-08-26T20:41:13.2913015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.2913452Z self.key(current_states) 2025-08-26T20:41:13.2913577Z 2025-08-26T20:41:13.2913693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2914103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2914454Z return mod(**inputs) 2025-08-26T20:41:13.2914877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2915300Z outputs = self.roberta( 2025-08-26T20:41:13.2915720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2916163Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2916593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2917029Z layer_outputs = layer_module( 2025-08-26T20:41:13.2917418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2917808Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2918268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2918714Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2919137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2919605Z return func(*args, **kwargs) 2025-08-26T20:41:13.2920036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2920501Z self_outputs = self.self( 2025-08-26T20:41:13.2920900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2921318Z return func(*args, **kwargs) 2025-08-26T20:41:13.2921730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.2922176Z self.value(current_states) 2025-08-26T20:41:13.2922334Z 2025-08-26T20:41:13.2922700Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2922962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2923372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2923755Z return mod(**inputs) 2025-08-26T20:41:13.2924183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2924631Z outputs = self.roberta( 2025-08-26T20:41:13.2925063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2925578Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2926025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2926475Z layer_outputs = layer_module( 2025-08-26T20:41:13.2926871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2927288Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2927741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2928203Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2928634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2929035Z return func(*args, **kwargs) 2025-08-26T20:41:13.2929452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2929893Z self_outputs = self.self( 2025-08-26T20:41:13.2930289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2930698Z return func(*args, **kwargs) 2025-08-26T20:41:13.2931145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.2931627Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.2931851Z 2025-08-26T20:41:13.2931984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2932376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2932724Z return mod(**inputs) 2025-08-26T20:41:13.2933123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2933537Z outputs = self.roberta( 2025-08-26T20:41:13.2933944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2934372Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2934767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2935161Z layer_outputs = layer_module( 2025-08-26T20:41:13.2935520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2935911Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2936337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2936788Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2937193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2937602Z return func(*args, **kwargs) 2025-08-26T20:41:13.2938011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.2938553Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.2939032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.2939458Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2939616Z 2025-08-26T20:41:13.2939728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2940116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2940465Z return mod(**inputs) 2025-08-26T20:41:13.2940857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2941276Z outputs = self.roberta( 2025-08-26T20:41:13.2941677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2942110Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2942541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2942954Z layer_outputs = layer_module( 2025-08-26T20:41:13.2943331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2943722Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2944153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2944590Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2945011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2945434Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2945893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2946440Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2946926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.2947367Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2947523Z 2025-08-26T20:41:13.2947637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2948031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2948378Z return mod(**inputs) 2025-08-26T20:41:13.2948767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2949189Z outputs = self.roberta( 2025-08-26T20:41:13.2949596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2950017Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2950433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2950849Z layer_outputs = layer_module( 2025-08-26T20:41:13.2951241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2951628Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2952054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2952484Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2952912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2953358Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2953818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.2954328Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.2954794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.2955256Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.2955668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.2956046Z return self.act(input) 2025-08-26T20:41:13.2956171Z 2025-08-26T20:41:13.2956294Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2956688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2957051Z return mod(**inputs) 2025-08-26T20:41:13.2957461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2957893Z outputs = self.roberta( 2025-08-26T20:41:13.2958298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2958736Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2959161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2959737Z layer_outputs = layer_module( 2025-08-26T20:41:13.2960134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2960526Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2960973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.2961429Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.2961903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.2962338Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.2962818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.2963357Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.2963854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.2964303Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.2964459Z 2025-08-26T20:41:13.2964584Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2964983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2965354Z return mod(**inputs) 2025-08-26T20:41:13.2965767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2966204Z outputs = self.roberta( 2025-08-26T20:41:13.2966612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2967129Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2967562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2967999Z layer_outputs = layer_module( 2025-08-26T20:41:13.2968391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2968787Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2969253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2969702Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2970134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2970549Z return func(*args, **kwargs) 2025-08-26T20:41:13.2970964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2971393Z self_outputs = self.self( 2025-08-26T20:41:13.2971792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2972196Z return func(*args, **kwargs) 2025-08-26T20:41:13.2972609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.2973245Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.2973540Z 2025-08-26T20:41:13.2973655Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2974051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2974409Z return mod(**inputs) 2025-08-26T20:41:13.2974804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2975228Z outputs = self.roberta( 2025-08-26T20:41:13.2975633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2976071Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2976502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2976937Z layer_outputs = layer_module( 2025-08-26T20:41:13.2977350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2977757Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2978215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2978654Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2979065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2979459Z return func(*args, **kwargs) 2025-08-26T20:41:13.2979870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2980286Z self_outputs = self.self( 2025-08-26T20:41:13.2980662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2981074Z return func(*args, **kwargs) 2025-08-26T20:41:13.2981501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.2981935Z self.key(current_states) 2025-08-26T20:41:13.2982061Z 2025-08-26T20:41:13.2982184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2982603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2982970Z return mod(**inputs) 2025-08-26T20:41:13.2983377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2983793Z outputs = self.roberta( 2025-08-26T20:41:13.2984189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2984644Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2985072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2985503Z layer_outputs = layer_module( 2025-08-26T20:41:13.2985886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2986276Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2986712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2987151Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2987577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2987982Z return func(*args, **kwargs) 2025-08-26T20:41:13.2988412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2988851Z self_outputs = self.self( 2025-08-26T20:41:13.2989283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2989693Z return func(*args, **kwargs) 2025-08-26T20:41:13.2990117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.2990570Z self.value(current_states) 2025-08-26T20:41:13.2990709Z 2025-08-26T20:41:13.2990802Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.2991074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.2991479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.2991838Z return mod(**inputs) 2025-08-26T20:41:13.2992250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.2992687Z outputs = self.roberta( 2025-08-26T20:41:13.2993126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.2993558Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.2994002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.2994439Z layer_outputs = layer_module( 2025-08-26T20:41:13.2994822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.2995229Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.2995665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.2996110Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.2996692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2997110Z return func(*args, **kwargs) 2025-08-26T20:41:13.2997532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.2997976Z self_outputs = self.self( 2025-08-26T20:41:13.2998429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.2998838Z return func(*args, **kwargs) 2025-08-26T20:41:13.2999257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.2999841Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3000060Z 2025-08-26T20:41:13.3000175Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3000617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3000981Z return mod(**inputs) 2025-08-26T20:41:13.3001396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3001822Z outputs = self.roberta( 2025-08-26T20:41:13.3002238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3002677Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3003103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3003524Z layer_outputs = layer_module( 2025-08-26T20:41:13.3003909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3004309Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3004749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3005196Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3005610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3006024Z return func(*args, **kwargs) 2025-08-26T20:41:13.3006444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3006928Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3007410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3007837Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3007992Z 2025-08-26T20:41:13.3008104Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3008493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3008885Z return mod(**inputs) 2025-08-26T20:41:13.3009291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3009754Z outputs = self.roberta( 2025-08-26T20:41:13.3010170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3010608Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3011039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3011451Z layer_outputs = layer_module( 2025-08-26T20:41:13.3011824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3012229Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3012670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3013116Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3013554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3014070Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3014528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3015039Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3015523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3015962Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3016149Z 2025-08-26T20:41:13.3016262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3016668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3017021Z return mod(**inputs) 2025-08-26T20:41:13.3017412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3017839Z outputs = self.roberta( 2025-08-26T20:41:13.3018241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3018666Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3019083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3019497Z layer_outputs = layer_module( 2025-08-26T20:41:13.3019869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3020261Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3020724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3021149Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3021576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3021999Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3022457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3022963Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3023424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3023893Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3024329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3024702Z return self.act(input) 2025-08-26T20:41:13.3024821Z 2025-08-26T20:41:13.3024974Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3025397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3025781Z return mod(**inputs) 2025-08-26T20:41:13.3026203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3026693Z outputs = self.roberta( 2025-08-26T20:41:13.3027145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3027585Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3028008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3028446Z layer_outputs = layer_module( 2025-08-26T20:41:13.3028824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3029208Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3029665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3030131Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3030537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3030936Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3031360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3031872Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3032332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3032740Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3032882Z 2025-08-26T20:41:13.3032998Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3033361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3033695Z return mod(**inputs) 2025-08-26T20:41:13.3034066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3034459Z outputs = self.roberta( 2025-08-26T20:41:13.3034830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3035249Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3035669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3036090Z layer_outputs = layer_module( 2025-08-26T20:41:13.3036463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3036842Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3037289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3037732Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3038142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3038542Z return func(*args, **kwargs) 2025-08-26T20:41:13.3038945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3039368Z self_outputs = self.self( 2025-08-26T20:41:13.3040069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3040490Z return func(*args, **kwargs) 2025-08-26T20:41:13.3040928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.3041490Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.3041766Z 2025-08-26T20:41:13.3041874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3042244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3042573Z return mod(**inputs) 2025-08-26T20:41:13.3042940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3043339Z outputs = self.roberta( 2025-08-26T20:41:13.3043724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3044126Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3044520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3044933Z layer_outputs = layer_module( 2025-08-26T20:41:13.3045288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3045660Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3046077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3046515Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3046941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3047322Z return func(*args, **kwargs) 2025-08-26T20:41:13.3047712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3048115Z self_outputs = self.self( 2025-08-26T20:41:13.3048478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3048851Z return func(*args, **kwargs) 2025-08-26T20:41:13.3049235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.3049633Z self.key(current_states) 2025-08-26T20:41:13.3049746Z 2025-08-26T20:41:13.3049858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3050221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3050551Z return mod(**inputs) 2025-08-26T20:41:13.3050925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3051321Z outputs = self.roberta( 2025-08-26T20:41:13.3051696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3052095Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3052507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3052930Z layer_outputs = layer_module( 2025-08-26T20:41:13.3053279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3053639Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3054042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3054476Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3054861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3055245Z return func(*args, **kwargs) 2025-08-26T20:41:13.3055657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3056076Z self_outputs = self.self( 2025-08-26T20:41:13.3056463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3056876Z return func(*args, **kwargs) 2025-08-26T20:41:13.3057280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.3057704Z self.value(current_states) 2025-08-26T20:41:13.3057839Z 2025-08-26T20:41:13.3057927Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.3058193Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3058578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3058944Z return mod(**inputs) 2025-08-26T20:41:13.3059363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3059785Z outputs = self.roberta( 2025-08-26T20:41:13.3060190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3060606Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3061026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3061474Z layer_outputs = layer_module( 2025-08-26T20:41:13.3061854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3062248Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3062691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3063127Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3063539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3063937Z return func(*args, **kwargs) 2025-08-26T20:41:13.3064351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3064772Z self_outputs = self.self( 2025-08-26T20:41:13.3065155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3065556Z return func(*args, **kwargs) 2025-08-26T20:41:13.3065969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.3066452Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3066660Z 2025-08-26T20:41:13.3066773Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3067173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3067554Z return mod(**inputs) 2025-08-26T20:41:13.3067956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3068368Z outputs = self.roberta( 2025-08-26T20:41:13.3068786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3069213Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3069663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3070079Z layer_outputs = layer_module( 2025-08-26T20:41:13.3070475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3070874Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3071319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3071771Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3072185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3072589Z return func(*args, **kwargs) 2025-08-26T20:41:13.3073004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3073496Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3073982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3074422Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3074601Z 2025-08-26T20:41:13.3074712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3075100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3075461Z return mod(**inputs) 2025-08-26T20:41:13.3075849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3076268Z outputs = self.roberta( 2025-08-26T20:41:13.3076672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3077120Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3077541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3077957Z layer_outputs = layer_module( 2025-08-26T20:41:13.3078334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3078727Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3079172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3079825Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3080616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3081337Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3082092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3082979Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3083550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3083996Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3084154Z 2025-08-26T20:41:13.3084270Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3084662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3085012Z return mod(**inputs) 2025-08-26T20:41:13.3085401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3085828Z outputs = self.roberta( 2025-08-26T20:41:13.3086233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3086751Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3087209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3087631Z layer_outputs = layer_module( 2025-08-26T20:41:13.3088007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3088397Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3088841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3089272Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3089705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3090127Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3090584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3091088Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3091553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3092046Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3092459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3092824Z return self.act(input) 2025-08-26T20:41:13.3092945Z 2025-08-26T20:41:13.3093062Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3093445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3093828Z return mod(**inputs) 2025-08-26T20:41:13.3094230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3094647Z outputs = self.roberta( 2025-08-26T20:41:13.3095043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3095467Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3095882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3096476Z layer_outputs = layer_module( 2025-08-26T20:41:13.3096856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3097236Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3097667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3098104Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3098531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3098952Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3099401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3099918Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3100402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3100833Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3100980Z 2025-08-26T20:41:13.3101098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3101479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3101559Z return mod(**inputs) 2025-08-26T20:41:13.3101887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3102001Z outputs = self.roberta( 2025-08-26T20:41:13.3102288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3102369Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3102658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3102736Z layer_outputs = layer_module( 2025-08-26T20:41:13.3102981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3103073Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3103360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3103458Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3103726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3103846Z return func(*args, **kwargs) 2025-08-26T20:41:13.3104128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3104212Z self_outputs = self.self( 2025-08-26T20:41:13.3104475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3104551Z return func(*args, **kwargs) 2025-08-26T20:41:13.3104841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.3105097Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.3105103Z 2025-08-26T20:41:13.3105222Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3105438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3105510Z return mod(**inputs) 2025-08-26T20:41:13.3105797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3105869Z outputs = self.roberta( 2025-08-26T20:41:13.3106155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3106234Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3106520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3106599Z layer_outputs = layer_module( 2025-08-26T20:41:13.3106838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3106932Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3107209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3107305Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3107563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3107638Z return func(*args, **kwargs) 2025-08-26T20:41:13.3107922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3107997Z self_outputs = self.self( 2025-08-26T20:41:13.3108263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3108336Z return func(*args, **kwargs) 2025-08-26T20:41:13.3108637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.3108737Z self.key(current_states) 2025-08-26T20:41:13.3108743Z 2025-08-26T20:41:13.3108861Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3109092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3109191Z return mod(**inputs) 2025-08-26T20:41:13.3109478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3109551Z outputs = self.roberta( 2025-08-26T20:41:13.3109832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3109920Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3110201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3110283Z layer_outputs = layer_module( 2025-08-26T20:41:13.3110523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3110631Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3110920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3111006Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3111272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3111345Z return func(*args, **kwargs) 2025-08-26T20:41:13.3111642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3111722Z self_outputs = self.self( 2025-08-26T20:41:13.3111983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3112065Z return func(*args, **kwargs) 2025-08-26T20:41:13.3112348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.3112428Z self.value(current_states) 2025-08-26T20:41:13.3112432Z 2025-08-26T20:41:13.3112516Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.3112621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3112834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3112904Z return mod(**inputs) 2025-08-26T20:41:13.3113190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3113265Z outputs = self.roberta( 2025-08-26T20:41:13.3113546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3113631Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3113911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3113992Z layer_outputs = layer_module( 2025-08-26T20:41:13.3114230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3114313Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3114603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3114691Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3114971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3115047Z return func(*args, **kwargs) 2025-08-26T20:41:13.3115354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3115433Z self_outputs = self.self( 2025-08-26T20:41:13.3115689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3115772Z return func(*args, **kwargs) 2025-08-26T20:41:13.3116055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.3116206Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3116210Z 2025-08-26T20:41:13.3116320Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3116536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3116620Z return mod(**inputs) 2025-08-26T20:41:13.3116912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3116996Z outputs = self.roberta( 2025-08-26T20:41:13.3117306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3117405Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3117684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3117762Z layer_outputs = layer_module( 2025-08-26T20:41:13.3118010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3118123Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3118410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3118498Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3118756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3118840Z return func(*args, **kwargs) 2025-08-26T20:41:13.3119117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3119268Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3119666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3119766Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3119780Z 2025-08-26T20:41:13.3119898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3120120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3120203Z return mod(**inputs) 2025-08-26T20:41:13.3120493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3120580Z outputs = self.roberta( 2025-08-26T20:41:13.3120873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3120954Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3121241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3121319Z layer_outputs = layer_module( 2025-08-26T20:41:13.3121567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3121654Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3121957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3122058Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3122356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3122451Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3122773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3122902Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3123167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3123251Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3123255Z 2025-08-26T20:41:13.3123369Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3123585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3123665Z return mod(**inputs) 2025-08-26T20:41:13.3123943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3124038Z outputs = self.roberta( 2025-08-26T20:41:13.3124323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3124402Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3124685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3124761Z layer_outputs = layer_module( 2025-08-26T20:41:13.3125028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3125111Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3125389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3125484Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3125744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3125829Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3126127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3126248Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3126518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3126633Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3126855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3126927Z return self.act(input) 2025-08-26T20:41:13.3126930Z 2025-08-26T20:41:13.3127041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3127244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3127311Z return mod(**inputs) 2025-08-26T20:41:13.3127578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3127646Z outputs = self.roberta( 2025-08-26T20:41:13.3127914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3127990Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3128273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3128354Z layer_outputs = layer_module( 2025-08-26T20:41:13.3128595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3128684Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3128949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3129031Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3129300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3129376Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3129682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3129819Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3130093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3130176Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3130198Z 2025-08-26T20:41:13.3130305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3130517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3130585Z return mod(**inputs) 2025-08-26T20:41:13.3130855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3130925Z outputs = self.roberta( 2025-08-26T20:41:13.3131190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3131294Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3131590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3131674Z layer_outputs = layer_module( 2025-08-26T20:41:13.3131911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3132014Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3132279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3132363Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3132617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3132688Z return func(*args, **kwargs) 2025-08-26T20:41:13.3132960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3133034Z self_outputs = self.self( 2025-08-26T20:41:13.3133278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3133361Z return func(*args, **kwargs) 2025-08-26T20:41:13.3133630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.3133850Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.3133854Z 2025-08-26T20:41:13.3133958Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3134169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3134236Z return mod(**inputs) 2025-08-26T20:41:13.3134503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3134596Z outputs = self.roberta( 2025-08-26T20:41:13.3134861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3134956Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3135239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3135316Z layer_outputs = layer_module( 2025-08-26T20:41:13.3135562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3135669Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3135941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3136027Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3136271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3136351Z return func(*args, **kwargs) 2025-08-26T20:41:13.3136617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3136715Z self_outputs = self.self( 2025-08-26T20:41:13.3136965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3137048Z return func(*args, **kwargs) 2025-08-26T20:41:13.3137327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.3137403Z self.key(current_states) 2025-08-26T20:41:13.3137406Z 2025-08-26T20:41:13.3137525Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3137758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3137837Z return mod(**inputs) 2025-08-26T20:41:13.3138115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3138189Z outputs = self.roberta( 2025-08-26T20:41:13.3138478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3138554Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3138839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3138916Z layer_outputs = layer_module( 2025-08-26T20:41:13.3139154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3139245Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3139537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3139631Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3139902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3139984Z return func(*args, **kwargs) 2025-08-26T20:41:13.3140263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3140336Z self_outputs = self.self( 2025-08-26T20:41:13.3140601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3140674Z return func(*args, **kwargs) 2025-08-26T20:41:13.3140958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.3141038Z self.value(current_states) 2025-08-26T20:41:13.3141042Z 2025-08-26T20:41:13.3141146Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.3141267Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3141501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3141580Z return mod(**inputs) 2025-08-26T20:41:13.3141859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3141931Z outputs = self.roberta( 2025-08-26T20:41:13.3142219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3142298Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3142588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3142665Z layer_outputs = layer_module( 2025-08-26T20:41:13.3142909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3142993Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3143277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3143405Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3143662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3143744Z return func(*args, **kwargs) 2025-08-26T20:41:13.3144022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3144094Z self_outputs = self.self( 2025-08-26T20:41:13.3144377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3144453Z return func(*args, **kwargs) 2025-08-26T20:41:13.3144737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.3144880Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3144885Z 2025-08-26T20:41:13.3144999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3145209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3145279Z return mod(**inputs) 2025-08-26T20:41:13.3145563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3145636Z outputs = self.roberta( 2025-08-26T20:41:13.3145921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3146001Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3146277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3146362Z layer_outputs = layer_module( 2025-08-26T20:41:13.3146598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3146689Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3146966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3147053Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3147310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3147386Z return func(*args, **kwargs) 2025-08-26T20:41:13.3147669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3147823Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3148153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3148246Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3148250Z 2025-08-26T20:41:13.3148360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3148579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3148650Z return mod(**inputs) 2025-08-26T20:41:13.3148935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3149008Z outputs = self.roberta( 2025-08-26T20:41:13.3149287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3149373Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3149651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3149736Z layer_outputs = layer_module( 2025-08-26T20:41:13.3149998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3150090Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3150370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3150460Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3150743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3150846Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3151171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3151300Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3151581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3151674Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3151678Z 2025-08-26T20:41:13.3151780Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3151991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3152058Z return mod(**inputs) 2025-08-26T20:41:13.3152325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3152395Z outputs = self.roberta( 2025-08-26T20:41:13.3152663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3152743Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3153010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3153090Z layer_outputs = layer_module( 2025-08-26T20:41:13.3153314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3153393Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3153664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3153748Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3154013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3154093Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3154416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3154570Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3154865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3154993Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3155220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3155304Z return self.act(input) 2025-08-26T20:41:13.3155308Z 2025-08-26T20:41:13.3155417Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3155630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3155710Z return mod(**inputs) 2025-08-26T20:41:13.3155988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3156068Z outputs = self.roberta( 2025-08-26T20:41:13.3156346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3156442Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3156732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3156808Z layer_outputs = layer_module( 2025-08-26T20:41:13.3157053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3157137Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3157440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3157528Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3157806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3157894Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3158208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3158354Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3158641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3158729Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3158741Z 2025-08-26T20:41:13.3158850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3159072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3159149Z return mod(**inputs) 2025-08-26T20:41:13.3159521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3159627Z outputs = self.roberta( 2025-08-26T20:41:13.3160056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3160135Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3160418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3160495Z layer_outputs = layer_module( 2025-08-26T20:41:13.3160737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3160826Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3161152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3161249Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3161523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3161607Z return func(*args, **kwargs) 2025-08-26T20:41:13.3161887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3161972Z self_outputs = self.self( 2025-08-26T20:41:13.3162223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3162294Z return func(*args, **kwargs) 2025-08-26T20:41:13.3162569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.3162783Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.3162788Z 2025-08-26T20:41:13.3162898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3163103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3163189Z return mod(**inputs) 2025-08-26T20:41:13.3163467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3163537Z outputs = self.roberta( 2025-08-26T20:41:13.3163813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3163889Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3164158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3164258Z layer_outputs = layer_module( 2025-08-26T20:41:13.3164484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3164573Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3164834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3164926Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3165172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3165243Z return func(*args, **kwargs) 2025-08-26T20:41:13.3165511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3165585Z self_outputs = self.self( 2025-08-26T20:41:13.3165848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3165924Z return func(*args, **kwargs) 2025-08-26T20:41:13.3166203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.3166290Z self.key(current_states) 2025-08-26T20:41:13.3166293Z 2025-08-26T20:41:13.3166404Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3166623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3166693Z return mod(**inputs) 2025-08-26T20:41:13.3166969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3167050Z outputs = self.roberta( 2025-08-26T20:41:13.3167328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3167414Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3167709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3167818Z layer_outputs = layer_module( 2025-08-26T20:41:13.3168059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3168143Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3168432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3168520Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3168783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3168859Z return func(*args, **kwargs) 2025-08-26T20:41:13.3169140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3169225Z self_outputs = self.self( 2025-08-26T20:41:13.3169484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3169583Z return func(*args, **kwargs) 2025-08-26T20:41:13.3169865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.3169945Z self.value(current_states) 2025-08-26T20:41:13.3169955Z 2025-08-26T20:41:13.3170042Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.3170151Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3170369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3170460Z return mod(**inputs) 2025-08-26T20:41:13.3170751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3170827Z outputs = self.roberta( 2025-08-26T20:41:13.3171109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3171196Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3171480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3171564Z layer_outputs = layer_module( 2025-08-26T20:41:13.3171803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3171887Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3172178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3172266Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3172535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3172608Z return func(*args, **kwargs) 2025-08-26T20:41:13.3172890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3172976Z self_outputs = self.self( 2025-08-26T20:41:13.3173234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3173315Z return func(*args, **kwargs) 2025-08-26T20:41:13.3173594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.3173744Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3173750Z 2025-08-26T20:41:13.3173859Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3174088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3174169Z return mod(**inputs) 2025-08-26T20:41:13.3174465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3174546Z outputs = self.roberta( 2025-08-26T20:41:13.3174822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3174900Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3175188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3175263Z layer_outputs = layer_module( 2025-08-26T20:41:13.3175509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3175594Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3175879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3175966Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3176226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3176329Z return func(*args, **kwargs) 2025-08-26T20:41:13.3176607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3176752Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3177026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3177137Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3177141Z 2025-08-26T20:41:13.3177257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3177471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3177549Z return mod(**inputs) 2025-08-26T20:41:13.3177830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3177913Z outputs = self.roberta( 2025-08-26T20:41:13.3178190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3178268Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3178554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3178631Z layer_outputs = layer_module( 2025-08-26T20:41:13.3178878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3178962Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3179245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3179344Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3179618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3179706Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3180023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3180150Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3180436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3180526Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3180530Z 2025-08-26T20:41:13.3180663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3180876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3180970Z return mod(**inputs) 2025-08-26T20:41:13.3181250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3181326Z outputs = self.roberta( 2025-08-26T20:41:13.3181612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3181689Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3181980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3182058Z layer_outputs = layer_module( 2025-08-26T20:41:13.3182297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3182386Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3182666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3182781Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3183057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3183146Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3183459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3183585Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3183895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3184017Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3184255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3184332Z return self.act(input) 2025-08-26T20:41:13.3184337Z 2025-08-26T20:41:13.3184446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3184665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3184735Z return mod(**inputs) 2025-08-26T20:41:13.3185022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3185094Z outputs = self.roberta( 2025-08-26T20:41:13.3185379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3185459Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3185738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3185822Z layer_outputs = layer_module( 2025-08-26T20:41:13.3186061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3186154Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3186433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3186521Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3186804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3186885Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3187209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3187366Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3187670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3187760Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3187763Z 2025-08-26T20:41:13.3187873Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3188096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3188166Z return mod(**inputs) 2025-08-26T20:41:13.3188451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3188523Z outputs = self.roberta( 2025-08-26T20:41:13.3188803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3188889Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3189166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3189250Z layer_outputs = layer_module( 2025-08-26T20:41:13.3189505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3189589Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3189876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3189963Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3190230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3190322Z return func(*args, **kwargs) 2025-08-26T20:41:13.3190614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3190688Z self_outputs = self.self( 2025-08-26T20:41:13.3190955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3191036Z return func(*args, **kwargs) 2025-08-26T20:41:13.3191316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.3191544Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.3191548Z 2025-08-26T20:41:13.3191657Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3191868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3191947Z return mod(**inputs) 2025-08-26T20:41:13.3192226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3192306Z outputs = self.roberta( 2025-08-26T20:41:13.3192587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3192671Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3192950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3193026Z layer_outputs = layer_module( 2025-08-26T20:41:13.3193271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3193354Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3193653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3193744Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3194039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3194122Z return func(*args, **kwargs) 2025-08-26T20:41:13.3194416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3194503Z self_outputs = self.self( 2025-08-26T20:41:13.3194771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3194845Z return func(*args, **kwargs) 2025-08-26T20:41:13.3195135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.3195215Z self.key(current_states) 2025-08-26T20:41:13.3195220Z 2025-08-26T20:41:13.3195341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3195574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3195654Z return mod(**inputs) 2025-08-26T20:41:13.3195940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3196035Z outputs = self.roberta( 2025-08-26T20:41:13.3197347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3197765Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3198167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3198259Z layer_outputs = layer_module( 2025-08-26T20:41:13.3198529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3199047Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3199374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3199558Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3199871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3199969Z return func(*args, **kwargs) 2025-08-26T20:41:13.3200269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3200349Z self_outputs = self.self( 2025-08-26T20:41:13.3200625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3200704Z return func(*args, **kwargs) 2025-08-26T20:41:13.3201014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.3201099Z self.value(current_states) 2025-08-26T20:41:13.3201105Z 2025-08-26T20:41:13.3201203Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.3201335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3201574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3201678Z return mod(**inputs) 2025-08-26T20:41:13.3201973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3202051Z outputs = self.roberta( 2025-08-26T20:41:13.3202348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3202431Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3202772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3202911Z layer_outputs = layer_module( 2025-08-26T20:41:13.3203171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3203297Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3203597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3203694Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3203968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3204054Z return func(*args, **kwargs) 2025-08-26T20:41:13.3204391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3204479Z self_outputs = self.self( 2025-08-26T20:41:13.3204756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3204832Z return func(*args, **kwargs) 2025-08-26T20:41:13.3205132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.3205332Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3205338Z 2025-08-26T20:41:13.3205454Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3205697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3205771Z return mod(**inputs) 2025-08-26T20:41:13.3206065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3206141Z outputs = self.roberta( 2025-08-26T20:41:13.3206451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3206534Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3206816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3206903Z layer_outputs = layer_module( 2025-08-26T20:41:13.3207143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3207238Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3207516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3207606Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3207885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3207962Z return func(*args, **kwargs) 2025-08-26T20:41:13.3208252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3208394Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3208682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3208776Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3208780Z 2025-08-26T20:41:13.3208893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3209118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3209189Z return mod(**inputs) 2025-08-26T20:41:13.3209513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3209588Z outputs = self.roberta( 2025-08-26T20:41:13.3209881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3209971Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3225019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3225220Z layer_outputs = layer_module( 2025-08-26T20:41:13.3225511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3225602Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3225893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3225994Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3226261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3226357Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3226664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3226801Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3227127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3227223Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3227229Z 2025-08-26T20:41:13.3227348Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3227559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3227638Z return mod(**inputs) 2025-08-26T20:41:13.3227902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3228003Z outputs = self.roberta( 2025-08-26T20:41:13.3228276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3228354Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3228622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3228699Z layer_outputs = layer_module( 2025-08-26T20:41:13.3228932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3229017Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3229276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3229368Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3229628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3229714Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3230007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3230130Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3230395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3230508Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3230727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3230798Z return self.act(input) 2025-08-26T20:41:13.3230803Z 2025-08-26T20:41:13.3230918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3231125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3231217Z return mod(**inputs) 2025-08-26T20:41:13.3231489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3231575Z outputs = self.roberta( 2025-08-26T20:41:13.3231848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3231925Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3232180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3232258Z layer_outputs = layer_module( 2025-08-26T20:41:13.3232478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3232567Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3232825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3232908Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3233167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3233261Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3233556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3233692Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3233958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3234038Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3234059Z 2025-08-26T20:41:13.3234167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3234380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3234447Z return mod(**inputs) 2025-08-26T20:41:13.3234714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3234784Z outputs = self.roberta( 2025-08-26T20:41:13.3235044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3235128Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3235441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3235519Z layer_outputs = layer_module( 2025-08-26T20:41:13.3235769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3235856Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3236141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3236242Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3236517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3236606Z return func(*args, **kwargs) 2025-08-26T20:41:13.3236892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3236979Z self_outputs = self.self( 2025-08-26T20:41:13.3237245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3237320Z return func(*args, **kwargs) 2025-08-26T20:41:13.3237614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.3237864Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.3237868Z 2025-08-26T20:41:13.3238027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3238248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3238318Z return mod(**inputs) 2025-08-26T20:41:13.3238611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3238684Z outputs = self.roberta( 2025-08-26T20:41:13.3238974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3239052Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3239345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3239764Z layer_outputs = layer_module( 2025-08-26T20:41:13.3240773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3240879Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3241183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3241276Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3241525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3241599Z return func(*args, **kwargs) 2025-08-26T20:41:13.3241872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3242852Z self_outputs = self.self( 2025-08-26T20:41:13.3243128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3243206Z return func(*args, **kwargs) 2025-08-26T20:41:13.3243485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.3243572Z self.key(current_states) 2025-08-26T20:41:13.3243578Z 2025-08-26T20:41:13.3243695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3243925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3243998Z return mod(**inputs) 2025-08-26T20:41:13.3244276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3244345Z outputs = self.roberta( 2025-08-26T20:41:13.3244607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3244696Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3244976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3245058Z layer_outputs = layer_module( 2025-08-26T20:41:13.3245296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3245382Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3245670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3245758Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3246024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3246099Z return func(*args, **kwargs) 2025-08-26T20:41:13.3246443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3246522Z self_outputs = self.self( 2025-08-26T20:41:13.3246800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3246887Z return func(*args, **kwargs) 2025-08-26T20:41:13.3247171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.3247256Z self.value(current_states) 2025-08-26T20:41:13.3247261Z 2025-08-26T20:41:13.3247353Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.3247470Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3247696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3247771Z return mod(**inputs) 2025-08-26T20:41:13.3248061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3248134Z outputs = self.roberta( 2025-08-26T20:41:13.3248418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3248531Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3248818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3248903Z layer_outputs = layer_module( 2025-08-26T20:41:13.3249149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3249240Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3249528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3249633Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3249901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3249974Z return func(*args, **kwargs) 2025-08-26T20:41:13.3250264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3250340Z self_outputs = self.self( 2025-08-26T20:41:13.3250600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3250678Z return func(*args, **kwargs) 2025-08-26T20:41:13.3250959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.3251111Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3251117Z 2025-08-26T20:41:13.3251229Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3251445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3251522Z return mod(**inputs) 2025-08-26T20:41:13.3251803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3251885Z outputs = self.roberta( 2025-08-26T20:41:13.3252168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3252252Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3252532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3252608Z layer_outputs = layer_module( 2025-08-26T20:41:13.3252857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3252958Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3253249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3253365Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3253624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3253705Z return func(*args, **kwargs) 2025-08-26T20:41:13.3253983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3254130Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3254419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3254512Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3254516Z 2025-08-26T20:41:13.3254620Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3254816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3254891Z return mod(**inputs) 2025-08-26T20:41:13.3255147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3255243Z outputs = self.roberta( 2025-08-26T20:41:13.3255506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3255579Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3255853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3255943Z layer_outputs = layer_module( 2025-08-26T20:41:13.3256178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3256261Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3256529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3256626Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3256890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3256977Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3257280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3257411Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3257672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3257755Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3257760Z 2025-08-26T20:41:13.3257872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3258070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3258144Z return mod(**inputs) 2025-08-26T20:41:13.3258417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3258492Z outputs = self.roberta( 2025-08-26T20:41:13.3258783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3258862Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3259152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3259231Z layer_outputs = layer_module( 2025-08-26T20:41:13.3259494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3259580Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3259881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3259981Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3260259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3260348Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3260669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3260801Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3261090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3261215Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3261456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3261551Z return self.act(input) 2025-08-26T20:41:13.3261555Z 2025-08-26T20:41:13.3261675Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3261893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3261963Z return mod(**inputs) 2025-08-26T20:41:13.3262260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3262330Z outputs = self.roberta( 2025-08-26T20:41:13.3262606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3262723Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3262990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3263071Z layer_outputs = layer_module( 2025-08-26T20:41:13.3263296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3263385Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3263671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3263759Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3264047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3264133Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3264460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3264605Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3264896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3264986Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3264990Z 2025-08-26T20:41:13.3265101Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3265322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3265393Z return mod(**inputs) 2025-08-26T20:41:13.3265680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3265755Z outputs = self.roberta( 2025-08-26T20:41:13.3266055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3266141Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3266439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3266529Z layer_outputs = layer_module( 2025-08-26T20:41:13.3266767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3266857Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3267136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3267224Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3267493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3267569Z return func(*args, **kwargs) 2025-08-26T20:41:13.3267859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3267930Z self_outputs = self.self( 2025-08-26T20:41:13.3268171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3268352Z return func(*args, **kwargs) 2025-08-26T20:41:13.3268659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:13.3268889Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:13.3268894Z 2025-08-26T20:41:13.3269000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3269219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3269316Z return mod(**inputs) 2025-08-26T20:41:13.3269603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3269688Z outputs = self.roberta( 2025-08-26T20:41:13.3269978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3270070Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3270358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3270437Z layer_outputs = layer_module( 2025-08-26T20:41:13.3270691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3270778Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3271085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3271174Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3271439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3271515Z return func(*args, **kwargs) 2025-08-26T20:41:13.3271797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3271881Z self_outputs = self.self( 2025-08-26T20:41:13.3272139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3272219Z return func(*args, **kwargs) 2025-08-26T20:41:13.3272500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:13.3272578Z self.key(current_states) 2025-08-26T20:41:13.3272582Z 2025-08-26T20:41:13.3272701Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3272948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3273027Z return mod(**inputs) 2025-08-26T20:41:13.3273326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3273403Z outputs = self.roberta( 2025-08-26T20:41:13.3273695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3273774Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3274064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3274140Z layer_outputs = layer_module( 2025-08-26T20:41:13.3274391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3274476Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3274760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3274856Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3275130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3275211Z return func(*args, **kwargs) 2025-08-26T20:41:13.3275490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3275567Z self_outputs = self.self( 2025-08-26T20:41:13.3275840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3275934Z return func(*args, **kwargs) 2025-08-26T20:41:13.3276234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:13.3276315Z self.value(current_states) 2025-08-26T20:41:13.3276320Z 2025-08-26T20:41:13.3276411Z cudagraph partition due to non gpu ops 2025-08-26T20:41:13.3276536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3276757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3276836Z return mod(**inputs) 2025-08-26T20:41:13.3277125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3277207Z outputs = self.roberta( 2025-08-26T20:41:13.3277496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3277577Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3277873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3277958Z layer_outputs = layer_module( 2025-08-26T20:41:13.3278210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3278294Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3278576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3278669Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3278927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3279006Z return func(*args, **kwargs) 2025-08-26T20:41:13.3279290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:13.3279370Z self_outputs = self.self( 2025-08-26T20:41:13.3279904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3279993Z return func(*args, **kwargs) 2025-08-26T20:41:13.3280312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:13.3280481Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:13.3280485Z 2025-08-26T20:41:13.3280607Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3280824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3280895Z return mod(**inputs) 2025-08-26T20:41:13.3281187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3281262Z outputs = self.roberta( 2025-08-26T20:41:13.3281558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3281639Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3281925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3282033Z layer_outputs = layer_module( 2025-08-26T20:41:13.3282274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3282372Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3282657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:13.3282756Z self_attention_outputs = self.attention( 2025-08-26T20:41:13.3283015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:13.3283108Z return func(*args, **kwargs) 2025-08-26T20:41:13.3283400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:13.3283541Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:13.3283829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:13.3283916Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3283920Z 2025-08-26T20:41:13.3284029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3284252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3284320Z return mod(**inputs) 2025-08-26T20:41:13.3284604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3284679Z outputs = self.roberta( 2025-08-26T20:41:13.3284961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3285045Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3285330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3285411Z layer_outputs = layer_module( 2025-08-26T20:41:13.3285647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3285737Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3286018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3286109Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3286397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3286494Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3286831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3286964Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3287244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:13.3287338Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3287342Z 2025-08-26T20:41:13.3287457Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3287667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3287734Z return mod(**inputs) 2025-08-26T20:41:13.3288005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3288075Z outputs = self.roberta( 2025-08-26T20:41:13.3288340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3288425Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3288706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3288786Z layer_outputs = layer_module( 2025-08-26T20:41:13.3289010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3289089Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3289363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3289465Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3289738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3289821Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3290145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:13.3290277Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:13.3290557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:13.3290681Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:13.3290898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:13.3290977Z return self.act(input) 2025-08-26T20:41:13.3290983Z 2025-08-26T20:41:13.3291089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3291290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3291382Z return mod(**inputs) 2025-08-26T20:41:13.3291650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-26T20:41:13.3291732Z outputs = self.roberta( 2025-08-26T20:41:13.3292011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:13.3292088Z encoder_outputs = self.encoder( 2025-08-26T20:41:13.3292371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:13.3292448Z layer_outputs = layer_module( 2025-08-26T20:41:13.3292693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:13.3292778Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:13.3293086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:13.3293180Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:13.3293478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:13.3293569Z return forward_fn(*input_tensors) 2025-08-26T20:41:13.3293885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:13.3294036Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:13.3294338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:13.3294431Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:13.3294435Z 2025-08-26T20:41:13.3294552Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3294771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3294850Z return mod(**inputs) 2025-08-26T20:41:13.3295146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-08-26T20:41:13.3295284Z prediction_scores = self.lm_head(sequence_output) 2025-08-26T20:41:13.3295580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1149, in forward 2025-08-26T20:41:13.3295660Z x = self.dense(features) 2025-08-26T20:41:13.3295663Z 2025-08-26T20:41:13.3295784Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3296003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3296102Z return mod(**inputs) 2025-08-26T20:41:13.3296757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-08-26T20:41:13.3296867Z prediction_scores = self.lm_head(sequence_output) 2025-08-26T20:41:13.3297152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1154, in forward 2025-08-26T20:41:13.3297224Z x = self.decoder(x) 2025-08-26T20:41:13.3297228Z 2025-08-26T20:41:13.3297339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:13.3297540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:13.3297615Z return mod(**inputs) 2025-08-26T20:41:13.3297886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1022, in forward 2025-08-26T20:41:13.3297964Z lm_loss = self.loss_function( 2025-08-26T20:41:13.3298226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-26T20:41:13.3298404Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-26T20:41:13.3298677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-26T20:41:13.3298893Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-26T20:41:13.3298898Z 2025-08-26T20:41:22.3390713Z Compilation time (from dynamo_timed): 15.642708544 2025-08-26T20:41:22.3516592Z pass 2025-08-26T20:41:22.3517044Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:22.3517964Z TIMING: _recursive_pre_grad_passes:0.00754 _recursive_joint_graph_passes:0.38911 _recursive_post_grad_passes:0.07928 async_compile.wait:0.80296 code_gen:8.14151 inductor_compile:9.45058 backend_compile:12.75579 gc:0.00032 entire_frame_compile:15.64271 total_wall_time:15.64271 2025-08-26T20:41:22.3519684Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12458 | FakeTensor.__torch_dispatch__:4402 | ProxyTorchDispatchMode.__torch_dispatch__:4539 2025-08-26T20:41:22.3520352Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-08-26T20:41:27.8977314Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:41:27.8978746Z from pkg_resources import resource_filename 2025-08-26T20:41:28.4882723Z 2025-08-26T20:41:29.6780656Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-26T20:41:29.6781726Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-26T20:41:29.6782707Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-26T20:41:29.6784037Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-26T20:41:29.8155236Z 2025-08-26T20:41:29.8156152Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:41:29.8171923Z cpu eval RobertaForQuestionAnswering 2025-08-26T20:41:30.2871593Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:30.4922311Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:30.7385789Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:38.8993859Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.8995538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.8995940Z return mod(**inputs) 2025-08-26T20:41:38.8996603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.8997085Z outputs = self.roberta( 2025-08-26T20:41:38.8997523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-26T20:41:38.8997999Z embedding_output = self.embeddings( 2025-08-26T20:41:38.8998498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-26T20:41:38.8999141Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:41:38.9000153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-08-26T20:41:38.9000700Z mask = input_ids.ne(padding_idx).int() 2025-08-26T20:41:38.9000862Z 2025-08-26T20:41:38.9000978Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9001205Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9001439Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9001668Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9001900Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9002127Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9002361Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9002584Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9003127Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9003360Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9003581Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9003865Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9004137Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9004532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9004891Z return mod(**inputs) 2025-08-26T20:41:38.9005328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9005764Z outputs = self.roberta( 2025-08-26T20:41:38.9006168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-26T20:41:38.9006608Z embedding_output = self.embeddings( 2025-08-26T20:41:38.9007051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-26T20:41:38.9007830Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:41:38.9008491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-26T20:41:38.9009194Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:41:38.9009467Z 2025-08-26T20:41:38.9009583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9009984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9010334Z return mod(**inputs) 2025-08-26T20:41:38.9010797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9011218Z outputs = self.roberta( 2025-08-26T20:41:38.9011624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-26T20:41:38.9012069Z embedding_output = self.embeddings( 2025-08-26T20:41:38.9012508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-26T20:41:38.9013090Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-26T20:41:38.9013773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-26T20:41:38.9014394Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-26T20:41:38.9014670Z 2025-08-26T20:41:38.9014790Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9015226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9015576Z return mod(**inputs) 2025-08-26T20:41:38.9015988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9016434Z outputs = self.roberta( 2025-08-26T20:41:38.9016843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9017287Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9017726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9018164Z layer_outputs = layer_module( 2025-08-26T20:41:38.9018560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9018986Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9019487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9019938Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9020384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9020816Z return func(*args, **kwargs) 2025-08-26T20:41:38.9021251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9021683Z self_outputs = self.self( 2025-08-26T20:41:38.9022091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9022516Z return func(*args, **kwargs) 2025-08-26T20:41:38.9022945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9023537Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9023834Z 2025-08-26T20:41:38.9023962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9024392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9024748Z return mod(**inputs) 2025-08-26T20:41:38.9025163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9025603Z outputs = self.roberta( 2025-08-26T20:41:38.9026024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9026459Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9026912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9027365Z layer_outputs = layer_module( 2025-08-26T20:41:38.9027748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9028148Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9028581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9029018Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9029438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9029850Z return func(*args, **kwargs) 2025-08-26T20:41:38.9030267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9030700Z self_outputs = self.self( 2025-08-26T20:41:38.9031098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9031509Z return func(*args, **kwargs) 2025-08-26T20:41:38.9032015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9032460Z self.key(current_states) 2025-08-26T20:41:38.9032590Z 2025-08-26T20:41:38.9032705Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9033098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9034382Z return mod(**inputs) 2025-08-26T20:41:38.9034796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9035225Z outputs = self.roberta( 2025-08-26T20:41:38.9035632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9036103Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9036544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9036990Z layer_outputs = layer_module( 2025-08-26T20:41:38.9037371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9037767Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9038208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9038664Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9039095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9039630Z return func(*args, **kwargs) 2025-08-26T20:41:38.9040061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9040496Z self_outputs = self.self( 2025-08-26T20:41:38.9040888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9041333Z return func(*args, **kwargs) 2025-08-26T20:41:38.9041741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9042165Z self.value(current_states) 2025-08-26T20:41:38.9042291Z 2025-08-26T20:41:38.9042384Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9042638Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9043031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9043407Z return mod(**inputs) 2025-08-26T20:41:38.9043812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9044235Z outputs = self.roberta( 2025-08-26T20:41:38.9044634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9045094Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9045516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9045937Z layer_outputs = layer_module( 2025-08-26T20:41:38.9046306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9046699Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9047125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9047559Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9047970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9048363Z return func(*args, **kwargs) 2025-08-26T20:41:38.9048773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9049193Z self_outputs = self.self( 2025-08-26T20:41:38.9049580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9049977Z return func(*args, **kwargs) 2025-08-26T20:41:38.9050385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9050873Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9051077Z 2025-08-26T20:41:38.9051196Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9051608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9051949Z return mod(**inputs) 2025-08-26T20:41:38.9052353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9052754Z outputs = self.roberta( 2025-08-26T20:41:38.9053148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9053574Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9053992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9054418Z layer_outputs = layer_module( 2025-08-26T20:41:38.9054800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9055192Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9055628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9056045Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9056476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9056890Z return func(*args, **kwargs) 2025-08-26T20:41:38.9057275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9057738Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9058225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9058660Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9058804Z 2025-08-26T20:41:38.9058930Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9059295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9059622Z return mod(**inputs) 2025-08-26T20:41:38.9060003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9060407Z outputs = self.roberta( 2025-08-26T20:41:38.9060788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9061182Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9061582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9061983Z layer_outputs = layer_module( 2025-08-26T20:41:38.9062336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9062708Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9063108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9063521Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9063929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9064329Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9064784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9065284Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9065762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9066173Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9066331Z 2025-08-26T20:41:38.9066450Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9066838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9067167Z return mod(**inputs) 2025-08-26T20:41:38.9067547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9067950Z outputs = self.roberta( 2025-08-26T20:41:38.9068330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9068725Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9069119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9069521Z layer_outputs = layer_module( 2025-08-26T20:41:38.9069879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9070252Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9070648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9071075Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9071478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9071871Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9072300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9072801Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9073308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9073771Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9074180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9074540Z return self.act(input) 2025-08-26T20:41:38.9074667Z 2025-08-26T20:41:38.9074779Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9075168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9075517Z return mod(**inputs) 2025-08-26T20:41:38.9075920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9076331Z outputs = self.roberta( 2025-08-26T20:41:38.9076735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9077162Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9077590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9078025Z layer_outputs = layer_module( 2025-08-26T20:41:38.9078396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9078798Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9079251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9079816Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9080257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9080710Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9081238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9081764Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9082285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9082719Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9082877Z 2025-08-26T20:41:38.9082991Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9083399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9083757Z return mod(**inputs) 2025-08-26T20:41:38.9084157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9084572Z outputs = self.roberta( 2025-08-26T20:41:38.9084975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9085399Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9085817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9086258Z layer_outputs = layer_module( 2025-08-26T20:41:38.9086632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9087029Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9087467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9087913Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9088327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9088752Z return func(*args, **kwargs) 2025-08-26T20:41:38.9089142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9089541Z self_outputs = self.self( 2025-08-26T20:41:38.9089909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9090278Z return func(*args, **kwargs) 2025-08-26T20:41:38.9090667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9091204Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9091471Z 2025-08-26T20:41:38.9091583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9091953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9092274Z return mod(**inputs) 2025-08-26T20:41:38.9092655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9093054Z outputs = self.roberta( 2025-08-26T20:41:38.9093456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9093872Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9094288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9094713Z layer_outputs = layer_module( 2025-08-26T20:41:38.9095076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9095444Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9095840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9096531Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9096930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9097343Z return func(*args, **kwargs) 2025-08-26T20:41:38.9097734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9098134Z self_outputs = self.self( 2025-08-26T20:41:38.9098500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9098880Z return func(*args, **kwargs) 2025-08-26T20:41:38.9099265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9099657Z self.key(current_states) 2025-08-26T20:41:38.9099780Z 2025-08-26T20:41:38.9099888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9100256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9100586Z return mod(**inputs) 2025-08-26T20:41:38.9100967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9101382Z outputs = self.roberta( 2025-08-26T20:41:38.9101761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9102176Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9102572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9102962Z layer_outputs = layer_module( 2025-08-26T20:41:38.9103347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9103728Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9104171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9104617Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9105042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9105426Z return func(*args, **kwargs) 2025-08-26T20:41:38.9105824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9106237Z self_outputs = self.self( 2025-08-26T20:41:38.9106629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9107040Z return func(*args, **kwargs) 2025-08-26T20:41:38.9107463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9107898Z self.value(current_states) 2025-08-26T20:41:38.9108028Z 2025-08-26T20:41:38.9108127Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9108390Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9108789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9109161Z return mod(**inputs) 2025-08-26T20:41:38.9109574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9110007Z outputs = self.roberta( 2025-08-26T20:41:38.9110415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9110852Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9111302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9111724Z layer_outputs = layer_module( 2025-08-26T20:41:38.9112112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9112502Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9112927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9113370Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9113778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9114169Z return func(*args, **kwargs) 2025-08-26T20:41:38.9114581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9115003Z self_outputs = self.self( 2025-08-26T20:41:38.9115396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9115763Z return func(*args, **kwargs) 2025-08-26T20:41:38.9116154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9116630Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9116820Z 2025-08-26T20:41:38.9116935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9117313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9117654Z return mod(**inputs) 2025-08-26T20:41:38.9118060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9118502Z outputs = self.roberta( 2025-08-26T20:41:38.9118905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9119317Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9119841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9120252Z layer_outputs = layer_module( 2025-08-26T20:41:38.9120628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9121020Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9121464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9121903Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9122295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9122684Z return func(*args, **kwargs) 2025-08-26T20:41:38.9123075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9123528Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9123983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9124395Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9124537Z 2025-08-26T20:41:38.9124650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9125014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9125340Z return mod(**inputs) 2025-08-26T20:41:38.9125721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9126122Z outputs = self.roberta( 2025-08-26T20:41:38.9127186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9127595Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9128016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9128421Z layer_outputs = layer_module( 2025-08-26T20:41:38.9128775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9129148Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9129547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9129957Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9130367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9130764Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9131199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9131707Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9132153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9132560Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9132700Z 2025-08-26T20:41:38.9132814Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9133177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9133498Z return mod(**inputs) 2025-08-26T20:41:38.9133900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9134308Z outputs = self.roberta( 2025-08-26T20:41:38.9134693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9135104Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9135533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9135936Z layer_outputs = layer_module( 2025-08-26T20:41:38.9136292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9136659Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9137059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9137478Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9137886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9138288Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9138736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9139246Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9139719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9140172Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9140563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9140906Z return self.act(input) 2025-08-26T20:41:38.9141028Z 2025-08-26T20:41:38.9141136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9141554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9141901Z return mod(**inputs) 2025-08-26T20:41:38.9142317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9142737Z outputs = self.roberta( 2025-08-26T20:41:38.9143153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9143589Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9144000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9144397Z layer_outputs = layer_module( 2025-08-26T20:41:38.9144744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9145113Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9145522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9145938Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9146360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9146807Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9147239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9147732Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9148217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9148676Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9148836Z 2025-08-26T20:41:38.9148950Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9149355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9149715Z return mod(**inputs) 2025-08-26T20:41:38.9150120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9150517Z outputs = self.roberta( 2025-08-26T20:41:38.9150901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9151296Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9151679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9152057Z layer_outputs = layer_module( 2025-08-26T20:41:38.9152401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9152760Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9153154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9153570Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9153957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9154336Z return func(*args, **kwargs) 2025-08-26T20:41:38.9154770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9155203Z self_outputs = self.self( 2025-08-26T20:41:38.9155592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9155986Z return func(*args, **kwargs) 2025-08-26T20:41:38.9156433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9157010Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9157316Z 2025-08-26T20:41:38.9157437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9157826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9158191Z return mod(**inputs) 2025-08-26T20:41:38.9158594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9159020Z outputs = self.roberta( 2025-08-26T20:41:38.9159523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9159998Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9160451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9160872Z layer_outputs = layer_module( 2025-08-26T20:41:38.9161245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9161676Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9162075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9162496Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9162883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9163260Z return func(*args, **kwargs) 2025-08-26T20:41:38.9163645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9164062Z self_outputs = self.self( 2025-08-26T20:41:38.9164427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9164805Z return func(*args, **kwargs) 2025-08-26T20:41:38.9165194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9165587Z self.key(current_states) 2025-08-26T20:41:38.9165712Z 2025-08-26T20:41:38.9165818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9166182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9166510Z return mod(**inputs) 2025-08-26T20:41:38.9166887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9167278Z outputs = self.roberta( 2025-08-26T20:41:38.9167662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9168061Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9168457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9168847Z layer_outputs = layer_module( 2025-08-26T20:41:38.9169207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9169598Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9170038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9170486Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9170904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9171322Z return func(*args, **kwargs) 2025-08-26T20:41:38.9171724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9172123Z self_outputs = self.self( 2025-08-26T20:41:38.9172517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9172933Z return func(*args, **kwargs) 2025-08-26T20:41:38.9173357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9173794Z self.value(current_states) 2025-08-26T20:41:38.9173926Z 2025-08-26T20:41:38.9174024Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9174286Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9174689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9175063Z return mod(**inputs) 2025-08-26T20:41:38.9175484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9175929Z outputs = self.roberta( 2025-08-26T20:41:38.9176340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9176804Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9177234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9177671Z layer_outputs = layer_module( 2025-08-26T20:41:38.9178048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9178454Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9178919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9179370Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9179794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9180200Z return func(*args, **kwargs) 2025-08-26T20:41:38.9180625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9181045Z self_outputs = self.self( 2025-08-26T20:41:38.9181433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9181822Z return func(*args, **kwargs) 2025-08-26T20:41:38.9182231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9182723Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9182921Z 2025-08-26T20:41:38.9183044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9183433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9183785Z return mod(**inputs) 2025-08-26T20:41:38.9184190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9184611Z outputs = self.roberta( 2025-08-26T20:41:38.9185010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9185434Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9185854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9186282Z layer_outputs = layer_module( 2025-08-26T20:41:38.9186673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9187078Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9187541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9187987Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9188405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9188804Z return func(*args, **kwargs) 2025-08-26T20:41:38.9189212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9189691Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9190174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9190639Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9190803Z 2025-08-26T20:41:38.9190926Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9191333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9191713Z return mod(**inputs) 2025-08-26T20:41:38.9192135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9192536Z outputs = self.roberta( 2025-08-26T20:41:38.9192938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9193359Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9193781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9194225Z layer_outputs = layer_module( 2025-08-26T20:41:38.9194601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9195002Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9195445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9195890Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9196476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9196914Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9197373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9197876Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9198353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9198790Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9198940Z 2025-08-26T20:41:38.9199064Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9199530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9199897Z return mod(**inputs) 2025-08-26T20:41:38.9200321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9200763Z outputs = self.roberta( 2025-08-26T20:41:38.9201167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9201566Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9201971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9202378Z layer_outputs = layer_module( 2025-08-26T20:41:38.9202793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9203188Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9203588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9203996Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9204403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9204804Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9205236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9205714Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9206159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9206600Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9206988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9207361Z return self.act(input) 2025-08-26T20:41:38.9207490Z 2025-08-26T20:41:38.9207605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9207997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9208348Z return mod(**inputs) 2025-08-26T20:41:38.9208749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9209194Z outputs = self.roberta( 2025-08-26T20:41:38.9209591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9209996Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9210396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9210822Z layer_outputs = layer_module( 2025-08-26T20:41:38.9211187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9211580Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9212011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9212442Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9212860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9213287Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9213742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9214267Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9214750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9215175Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9215330Z 2025-08-26T20:41:38.9215443Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9215832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9216180Z return mod(**inputs) 2025-08-26T20:41:38.9216582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9216999Z outputs = self.roberta( 2025-08-26T20:41:38.9217429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9217856Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9218311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9218729Z layer_outputs = layer_module( 2025-08-26T20:41:38.9219103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9219495Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9219928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9220363Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9220774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9221181Z return func(*args, **kwargs) 2025-08-26T20:41:38.9221569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9221972Z self_outputs = self.self( 2025-08-26T20:41:38.9222363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9222751Z return func(*args, **kwargs) 2025-08-26T20:41:38.9223162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9223747Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9224028Z 2025-08-26T20:41:38.9224147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9224563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9224931Z return mod(**inputs) 2025-08-26T20:41:38.9225333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9225757Z outputs = self.roberta( 2025-08-26T20:41:38.9226167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9226581Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9227001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9227419Z layer_outputs = layer_module( 2025-08-26T20:41:38.9227789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9228184Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9228606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9229040Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9229451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9229846Z return func(*args, **kwargs) 2025-08-26T20:41:38.9230228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9230629Z self_outputs = self.self( 2025-08-26T20:41:38.9231009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9231416Z return func(*args, **kwargs) 2025-08-26T20:41:38.9231802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9232205Z self.key(current_states) 2025-08-26T20:41:38.9232336Z 2025-08-26T20:41:38.9232480Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9232873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9233242Z return mod(**inputs) 2025-08-26T20:41:38.9233648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9234061Z outputs = self.roberta( 2025-08-26T20:41:38.9234465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9234887Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9235302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9235719Z layer_outputs = layer_module( 2025-08-26T20:41:38.9236100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9236489Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9236918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9237379Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9237787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9238195Z return func(*args, **kwargs) 2025-08-26T20:41:38.9238615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9239044Z self_outputs = self.self( 2025-08-26T20:41:38.9239511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9239967Z return func(*args, **kwargs) 2025-08-26T20:41:38.9240399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9240839Z self.value(current_states) 2025-08-26T20:41:38.9240970Z 2025-08-26T20:41:38.9241078Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9241338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9241727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9242082Z return mod(**inputs) 2025-08-26T20:41:38.9242486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9242910Z outputs = self.roberta( 2025-08-26T20:41:38.9243309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9243737Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9244160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9244583Z layer_outputs = layer_module( 2025-08-26T20:41:38.9244952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9245350Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9245782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9246220Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9246637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9247033Z return func(*args, **kwargs) 2025-08-26T20:41:38.9247446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9247920Z self_outputs = self.self( 2025-08-26T20:41:38.9248312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9248725Z return func(*args, **kwargs) 2025-08-26T20:41:38.9249142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9249632Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9249838Z 2025-08-26T20:41:38.9249951Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9250342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9250686Z return mod(**inputs) 2025-08-26T20:41:38.9251101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9251525Z outputs = self.roberta( 2025-08-26T20:41:38.9251932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9252354Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9252763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9253197Z layer_outputs = layer_module( 2025-08-26T20:41:38.9253570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9253960Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9254378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9254813Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9255236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9255613Z return func(*args, **kwargs) 2025-08-26T20:41:38.9255997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9256448Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9256908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9257349Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9257496Z 2025-08-26T20:41:38.9257617Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9258003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9258341Z return mod(**inputs) 2025-08-26T20:41:38.9258745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9259163Z outputs = self.roberta( 2025-08-26T20:41:38.9259564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9259966Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9260377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9260793Z layer_outputs = layer_module( 2025-08-26T20:41:38.9261143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9261507Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9261905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9262316Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9262749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9263149Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9263594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9264077Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9264529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9264940Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9265082Z 2025-08-26T20:41:38.9265196Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9265565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9265894Z return mod(**inputs) 2025-08-26T20:41:38.9266279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9266681Z outputs = self.roberta( 2025-08-26T20:41:38.9267066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9267484Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9267881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9268280Z layer_outputs = layer_module( 2025-08-26T20:41:38.9268632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9268999Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9269395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9269824Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9270232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9270628Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9271062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9271538Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9271981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9272449Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9272861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9273239Z return self.act(input) 2025-08-26T20:41:38.9273367Z 2025-08-26T20:41:38.9273481Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9273887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9274254Z return mod(**inputs) 2025-08-26T20:41:38.9274660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9275076Z outputs = self.roberta( 2025-08-26T20:41:38.9275489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9275919Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9276339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9276764Z layer_outputs = layer_module( 2025-08-26T20:41:38.9277141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9277566Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9278002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9278473Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9278913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9279345Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9279913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9280458Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9280970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9281400Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9281557Z 2025-08-26T20:41:38.9281673Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9282069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9282449Z return mod(**inputs) 2025-08-26T20:41:38.9282858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9283284Z outputs = self.roberta( 2025-08-26T20:41:38.9283695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9284122Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9284544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9284983Z layer_outputs = layer_module( 2025-08-26T20:41:38.9285363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9285764Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9286211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9286664Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9287086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9287504Z return func(*args, **kwargs) 2025-08-26T20:41:38.9287930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9288377Z self_outputs = self.self( 2025-08-26T20:41:38.9288765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9289163Z return func(*args, **kwargs) 2025-08-26T20:41:38.9289575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9290166Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9290461Z 2025-08-26T20:41:38.9290586Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9290991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9291342Z return mod(**inputs) 2025-08-26T20:41:38.9291755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9292192Z outputs = self.roberta( 2025-08-26T20:41:38.9292610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9293048Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9293504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9293945Z layer_outputs = layer_module( 2025-08-26T20:41:38.9294346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9294757Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9295201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9295649Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9296078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9297737Z return func(*args, **kwargs) 2025-08-26T20:41:38.9298246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9298741Z self_outputs = self.self( 2025-08-26T20:41:38.9299168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9299664Z return func(*args, **kwargs) 2025-08-26T20:41:38.9300339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9300782Z self.key(current_states) 2025-08-26T20:41:38.9300922Z 2025-08-26T20:41:38.9301045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9301493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9301876Z return mod(**inputs) 2025-08-26T20:41:38.9302297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9302810Z outputs = self.roberta( 2025-08-26T20:41:38.9303254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9303712Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9304151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9304574Z layer_outputs = layer_module( 2025-08-26T20:41:38.9304967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9305373Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9305820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9306266Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9306681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9307088Z return func(*args, **kwargs) 2025-08-26T20:41:38.9307504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9307954Z self_outputs = self.self( 2025-08-26T20:41:38.9308361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9308757Z return func(*args, **kwargs) 2025-08-26T20:41:38.9309171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9309596Z self.value(current_states) 2025-08-26T20:41:38.9309725Z 2025-08-26T20:41:38.9309824Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9310086Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9310527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9310936Z return mod(**inputs) 2025-08-26T20:41:38.9311357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9311823Z outputs = self.roberta( 2025-08-26T20:41:38.9312232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9312654Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9313091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9313557Z layer_outputs = layer_module( 2025-08-26T20:41:38.9313927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9314323Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9314759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9315204Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9315625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9316045Z return func(*args, **kwargs) 2025-08-26T20:41:38.9316457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9316876Z self_outputs = self.self( 2025-08-26T20:41:38.9317279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9317704Z return func(*args, **kwargs) 2025-08-26T20:41:38.9318119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9318644Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9318852Z 2025-08-26T20:41:38.9318968Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9319364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9320032Z return mod(**inputs) 2025-08-26T20:41:38.9320442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9320913Z outputs = self.roberta( 2025-08-26T20:41:38.9321323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9321762Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9322178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9322610Z layer_outputs = layer_module( 2025-08-26T20:41:38.9322993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9323388Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9323834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9324277Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9324688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9325100Z return func(*args, **kwargs) 2025-08-26T20:41:38.9325517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9326002Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9326500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9326963Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9327110Z 2025-08-26T20:41:38.9327227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9327619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9327950Z return mod(**inputs) 2025-08-26T20:41:38.9328336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9328765Z outputs = self.roberta( 2025-08-26T20:41:38.9329178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9329605Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9330023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9330452Z layer_outputs = layer_module( 2025-08-26T20:41:38.9330818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9331196Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9331619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9332063Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9332503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9332932Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9333411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9333950Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9334428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9334862Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9335011Z 2025-08-26T20:41:38.9335133Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9335521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9335865Z return mod(**inputs) 2025-08-26T20:41:38.9336266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9336685Z outputs = self.roberta( 2025-08-26T20:41:38.9337087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9337501Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9337919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9338340Z layer_outputs = layer_module( 2025-08-26T20:41:38.9338711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9339097Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9339512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9339923Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9340328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9340750Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9341201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9341710Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9342177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9342643Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9343048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9343426Z return self.act(input) 2025-08-26T20:41:38.9343550Z 2025-08-26T20:41:38.9343663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9344062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9344432Z return mod(**inputs) 2025-08-26T20:41:38.9344836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9345257Z outputs = self.roberta( 2025-08-26T20:41:38.9345696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9346137Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9346567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9347017Z layer_outputs = layer_module( 2025-08-26T20:41:38.9347387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9347780Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9348232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9348696Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9349136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9349592Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9350057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9350577Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9351092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9351529Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9351686Z 2025-08-26T20:41:38.9351798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9352186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9352537Z return mod(**inputs) 2025-08-26T20:41:38.9352939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9353354Z outputs = self.roberta( 2025-08-26T20:41:38.9353760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9354233Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9354656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9355080Z layer_outputs = layer_module( 2025-08-26T20:41:38.9355514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9355912Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9356347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9356784Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9357213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9357626Z return func(*args, **kwargs) 2025-08-26T20:41:38.9358069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9358492Z self_outputs = self.self( 2025-08-26T20:41:38.9358879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9359269Z return func(*args, **kwargs) 2025-08-26T20:41:38.9359766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9360391Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9360703Z 2025-08-26T20:41:38.9360824Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9361220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9361576Z return mod(**inputs) 2025-08-26T20:41:38.9361990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9362448Z outputs = self.roberta( 2025-08-26T20:41:38.9362857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9363285Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9363710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9364140Z layer_outputs = layer_module( 2025-08-26T20:41:38.9364515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9364933Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9365363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9365809Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9366232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9366651Z return func(*args, **kwargs) 2025-08-26T20:41:38.9367065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9367491Z self_outputs = self.self( 2025-08-26T20:41:38.9367884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9368286Z return func(*args, **kwargs) 2025-08-26T20:41:38.9368720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9369145Z self.key(current_states) 2025-08-26T20:41:38.9369279Z 2025-08-26T20:41:38.9369396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9369796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9370154Z return mod(**inputs) 2025-08-26T20:41:38.9370565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9370991Z outputs = self.roberta( 2025-08-26T20:41:38.9371406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9371837Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9372265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9372686Z layer_outputs = layer_module( 2025-08-26T20:41:38.9373090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9373487Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9373938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9374388Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9375015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9375416Z return func(*args, **kwargs) 2025-08-26T20:41:38.9375822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9376249Z self_outputs = self.self( 2025-08-26T20:41:38.9376640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9377038Z return func(*args, **kwargs) 2025-08-26T20:41:38.9377589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9378013Z self.value(current_states) 2025-08-26T20:41:38.9378160Z 2025-08-26T20:41:38.9378257Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9378498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9378882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9379238Z return mod(**inputs) 2025-08-26T20:41:38.9379651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9380077Z outputs = self.roberta( 2025-08-26T20:41:38.9380537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9380943Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9381346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9381749Z layer_outputs = layer_module( 2025-08-26T20:41:38.9382126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9382522Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9382928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9383340Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9383738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9384134Z return func(*args, **kwargs) 2025-08-26T20:41:38.9384547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9384967Z self_outputs = self.self( 2025-08-26T20:41:38.9385361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9385763Z return func(*args, **kwargs) 2025-08-26T20:41:38.9386165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9386653Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9386866Z 2025-08-26T20:41:38.9386978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9387369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9387722Z return mod(**inputs) 2025-08-26T20:41:38.9388123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9388563Z outputs = self.roberta( 2025-08-26T20:41:38.9388966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9389409Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9389824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9390250Z layer_outputs = layer_module( 2025-08-26T20:41:38.9390632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9391022Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9391444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9391877Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9392289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9392691Z return func(*args, **kwargs) 2025-08-26T20:41:38.9393097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9393598Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9394082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9394525Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9394681Z 2025-08-26T20:41:38.9394791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9395179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9395575Z return mod(**inputs) 2025-08-26T20:41:38.9395977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9396790Z outputs = self.roberta( 2025-08-26T20:41:38.9397217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9397644Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9398055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9398482Z layer_outputs = layer_module( 2025-08-26T20:41:38.9398871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9399273Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9399785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9400385Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9400838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9401283Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9401748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9402710Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9403187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9403631Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9403786Z 2025-08-26T20:41:38.9403910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9404306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9404650Z return mod(**inputs) 2025-08-26T20:41:38.9405129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9405577Z outputs = self.roberta( 2025-08-26T20:41:38.9406016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9406450Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9406864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9407291Z layer_outputs = layer_module( 2025-08-26T20:41:38.9407645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9408013Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9408407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9408828Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9409238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9409666Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9410098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9410576Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9411022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9411464Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9411875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9412284Z return self.act(input) 2025-08-26T20:41:38.9412398Z 2025-08-26T20:41:38.9412543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9412927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9413258Z return mod(**inputs) 2025-08-26T20:41:38.9413651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9414070Z outputs = self.roberta( 2025-08-26T20:41:38.9414471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9414897Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9415325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9415751Z layer_outputs = layer_module( 2025-08-26T20:41:38.9416098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9416474Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9416877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9417297Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9417699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9418094Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9418526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9419018Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9419481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9419916Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9420060Z 2025-08-26T20:41:38.9420168Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9420576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9420653Z return mod(**inputs) 2025-08-26T20:41:38.9420949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9421023Z outputs = self.roberta( 2025-08-26T20:41:38.9421314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9421395Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9421685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9421766Z layer_outputs = layer_module( 2025-08-26T20:41:38.9422014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9422104Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9422372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9422486Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9422732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9422805Z return func(*args, **kwargs) 2025-08-26T20:41:38.9423080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9423170Z self_outputs = self.self( 2025-08-26T20:41:38.9423426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9423499Z return func(*args, **kwargs) 2025-08-26T20:41:38.9423767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9423985Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9423990Z 2025-08-26T20:41:38.9424096Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9424306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9424373Z return mod(**inputs) 2025-08-26T20:41:38.9424655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9424728Z outputs = self.roberta( 2025-08-26T20:41:38.9424993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9425076Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9425344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9425427Z layer_outputs = layer_module( 2025-08-26T20:41:38.9425653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9425734Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9426006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9426088Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9426343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9426417Z return func(*args, **kwargs) 2025-08-26T20:41:38.9426710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9426783Z self_outputs = self.self( 2025-08-26T20:41:38.9427045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9427128Z return func(*args, **kwargs) 2025-08-26T20:41:38.9427392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9427472Z self.key(current_states) 2025-08-26T20:41:38.9427475Z 2025-08-26T20:41:38.9427579Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9427780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9427858Z return mod(**inputs) 2025-08-26T20:41:38.9428126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9428204Z outputs = self.roberta( 2025-08-26T20:41:38.9428468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9428569Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9428848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9428924Z layer_outputs = layer_module( 2025-08-26T20:41:38.9429172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9429260Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9429537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9429634Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9429877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9429955Z return func(*args, **kwargs) 2025-08-26T20:41:38.9430235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9430319Z self_outputs = self.self( 2025-08-26T20:41:38.9430580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9430662Z return func(*args, **kwargs) 2025-08-26T20:41:38.9430932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9431006Z self.value(current_states) 2025-08-26T20:41:38.9431010Z 2025-08-26T20:41:38.9431101Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9431207Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9431427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9431497Z return mod(**inputs) 2025-08-26T20:41:38.9431781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9431862Z outputs = self.roberta( 2025-08-26T20:41:38.9432141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9432225Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9432504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9432581Z layer_outputs = layer_module( 2025-08-26T20:41:38.9432825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9432911Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9433214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9433317Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9433582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9433665Z return func(*args, **kwargs) 2025-08-26T20:41:38.9433948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9434030Z self_outputs = self.self( 2025-08-26T20:41:38.9434288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9434369Z return func(*args, **kwargs) 2025-08-26T20:41:38.9434652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9434798Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9434802Z 2025-08-26T20:41:38.9434920Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9435136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9435231Z return mod(**inputs) 2025-08-26T20:41:38.9435517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9435590Z outputs = self.roberta( 2025-08-26T20:41:38.9435878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9435957Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9436254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9436334Z layer_outputs = layer_module( 2025-08-26T20:41:38.9436571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9436666Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9436946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9437039Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9437309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9437390Z return func(*args, **kwargs) 2025-08-26T20:41:38.9437668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9437808Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9438096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9438185Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9438188Z 2025-08-26T20:41:38.9438307Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9438518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9438589Z return mod(**inputs) 2025-08-26T20:41:38.9438886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9438962Z outputs = self.roberta( 2025-08-26T20:41:38.9439259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9439342Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9439738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9439827Z layer_outputs = layer_module( 2025-08-26T20:41:38.9440087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9440191Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9440494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9440593Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9440875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9440959Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9441284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9441417Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9441711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9441803Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9441840Z 2025-08-26T20:41:38.9441961Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9442174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9442245Z return mod(**inputs) 2025-08-26T20:41:38.9442537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9442612Z outputs = self.roberta( 2025-08-26T20:41:38.9442900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9442997Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9443281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9443368Z layer_outputs = layer_module( 2025-08-26T20:41:38.9443617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9443706Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9443972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9444054Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9444323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9444403Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9444710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9444834Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9445116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9445232Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9445452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9445532Z return self.act(input) 2025-08-26T20:41:38.9445536Z 2025-08-26T20:41:38.9445642Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9445851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9445916Z return mod(**inputs) 2025-08-26T20:41:38.9446188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9446264Z outputs = self.roberta( 2025-08-26T20:41:38.9446557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9446659Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9446929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9447010Z layer_outputs = layer_module( 2025-08-26T20:41:38.9447238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9447321Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9447596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9447686Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9447957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9448040Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9448344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9448505Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9448791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9448885Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9448889Z 2025-08-26T20:41:38.9449000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9449267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9449362Z return mod(**inputs) 2025-08-26T20:41:38.9449635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9449711Z outputs = self.roberta( 2025-08-26T20:41:38.9449980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9450061Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9450329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9450402Z layer_outputs = layer_module( 2025-08-26T20:41:38.9450636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9450714Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9450991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9451079Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9451338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9451410Z return func(*args, **kwargs) 2025-08-26T20:41:38.9451682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9451762Z self_outputs = self.self( 2025-08-26T20:41:38.9452016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9452092Z return func(*args, **kwargs) 2025-08-26T20:41:38.9452361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9452574Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9452580Z 2025-08-26T20:41:38.9452695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9452920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9452994Z return mod(**inputs) 2025-08-26T20:41:38.9453284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9453356Z outputs = self.roberta( 2025-08-26T20:41:38.9453626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9453700Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9453975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9454050Z layer_outputs = layer_module( 2025-08-26T20:41:38.9454285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9454367Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9454634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9454726Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9454993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9455070Z return func(*args, **kwargs) 2025-08-26T20:41:38.9455334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9455405Z self_outputs = self.self( 2025-08-26T20:41:38.9455659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9455747Z return func(*args, **kwargs) 2025-08-26T20:41:38.9456018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9456091Z self.key(current_states) 2025-08-26T20:41:38.9456094Z 2025-08-26T20:41:38.9456205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9456406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9456473Z return mod(**inputs) 2025-08-26T20:41:38.9456749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9456818Z outputs = self.roberta( 2025-08-26T20:41:38.9457088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9457161Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9457426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9457507Z layer_outputs = layer_module( 2025-08-26T20:41:38.9457730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9457818Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9458083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9458166Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9458436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9458512Z return func(*args, **kwargs) 2025-08-26T20:41:38.9458797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9458874Z self_outputs = self.self( 2025-08-26T20:41:38.9459165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9459241Z return func(*args, **kwargs) 2025-08-26T20:41:38.9459537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9459629Z self.value(current_states) 2025-08-26T20:41:38.9459633Z 2025-08-26T20:41:38.9459722Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9459841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9460052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9460121Z return mod(**inputs) 2025-08-26T20:41:38.9460410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9460484Z outputs = self.roberta( 2025-08-26T20:41:38.9460770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9460848Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9461127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9461228Z layer_outputs = layer_module( 2025-08-26T20:41:38.9461466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9461557Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9461844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9461941Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9462207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9462299Z return func(*args, **kwargs) 2025-08-26T20:41:38.9462589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9462665Z self_outputs = self.self( 2025-08-26T20:41:38.9462943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9463017Z return func(*args, **kwargs) 2025-08-26T20:41:38.9463302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9463454Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9463458Z 2025-08-26T20:41:38.9463569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9463788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9463859Z return mod(**inputs) 2025-08-26T20:41:38.9464159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9464238Z outputs = self.roberta( 2025-08-26T20:41:38.9464521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9464612Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9464891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9464973Z layer_outputs = layer_module( 2025-08-26T20:41:38.9465213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9465296Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9465585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9465677Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9465964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9466040Z return func(*args, **kwargs) 2025-08-26T20:41:38.9466339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9466489Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9466768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9466866Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9466870Z 2025-08-26T20:41:38.9466982Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9467207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9467281Z return mod(**inputs) 2025-08-26T20:41:38.9467569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9467653Z outputs = self.roberta( 2025-08-26T20:41:38.9467935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9468045Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9468325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9468403Z layer_outputs = layer_module( 2025-08-26T20:41:38.9468650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9468733Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9469057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9469149Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9469435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9469521Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9469840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9469981Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9470281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9470377Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9470381Z 2025-08-26T20:41:38.9470493Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9470706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9470799Z return mod(**inputs) 2025-08-26T20:41:38.9471082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9471165Z outputs = self.roberta( 2025-08-26T20:41:38.9471441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9471527Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9471807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9471883Z layer_outputs = layer_module( 2025-08-26T20:41:38.9472129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9472215Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9472520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9472609Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9472903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9472997Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9473310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9473447Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9473747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9473868Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9474105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9474184Z return self.act(input) 2025-08-26T20:41:38.9474187Z 2025-08-26T20:41:38.9474305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9474520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9474615Z return mod(**inputs) 2025-08-26T20:41:38.9474895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9474969Z outputs = self.roberta( 2025-08-26T20:41:38.9475257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9475336Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9475618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9475715Z layer_outputs = layer_module( 2025-08-26T20:41:38.9475953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9476044Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9476324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9476421Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9476697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9476785Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9477096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9477241Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9477528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9477614Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9477618Z 2025-08-26T20:41:38.9477737Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9477950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9478021Z return mod(**inputs) 2025-08-26T20:41:38.9478308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9478382Z outputs = self.roberta( 2025-08-26T20:41:38.9478669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9478749Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9479036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9479132Z layer_outputs = layer_module( 2025-08-26T20:41:38.9479372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9479574Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9479862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9479956Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9480223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9480301Z return func(*args, **kwargs) 2025-08-26T20:41:38.9480598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9480689Z self_outputs = self.self( 2025-08-26T20:41:38.9480956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9481033Z return func(*args, **kwargs) 2025-08-26T20:41:38.9481315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9481572Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9481576Z 2025-08-26T20:41:38.9481687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9481914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9481988Z return mod(**inputs) 2025-08-26T20:41:38.9482279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9482375Z outputs = self.roberta( 2025-08-26T20:41:38.9482664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9482752Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9483041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9483126Z layer_outputs = layer_module( 2025-08-26T20:41:38.9483367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9483451Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9483743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9483831Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9484114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9484185Z return func(*args, **kwargs) 2025-08-26T20:41:38.9484463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9484537Z self_outputs = self.self( 2025-08-26T20:41:38.9484801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9484885Z return func(*args, **kwargs) 2025-08-26T20:41:38.9485167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9485252Z self.key(current_states) 2025-08-26T20:41:38.9485256Z 2025-08-26T20:41:38.9485366Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9485584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9485664Z return mod(**inputs) 2025-08-26T20:41:38.9485971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9486056Z outputs = self.roberta( 2025-08-26T20:41:38.9486352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9486434Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9486717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9486794Z layer_outputs = layer_module( 2025-08-26T20:41:38.9487037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9487122Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9487405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9487493Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9487750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9487832Z return func(*args, **kwargs) 2025-08-26T20:41:38.9488128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9488208Z self_outputs = self.self( 2025-08-26T20:41:38.9488464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9488536Z return func(*args, **kwargs) 2025-08-26T20:41:38.9488818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9488913Z self.value(current_states) 2025-08-26T20:41:38.9488917Z 2025-08-26T20:41:38.9489011Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9489126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9489339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9489418Z return mod(**inputs) 2025-08-26T20:41:38.9489709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9489791Z outputs = self.roberta( 2025-08-26T20:41:38.9490073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9490158Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9490439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9490519Z layer_outputs = layer_module( 2025-08-26T20:41:38.9490766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9490846Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9491121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9491211Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9491469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9491552Z return func(*args, **kwargs) 2025-08-26T20:41:38.9491836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9491918Z self_outputs = self.self( 2025-08-26T20:41:38.9492176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9492252Z return func(*args, **kwargs) 2025-08-26T20:41:38.9492561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9492700Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9492704Z 2025-08-26T20:41:38.9492829Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9493032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9493106Z return mod(**inputs) 2025-08-26T20:41:38.9493375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9493447Z outputs = self.roberta( 2025-08-26T20:41:38.9493730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9493810Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9494094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9494170Z layer_outputs = layer_module( 2025-08-26T20:41:38.9494409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9494526Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9494804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9494902Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9495159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9495239Z return func(*args, **kwargs) 2025-08-26T20:41:38.9495519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9495677Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9495969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9496060Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9496065Z 2025-08-26T20:41:38.9496381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9496679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9496752Z return mod(**inputs) 2025-08-26T20:41:38.9497046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9497120Z outputs = self.roberta( 2025-08-26T20:41:38.9497409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9497494Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9497784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9497860Z layer_outputs = layer_module( 2025-08-26T20:41:38.9498096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9498191Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9498468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9498569Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9498846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9498929Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9499251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9499439Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9499761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9499853Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9499857Z 2025-08-26T20:41:38.9499973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9500202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9500274Z return mod(**inputs) 2025-08-26T20:41:38.9500564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9500639Z outputs = self.roberta( 2025-08-26T20:41:38.9500926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9501007Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9501285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9501371Z layer_outputs = layer_module( 2025-08-26T20:41:38.9501634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9501729Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9502006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9502095Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9502382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9502491Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9502812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9502943Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9503227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9503351Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9503579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9503665Z return self.act(input) 2025-08-26T20:41:38.9503668Z 2025-08-26T20:41:38.9503777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9504011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9504086Z return mod(**inputs) 2025-08-26T20:41:38.9504374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9504451Z outputs = self.roberta( 2025-08-26T20:41:38.9504714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9504796Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9505069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9505151Z layer_outputs = layer_module( 2025-08-26T20:41:38.9505388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9505473Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9505758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9505843Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9506128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9506208Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9506526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9506675Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9506948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9507042Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9507046Z 2025-08-26T20:41:38.9507158Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9507376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9507462Z return mod(**inputs) 2025-08-26T20:41:38.9507731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9507810Z outputs = self.roberta( 2025-08-26T20:41:38.9508076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9508201Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9508463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9508535Z layer_outputs = layer_module( 2025-08-26T20:41:38.9508767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9508847Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9509135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9509221Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9509465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9509545Z return func(*args, **kwargs) 2025-08-26T20:41:38.9509811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9509891Z self_outputs = self.self( 2025-08-26T20:41:38.9510135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9510212Z return func(*args, **kwargs) 2025-08-26T20:41:38.9510473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9510683Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9510686Z 2025-08-26T20:41:38.9510800Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9511002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9511077Z return mod(**inputs) 2025-08-26T20:41:38.9511346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9511416Z outputs = self.roberta( 2025-08-26T20:41:38.9511685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9511759Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9512031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9512104Z layer_outputs = layer_module( 2025-08-26T20:41:38.9512350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9512431Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9512708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9512800Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9513045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9513126Z return func(*args, **kwargs) 2025-08-26T20:41:38.9513404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9513480Z self_outputs = self.self( 2025-08-26T20:41:38.9513746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9513822Z return func(*args, **kwargs) 2025-08-26T20:41:38.9514108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9514185Z self.key(current_states) 2025-08-26T20:41:38.9514189Z 2025-08-26T20:41:38.9514301Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9514544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9514616Z return mod(**inputs) 2025-08-26T20:41:38.9514903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9514976Z outputs = self.roberta( 2025-08-26T20:41:38.9515264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9515359Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9515643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9515729Z layer_outputs = layer_module( 2025-08-26T20:41:38.9515969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9516062Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9516342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9516429Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9516697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9516772Z return func(*args, **kwargs) 2025-08-26T20:41:38.9517074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9517152Z self_outputs = self.self( 2025-08-26T20:41:38.9517421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9517503Z return func(*args, **kwargs) 2025-08-26T20:41:38.9517795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9517882Z self.value(current_states) 2025-08-26T20:41:38.9517886Z 2025-08-26T20:41:38.9517975Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9518093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9518306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9518377Z return mod(**inputs) 2025-08-26T20:41:38.9518671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9518748Z outputs = self.roberta( 2025-08-26T20:41:38.9519059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9519141Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9519529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9519627Z layer_outputs = layer_module( 2025-08-26T20:41:38.9519882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9519979Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9520281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9520373Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9520650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9520739Z return func(*args, **kwargs) 2025-08-26T20:41:38.9521030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9521106Z self_outputs = self.self( 2025-08-26T20:41:38.9521402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9521472Z return func(*args, **kwargs) 2025-08-26T20:41:38.9521734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9521879Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9521883Z 2025-08-26T20:41:38.9521997Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9522243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9522315Z return mod(**inputs) 2025-08-26T20:41:38.9522612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9522696Z outputs = self.roberta( 2025-08-26T20:41:38.9522989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9523075Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9523364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9523448Z layer_outputs = layer_module( 2025-08-26T20:41:38.9523691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9523776Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9524073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9524163Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9524437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9524514Z return func(*args, **kwargs) 2025-08-26T20:41:38.9524819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9524968Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9525267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9525363Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9525367Z 2025-08-26T20:41:38.9525481Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9525705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9525801Z return mod(**inputs) 2025-08-26T20:41:38.9526094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9526193Z outputs = self.roberta( 2025-08-26T20:41:38.9526482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9526569Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9526866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9526943Z layer_outputs = layer_module( 2025-08-26T20:41:38.9527192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9527279Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9527575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9527668Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9527953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9528064Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9528386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9528525Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9528808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9528898Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9528919Z 2025-08-26T20:41:38.9529024Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9529228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9529303Z return mod(**inputs) 2025-08-26T20:41:38.9529581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9529658Z outputs = self.roberta( 2025-08-26T20:41:38.9529912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9529984Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9530249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9530320Z layer_outputs = layer_module( 2025-08-26T20:41:38.9530545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9530623Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9530887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9530970Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9531224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9531309Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9531597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9531723Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9531988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9532106Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9532350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9532424Z return self.act(input) 2025-08-26T20:41:38.9532428Z 2025-08-26T20:41:38.9532559Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9532763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9532838Z return mod(**inputs) 2025-08-26T20:41:38.9533105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9533176Z outputs = self.roberta( 2025-08-26T20:41:38.9533448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9533521Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9533806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9533877Z layer_outputs = layer_module( 2025-08-26T20:41:38.9534096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9534184Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9534456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9534544Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9534798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9534873Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9535169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9535334Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9535598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9535680Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9535685Z 2025-08-26T20:41:38.9535798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9535996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9536064Z return mod(**inputs) 2025-08-26T20:41:38.9536331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9536401Z outputs = self.roberta( 2025-08-26T20:41:38.9536669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9536747Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9537011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9537094Z layer_outputs = layer_module( 2025-08-26T20:41:38.9537320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9537412Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9537680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9537773Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9538023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9538097Z return func(*args, **kwargs) 2025-08-26T20:41:38.9538371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9538448Z self_outputs = self.self( 2025-08-26T20:41:38.9538718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9538791Z return func(*args, **kwargs) 2025-08-26T20:41:38.9539088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9539303Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9539307Z 2025-08-26T20:41:38.9539409Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9539614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9539680Z return mod(**inputs) 2025-08-26T20:41:38.9539955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9540024Z outputs = self.roberta( 2025-08-26T20:41:38.9540286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9540371Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9540652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9540731Z layer_outputs = layer_module( 2025-08-26T20:41:38.9540955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9541033Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9541303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9541400Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9541657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9541727Z return func(*args, **kwargs) 2025-08-26T20:41:38.9541990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9542070Z self_outputs = self.self( 2025-08-26T20:41:38.9542315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9542389Z return func(*args, **kwargs) 2025-08-26T20:41:38.9542650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9542728Z self.key(current_states) 2025-08-26T20:41:38.9542732Z 2025-08-26T20:41:38.9542835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9543038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9543111Z return mod(**inputs) 2025-08-26T20:41:38.9543377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9543455Z outputs = self.roberta( 2025-08-26T20:41:38.9543720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9543797Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9544066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9544139Z layer_outputs = layer_module( 2025-08-26T20:41:38.9544368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9544448Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9544762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9544858Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9545186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9545266Z return func(*args, **kwargs) 2025-08-26T20:41:38.9545530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9545607Z self_outputs = self.self( 2025-08-26T20:41:38.9545851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9545920Z return func(*args, **kwargs) 2025-08-26T20:41:38.9546192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9546267Z self.value(current_states) 2025-08-26T20:41:38.9546270Z 2025-08-26T20:41:38.9546361Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9546467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9546669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9546769Z return mod(**inputs) 2025-08-26T20:41:38.9547036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9547113Z outputs = self.roberta( 2025-08-26T20:41:38.9547374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9547447Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9547717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9548702Z layer_outputs = layer_module( 2025-08-26T20:41:38.9548938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9549018Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9549293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9549376Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9549619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9549699Z return func(*args, **kwargs) 2025-08-26T20:41:38.9549959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9550040Z self_outputs = self.self( 2025-08-26T20:41:38.9550299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9550374Z return func(*args, **kwargs) 2025-08-26T20:41:38.9550664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9550807Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9550812Z 2025-08-26T20:41:38.9550929Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9551140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9551217Z return mod(**inputs) 2025-08-26T20:41:38.9551517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9551590Z outputs = self.roberta( 2025-08-26T20:41:38.9551875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9551966Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9552254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9552328Z layer_outputs = layer_module( 2025-08-26T20:41:38.9552567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9552656Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9552920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9553009Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9553252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9553323Z return func(*args, **kwargs) 2025-08-26T20:41:38.9553594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9553724Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9553995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9554099Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9554103Z 2025-08-26T20:41:38.9554214Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9554416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9554482Z return mod(**inputs) 2025-08-26T20:41:38.9554758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9554828Z outputs = self.roberta( 2025-08-26T20:41:38.9555118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9555193Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9555459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9555540Z layer_outputs = layer_module( 2025-08-26T20:41:38.9555764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9555850Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9556111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9556204Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9556463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9556542Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9556846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9556969Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9557237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9557320Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9557324Z 2025-08-26T20:41:38.9557430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9557652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9557722Z return mod(**inputs) 2025-08-26T20:41:38.9558008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9558083Z outputs = self.roberta( 2025-08-26T20:41:38.9558385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9558464Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9558761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9558848Z layer_outputs = layer_module( 2025-08-26T20:41:38.9559084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9559175Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9559550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9559651Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9559946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9560035Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9560368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9560505Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9560824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9560947Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9561176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9561263Z return self.act(input) 2025-08-26T20:41:38.9561267Z 2025-08-26T20:41:38.9561382Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9561608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9561707Z return mod(**inputs) 2025-08-26T20:41:38.9562001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9562087Z outputs = self.roberta( 2025-08-26T20:41:38.9562378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9562468Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9562755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9562834Z layer_outputs = layer_module( 2025-08-26T20:41:38.9563086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9563172Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9563472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9563566Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9563858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9563944Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9564271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9564427Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9564716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9564815Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9564819Z 2025-08-26T20:41:38.9564932Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9565154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9565258Z return mod(**inputs) 2025-08-26T20:41:38.9565552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9565650Z outputs = self.roberta( 2025-08-26T20:41:38.9565939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9566028Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9566323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9566403Z layer_outputs = layer_module( 2025-08-26T20:41:38.9566660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9566750Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9567050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9567145Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9567415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9567520Z return func(*args, **kwargs) 2025-08-26T20:41:38.9567807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9567892Z self_outputs = self.self( 2025-08-26T20:41:38.9568161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9568239Z return func(*args, **kwargs) 2025-08-26T20:41:38.9568535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-26T20:41:38.9568791Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-26T20:41:38.9568795Z 2025-08-26T20:41:38.9568918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9569138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9569219Z return mod(**inputs) 2025-08-26T20:41:38.9569508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9569583Z outputs = self.roberta( 2025-08-26T20:41:38.9569882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9569956Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9570223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9570296Z layer_outputs = layer_module( 2025-08-26T20:41:38.9570521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9570607Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9570869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9570961Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9571214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9571296Z return func(*args, **kwargs) 2025-08-26T20:41:38.9571573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9571649Z self_outputs = self.self( 2025-08-26T20:41:38.9571918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9572012Z return func(*args, **kwargs) 2025-08-26T20:41:38.9572304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-26T20:41:38.9572396Z self.key(current_states) 2025-08-26T20:41:38.9572401Z 2025-08-26T20:41:38.9572521Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9572730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9572798Z return mod(**inputs) 2025-08-26T20:41:38.9573070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9573138Z outputs = self.roberta( 2025-08-26T20:41:38.9573398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9573481Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9573741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9573821Z layer_outputs = layer_module( 2025-08-26T20:41:38.9574046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9574148Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9574409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9574490Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9574736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9574809Z return func(*args, **kwargs) 2025-08-26T20:41:38.9575093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9575167Z self_outputs = self.self( 2025-08-26T20:41:38.9575413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9575495Z return func(*args, **kwargs) 2025-08-26T20:41:38.9575763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-26T20:41:38.9575847Z self.value(current_states) 2025-08-26T20:41:38.9575851Z 2025-08-26T20:41:38.9575938Z cudagraph partition due to non gpu ops 2025-08-26T20:41:38.9576054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9576259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9576328Z return mod(**inputs) 2025-08-26T20:41:38.9576605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9576685Z outputs = self.roberta( 2025-08-26T20:41:38.9576973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9577055Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9577340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9577426Z layer_outputs = layer_module( 2025-08-26T20:41:38.9577669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9577762Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9578043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9578135Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9578421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9578498Z return func(*args, **kwargs) 2025-08-26T20:41:38.9578803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-26T20:41:38.9578880Z self_outputs = self.self( 2025-08-26T20:41:38.9579143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9579218Z return func(*args, **kwargs) 2025-08-26T20:41:38.9579497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-26T20:41:38.9579647Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-26T20:41:38.9579651Z 2025-08-26T20:41:38.9579764Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9579984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9580054Z return mod(**inputs) 2025-08-26T20:41:38.9580340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9580443Z outputs = self.roberta( 2025-08-26T20:41:38.9580729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9580813Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9581098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9581174Z layer_outputs = layer_module( 2025-08-26T20:41:38.9581422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9581524Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9581815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-26T20:41:38.9581903Z self_attention_outputs = self.attention( 2025-08-26T20:41:38.9582179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-26T20:41:38.9582258Z return func(*args, **kwargs) 2025-08-26T20:41:38.9582536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-26T20:41:38.9582681Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:41:38.9582964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-26T20:41:38.9583059Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9583065Z 2025-08-26T20:41:38.9583173Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9583388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9583466Z return mod(**inputs) 2025-08-26T20:41:38.9583750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9583832Z outputs = self.roberta( 2025-08-26T20:41:38.9584111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9584195Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9584471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9584547Z layer_outputs = layer_module( 2025-08-26T20:41:38.9584791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9584876Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9585181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9585294Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9585572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9585662Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9585973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9586109Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9586388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-26T20:41:38.9586486Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9586490Z 2025-08-26T20:41:38.9586601Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9586817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9586896Z return mod(**inputs) 2025-08-26T20:41:38.9587178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9587277Z outputs = self.roberta( 2025-08-26T20:41:38.9587555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9587633Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9587918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9588012Z layer_outputs = layer_module( 2025-08-26T20:41:38.9588260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9588344Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9588623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9588722Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9588998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9589084Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9589393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-26T20:41:38.9589527Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:41:38.9589804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-26T20:41:38.9589926Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:41:38.9590161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:41:38.9590236Z return self.act(input) 2025-08-26T20:41:38.9590241Z 2025-08-26T20:41:38.9590359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9590570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9590641Z return mod(**inputs) 2025-08-26T20:41:38.9590925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-26T20:41:38.9590998Z outputs = self.roberta( 2025-08-26T20:41:38.9591279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-26T20:41:38.9591357Z encoder_outputs = self.encoder( 2025-08-26T20:41:38.9591655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-26T20:41:38.9591733Z layer_outputs = layer_module( 2025-08-26T20:41:38.9591984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:41:38.9592080Z return super().__call__(*args, **kwargs) 2025-08-26T20:41:38.9592359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-26T20:41:38.9592454Z layer_output = apply_chunking_to_forward( 2025-08-26T20:41:38.9592729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:41:38.9592810Z return forward_fn(*input_tensors) 2025-08-26T20:41:38.9593129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-26T20:41:38.9593272Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:41:38.9593559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-26T20:41:38.9593644Z hidden_states = self.dense(hidden_states) 2025-08-26T20:41:38.9593678Z 2025-08-26T20:41:38.9593795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9594011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9594082Z return mod(**inputs) 2025-08-26T20:41:38.9594371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1530, in forward 2025-08-26T20:41:38.9594459Z logits = self.qa_outputs(sequence_output) 2025-08-26T20:41:38.9594480Z 2025-08-26T20:41:38.9594602Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9594818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9594889Z return mod(**inputs) 2025-08-26T20:41:38.9595183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in forward 2025-08-26T20:41:38.9595297Z start_loss = loss_fct(start_logits, start_positions) 2025-08-26T20:41:38.9595301Z 2025-08-26T20:41:38.9595418Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:41:38.9595627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:41:38.9595702Z return mod(**inputs) 2025-08-26T20:41:38.9595986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in forward 2025-08-26T20:41:38.9596087Z end_loss = loss_fct(end_logits, end_positions) 2025-08-26T20:41:38.9596093Z 2025-08-26T20:41:46.9573244Z Compilation time (from dynamo_timed): 14.849337619 2025-08-26T20:41:46.9573574Z pass 2025-08-26T20:41:46.9573927Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:46.9574787Z TIMING: _recursive_pre_grad_passes:0.00744 _recursive_joint_graph_passes:0.39822 _recursive_post_grad_passes:0.08569 async_compile.wait:0.00263 code_gen:6.87012 inductor_compile:8.20115 backend_compile:11.5708 gc:0.00021 entire_frame_compile:14.84934 total_wall_time:14.84934 2025-08-26T20:41:46.9575853Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12459 | FakeTensor.__torch_dispatch__:4435 | ProxyTorchDispatchMode.__torch_dispatch__:4566 2025-08-26T20:41:46.9576365Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-08-26T20:41:52.3553306Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:41:52.3554311Z from pkg_resources import resource_filename 2025-08-26T20:41:52.9668230Z 2025-08-26T20:41:54.1077136Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:41:54.1081132Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:41:54.1088305Z cpu eval T5ForConditionalGeneration 2025-08-26T20:41:55.4879287Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:55.9028014Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:41:56.3341164Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:06.5463554Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5464721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5465223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5465664Z return mod(**inputs) 2025-08-26T20:42:06.5466093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5466541Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5467300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5467707Z layer_outputs = layer_module( 2025-08-26T20:42:06.5468109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5468521Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5468939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5469426Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5469878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5470328Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5470788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward 2025-08-26T20:42:06.5471235Z position_bias = position_bias + causal_mask 2025-08-26T20:42:06.5471402Z 2025-08-26T20:42:06.5471535Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5471937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5472294Z return mod(**inputs) 2025-08-26T20:42:06.5472672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5473098Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5473542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5473966Z layer_outputs = layer_module( 2025-08-26T20:42:06.5474362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5474776Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5475195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5475620Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5476050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.5476520Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5476971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5477389Z return self.weight * hidden_states 2025-08-26T20:42:06.5477548Z 2025-08-26T20:42:06.5477667Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5478124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5478500Z return mod(**inputs) 2025-08-26T20:42:06.5478953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5479381Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5480200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5480621Z layer_outputs = layer_module( 2025-08-26T20:42:06.5481011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5481416Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5481818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5482227Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5482641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5483057Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5483494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5483900Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5484053Z 2025-08-26T20:42:06.5484166Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5484559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5484918Z return mod(**inputs) 2025-08-26T20:42:06.5485310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5485753Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5486159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5486562Z layer_outputs = layer_module( 2025-08-26T20:42:06.5486932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5487333Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5487732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5488144Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5488552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5488961Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5489366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5489776Z key_states = self.k(current_states) 2025-08-26T20:42:06.5489919Z 2025-08-26T20:42:06.5490039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5490416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5490773Z return mod(**inputs) 2025-08-26T20:42:06.5491157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5491547Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5491937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5492376Z layer_outputs = layer_module( 2025-08-26T20:42:06.5492744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5493131Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5493558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5493973Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5494397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5494826Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5495226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5495689Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5495885Z 2025-08-26T20:42:06.5495997Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5496682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5497040Z return mod(**inputs) 2025-08-26T20:42:06.5497420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5497826Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5498217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5498662Z layer_outputs = layer_module( 2025-08-26T20:42:06.5499050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5499453Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5499864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5500280Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5500690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5501942Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5502362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5502857Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5503105Z 2025-08-26T20:42:06.5503224Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5503626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5503992Z return mod(**inputs) 2025-08-26T20:42:06.5504380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5504783Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5505192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5505606Z layer_outputs = layer_module( 2025-08-26T20:42:06.5505993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5506393Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5506803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5507222Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5507635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5508051Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5508454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.5508866Z value_states = self.v(current_states) 2025-08-26T20:42:06.5509022Z 2025-08-26T20:42:06.5509139Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5509534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5509921Z return mod(**inputs) 2025-08-26T20:42:06.5510311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5510773Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5511178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5511590Z layer_outputs = layer_module( 2025-08-26T20:42:06.5511965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5512364Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5512772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5513195Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5513605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5514016Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5514428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5514904Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5515081Z 2025-08-26T20:42:06.5515201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5515596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5515946Z return mod(**inputs) 2025-08-26T20:42:06.5516328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5516766Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5517168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5517573Z layer_outputs = layer_module( 2025-08-26T20:42:06.5517951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5518352Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5518765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5519177Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5519661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5520086Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5520499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5520946Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5521124Z 2025-08-26T20:42:06.5521249Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5521638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5521998Z return mod(**inputs) 2025-08-26T20:42:06.5522385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5522793Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5523186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5523594Z layer_outputs = layer_module( 2025-08-26T20:42:06.5523982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5524380Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5524801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5525236Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5525653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5526059Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5526462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.5526899Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.5527073Z 2025-08-26T20:42:06.5527186Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5527582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5527941Z return mod(**inputs) 2025-08-26T20:42:06.5528328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5528729Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5529136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5529548Z layer_outputs = layer_module( 2025-08-26T20:42:06.5529951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5530441Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5530835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5531243Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5531643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5532069Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5532457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.5532855Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.5533003Z 2025-08-26T20:42:06.5533114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5533513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5533871Z return mod(**inputs) 2025-08-26T20:42:06.5534244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5534670Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5535058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5535449Z layer_outputs = layer_module( 2025-08-26T20:42:06.5535820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5536204Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5536601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.5537002Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.5537401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.5537799Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.5538201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5538595Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5538739Z 2025-08-26T20:42:06.5538859Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5539241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5539582Z return mod(**inputs) 2025-08-26T20:42:06.5540045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5540461Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5540883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5541292Z layer_outputs = layer_module( 2025-08-26T20:42:06.5541682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5542079Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5542480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5542884Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5543275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5543682Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5544083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5544484Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5544626Z 2025-08-26T20:42:06.5545258Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5545637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5545991Z return mod(**inputs) 2025-08-26T20:42:06.5546367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5546768Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5547161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5547574Z layer_outputs = layer_module( 2025-08-26T20:42:06.5547948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5548333Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5548732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5549131Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5549542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5549960Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5550371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5550791Z key_states = self.k(current_states) 2025-08-26T20:42:06.5550956Z 2025-08-26T20:42:06.5551094Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5551493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5551848Z return mod(**inputs) 2025-08-26T20:42:06.5552217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5552628Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5553026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5553436Z layer_outputs = layer_module( 2025-08-26T20:42:06.5553825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5554224Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5554629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5555050Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5555481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5555901Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5556360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5556825Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5557033Z 2025-08-26T20:42:06.5557147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5557545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5557908Z return mod(**inputs) 2025-08-26T20:42:06.5558281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5558687Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5559085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5559580Z layer_outputs = layer_module( 2025-08-26T20:42:06.5559968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5560363Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5560800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5561232Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5561646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5562070Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5562476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5562998Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5563240Z 2025-08-26T20:42:06.5563357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5563757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5564118Z return mod(**inputs) 2025-08-26T20:42:06.5564499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5564936Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5565306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5565731Z layer_outputs = layer_module( 2025-08-26T20:42:06.5566076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5566446Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5566826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5567206Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5567607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5568015Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5568412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5568866Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5569079Z 2025-08-26T20:42:06.5569191Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5569554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5569883Z return mod(**inputs) 2025-08-26T20:42:06.5570237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5570632Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5571005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5571451Z layer_outputs = layer_module( 2025-08-26T20:42:06.5571830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5572222Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5572624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5573026Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5573416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5573841Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5574254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5574726Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5574938Z 2025-08-26T20:42:06.5575050Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5575427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5575758Z return mod(**inputs) 2025-08-26T20:42:06.5576111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5576509Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5576891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5577319Z layer_outputs = layer_module( 2025-08-26T20:42:06.5577691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5578081Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5578474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5578846Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5579251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5579658Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5580060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.5580453Z value_states = self.v(current_states) 2025-08-26T20:42:06.5580607Z 2025-08-26T20:42:06.5580718Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5581103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5581462Z return mod(**inputs) 2025-08-26T20:42:06.5581815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5582192Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5582585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5582990Z layer_outputs = layer_module( 2025-08-26T20:42:06.5583359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5583737Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5584137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5584542Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5584944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5585377Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5585787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5586221Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5586406Z 2025-08-26T20:42:06.5586517Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5586906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5587260Z return mod(**inputs) 2025-08-26T20:42:06.5587625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5588020Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5588412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5588806Z layer_outputs = layer_module( 2025-08-26T20:42:06.5589172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5589559Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5589974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5590379Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5590778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5591174Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5591573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5592030Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5592193Z 2025-08-26T20:42:06.5592305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5592669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5593007Z return mod(**inputs) 2025-08-26T20:42:06.5593381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5593780Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5594175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5594576Z layer_outputs = layer_module( 2025-08-26T20:42:06.5594959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5595369Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5595772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5596340Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5596761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5597188Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5597607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.5598055Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.5598235Z 2025-08-26T20:42:06.5598358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5598747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5599107Z return mod(**inputs) 2025-08-26T20:42:06.5599540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5599962Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5600416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5600826Z layer_outputs = layer_module( 2025-08-26T20:42:06.5601235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5601604Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5601981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5602353Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5602748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5603157Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5603563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.5603957Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.5604110Z 2025-08-26T20:42:06.5604200Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5604463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5604852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5605235Z return mod(**inputs) 2025-08-26T20:42:06.5605603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5606007Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5606401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5606798Z layer_outputs = layer_module( 2025-08-26T20:42:06.5607188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5607577Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5607978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5608400Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5608815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.5609228Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5609645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5610044Z return self.weight * hidden_states 2025-08-26T20:42:06.5610188Z 2025-08-26T20:42:06.5610305Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5610692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5611044Z return mod(**inputs) 2025-08-26T20:42:06.5611438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5611832Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5612224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5612613Z layer_outputs = layer_module( 2025-08-26T20:42:06.5612984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5613370Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5613770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5614187Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5614599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5615063Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5615497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.5615913Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.5616061Z 2025-08-26T20:42:06.5616177Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5616556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5616908Z return mod(**inputs) 2025-08-26T20:42:06.5617276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5617675Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5618058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5618457Z layer_outputs = layer_module( 2025-08-26T20:42:06.5618830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5619221Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5619623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5620050Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5620463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5620912Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5621358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.5621768Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.5621933Z 2025-08-26T20:42:06.5622046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5622431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5622783Z return mod(**inputs) 2025-08-26T20:42:06.5623154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5623544Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5623939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5624335Z layer_outputs = layer_module( 2025-08-26T20:42:06.5624708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5625096Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5625491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5625908Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5626321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5626763Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5627191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.5627597Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.5627747Z 2025-08-26T20:42:06.5627834Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5628090Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5628480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5628822Z return mod(**inputs) 2025-08-26T20:42:06.5629193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5629593Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5630009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5630396Z layer_outputs = layer_module( 2025-08-26T20:42:06.5630783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5631172Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5631576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5631990Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5632361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.5632772Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5633181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5633579Z return self.weight * hidden_states 2025-08-26T20:42:06.5633724Z 2025-08-26T20:42:06.5633841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5634222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5634611Z return mod(**inputs) 2025-08-26T20:42:06.5634989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5635391Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5635783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5636186Z layer_outputs = layer_module( 2025-08-26T20:42:06.5636565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5636975Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5637382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5637790Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5638198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5638615Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5639034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5639543Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5639700Z 2025-08-26T20:42:06.5639818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5640224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5640586Z return mod(**inputs) 2025-08-26T20:42:06.5640974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5641386Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5641782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5642180Z layer_outputs = layer_module( 2025-08-26T20:42:06.5642554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5642948Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5643335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5643737Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5644138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5644544Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5644963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5645383Z key_states = self.k(current_states) 2025-08-26T20:42:06.5645546Z 2025-08-26T20:42:06.5645660Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5646050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5646398Z return mod(**inputs) 2025-08-26T20:42:06.5646754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5647152Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5647540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5647938Z layer_outputs = layer_module( 2025-08-26T20:42:06.5648310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5648693Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5649103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5649534Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5649932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5650337Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5650743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5651197Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5651391Z 2025-08-26T20:42:06.5651530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5651918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5652278Z return mod(**inputs) 2025-08-26T20:42:06.5652661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5653065Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5653460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5653867Z layer_outputs = layer_module( 2025-08-26T20:42:06.5670297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5670746Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5671171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5671608Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5672027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5672435Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5672851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5673341Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5673571Z 2025-08-26T20:42:06.5673699Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5674106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5674467Z return mod(**inputs) 2025-08-26T20:42:06.5674860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5675279Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5675781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5676196Z layer_outputs = layer_module( 2025-08-26T20:42:06.5676639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5677053Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5677473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5677891Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5678298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5678722Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5679134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5679730Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5679970Z 2025-08-26T20:42:06.5680098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5680499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5680905Z return mod(**inputs) 2025-08-26T20:42:06.5681300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5681701Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5682089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5682486Z layer_outputs = layer_module( 2025-08-26T20:42:06.5682857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5683299Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5683700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5684101Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5684507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5684913Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5685313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5685786Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5686015Z 2025-08-26T20:42:06.5686129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5686526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5686882Z return mod(**inputs) 2025-08-26T20:42:06.5687259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5687655Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5688053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5688449Z layer_outputs = layer_module( 2025-08-26T20:42:06.5688824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5689214Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5689606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5690011Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5690411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5690815Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5691241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.5691652Z value_states = self.v(current_states) 2025-08-26T20:42:06.5691804Z 2025-08-26T20:42:06.5691938Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5692330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5692689Z return mod(**inputs) 2025-08-26T20:42:06.5693059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5693466Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5693872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5694277Z layer_outputs = layer_module( 2025-08-26T20:42:06.5694644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5695034Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5695477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5695887Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5696458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5696872Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5697287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5697732Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5697909Z 2025-08-26T20:42:06.5698032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5698486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5698842Z return mod(**inputs) 2025-08-26T20:42:06.5699222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5699615Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5699989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5700357Z layer_outputs = layer_module( 2025-08-26T20:42:06.5700709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5701079Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5701457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5701851Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5702246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5702659Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5703062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5703497Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5703670Z 2025-08-26T20:42:06.5703786Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5704166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5704517Z return mod(**inputs) 2025-08-26T20:42:06.5704895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5705271Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5705636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5706042Z layer_outputs = layer_module( 2025-08-26T20:42:06.5706395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5706800Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5707184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5707559Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5707939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5708320Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5708699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.5709106Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.5709269Z 2025-08-26T20:42:06.5709376Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5709749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5710093Z return mod(**inputs) 2025-08-26T20:42:06.5710469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5710869Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5711272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5711669Z layer_outputs = layer_module( 2025-08-26T20:42:06.5712039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5712436Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5712858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5713279Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5713692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5714112Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5714525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.5714941Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.5715093Z 2025-08-26T20:42:06.5715209Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5715614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5715979Z return mod(**inputs) 2025-08-26T20:42:06.5716357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5716782Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5717206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5717628Z layer_outputs = layer_module( 2025-08-26T20:42:06.5718019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5718430Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5718856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5719300Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5719801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.5720237Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5720681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5721117Z return self.weight * hidden_states 2025-08-26T20:42:06.5721270Z 2025-08-26T20:42:06.5721394Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5721814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5722160Z return mod(**inputs) 2025-08-26T20:42:06.5722535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5722952Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5723355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5723761Z layer_outputs = layer_module( 2025-08-26T20:42:06.5724135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5724525Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5724921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5725334Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5725741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5726206Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5726656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.5727033Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.5727170Z 2025-08-26T20:42:06.5727283Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5727641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5727996Z return mod(**inputs) 2025-08-26T20:42:06.5728366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5728763Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5729126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5729525Z layer_outputs = layer_module( 2025-08-26T20:42:06.5729908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5730287Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5730687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5731103Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5731488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5731907Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5732313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.5732691Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.5732831Z 2025-08-26T20:42:06.5732944Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5733311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5733635Z return mod(**inputs) 2025-08-26T20:42:06.5733984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5734359Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5734729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5735105Z layer_outputs = layer_module( 2025-08-26T20:42:06.5735463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5735829Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5736240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5736658Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5737058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5737499Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5737931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.5738328Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.5738474Z 2025-08-26T20:42:06.5738569Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5738821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5739212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5739560Z return mod(**inputs) 2025-08-26T20:42:06.5739933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5740355Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5740734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5741133Z layer_outputs = layer_module( 2025-08-26T20:42:06.5741486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5741857Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5742227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5742650Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5743052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.5743485Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5743915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5744317Z return self.weight * hidden_states 2025-08-26T20:42:06.5744460Z 2025-08-26T20:42:06.5744565Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5744933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5745262Z return mod(**inputs) 2025-08-26T20:42:06.5745621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5746019Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5746413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5746808Z layer_outputs = layer_module( 2025-08-26T20:42:06.5747180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5747562Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5747965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5748379Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5748787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5749202Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5749599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5750004Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5750182Z 2025-08-26T20:42:06.5750298Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5750708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5751056Z return mod(**inputs) 2025-08-26T20:42:06.5751428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5751829Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5752219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5752621Z layer_outputs = layer_module( 2025-08-26T20:42:06.5752986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5753373Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5753771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5754171Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5754573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5755005Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5755414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5755822Z key_states = self.k(current_states) 2025-08-26T20:42:06.5755964Z 2025-08-26T20:42:06.5756080Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5756459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5756824Z return mod(**inputs) 2025-08-26T20:42:06.5757252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5757677Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5758092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5758505Z layer_outputs = layer_module( 2025-08-26T20:42:06.5758892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5759305Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5759806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5760227Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5760653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5761126Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5761559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5762035Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5762235Z 2025-08-26T20:42:06.5762352Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5762752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5763119Z return mod(**inputs) 2025-08-26T20:42:06.5763510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5763929Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5764337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5764748Z layer_outputs = layer_module( 2025-08-26T20:42:06.5765135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5765554Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5765955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5766387Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5766800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5767215Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5767622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5768095Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5768316Z 2025-08-26T20:42:06.5768419Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5768808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5769160Z return mod(**inputs) 2025-08-26T20:42:06.5769536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5769961Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5770372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5770765Z layer_outputs = layer_module( 2025-08-26T20:42:06.5771136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5771508Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5771900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5772322Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5772724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5773137Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5773509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5773963Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5774180Z 2025-08-26T20:42:06.5774284Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5774667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5775016Z return mod(**inputs) 2025-08-26T20:42:06.5775382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5775781Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5776171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5776571Z layer_outputs = layer_module( 2025-08-26T20:42:06.5776937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5777332Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5777726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5778128Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5778528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5778926Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5779326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5779806Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5780028Z 2025-08-26T20:42:06.5780167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5780556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5780917Z return mod(**inputs) 2025-08-26T20:42:06.5781292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5781686Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5782075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5782459Z layer_outputs = layer_module( 2025-08-26T20:42:06.5782830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5783217Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5783612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5784010Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5784411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5784855Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5785252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.5785651Z value_states = self.v(current_states) 2025-08-26T20:42:06.5785798Z 2025-08-26T20:42:06.5785910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5786293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5786643Z return mod(**inputs) 2025-08-26T20:42:06.5787031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5787439Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5787827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5788223Z layer_outputs = layer_module( 2025-08-26T20:42:06.5788597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5788986Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5789395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5789796Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5790195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5790595Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5790993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5791425Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5791604Z 2025-08-26T20:42:06.5791715Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5792098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5792444Z return mod(**inputs) 2025-08-26T20:42:06.5792810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5793203Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5793593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5793981Z layer_outputs = layer_module( 2025-08-26T20:42:06.5794352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5794732Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5795146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5795573Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5795985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5796536Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5796944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5797389Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5797570Z 2025-08-26T20:42:06.5797682Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5798080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5798442Z return mod(**inputs) 2025-08-26T20:42:06.5798829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5799240Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5799714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5800210Z layer_outputs = layer_module( 2025-08-26T20:42:06.5800587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5801000Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5801419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5801844Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5802291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5802705Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5803121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.5803573Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.5803751Z 2025-08-26T20:42:06.5803872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5804264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5804612Z return mod(**inputs) 2025-08-26T20:42:06.5805026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5805444Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5805847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5806244Z layer_outputs = layer_module( 2025-08-26T20:42:06.5806623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5807022Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5807431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5807845Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5808243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5808652Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5809062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.5809447Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.5809581Z 2025-08-26T20:42:06.5809663Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5809905Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5810292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5810634Z return mod(**inputs) 2025-08-26T20:42:06.5811006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5811368Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5811727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5812105Z layer_outputs = layer_module( 2025-08-26T20:42:06.5812465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5812847Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5813241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5813658Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5814064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.5814465Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5814872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5815253Z return self.weight * hidden_states 2025-08-26T20:42:06.5815396Z 2025-08-26T20:42:06.5815501Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5815868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5816187Z return mod(**inputs) 2025-08-26T20:42:06.5816518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5816900Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5817267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5817639Z layer_outputs = layer_module( 2025-08-26T20:42:06.5817972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5818327Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5818698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5819087Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5819470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5819879Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5820289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.5820669Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.5820806Z 2025-08-26T20:42:06.5820918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5821280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5821602Z return mod(**inputs) 2025-08-26T20:42:06.5821948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5822323Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5822706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5823090Z layer_outputs = layer_module( 2025-08-26T20:42:06.5823461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5823847Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5824260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5824675Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5825098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5825521Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5825954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.5826362Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.5826511Z 2025-08-26T20:42:06.5826628Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5827011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5827361Z return mod(**inputs) 2025-08-26T20:42:06.5827731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5828128Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5828515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5828925Z layer_outputs = layer_module( 2025-08-26T20:42:06.5829299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5829687Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5830085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5830503Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5830913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5831380Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5831826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.5832229Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.5832389Z 2025-08-26T20:42:06.5832480Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5832745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5833141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5833499Z return mod(**inputs) 2025-08-26T20:42:06.5833872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5834283Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5834680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5835083Z layer_outputs = layer_module( 2025-08-26T20:42:06.5835457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5835855Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5836263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5836675Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5837081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.5837522Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5837965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5838385Z return self.weight * hidden_states 2025-08-26T20:42:06.5838537Z 2025-08-26T20:42:06.5838663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5839087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5839511Z return mod(**inputs) 2025-08-26T20:42:06.5839931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5840354Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5840760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5841204Z layer_outputs = layer_module( 2025-08-26T20:42:06.5841586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5841980Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5842392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5842806Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5843206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5843620Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5844023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5844449Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5844584Z 2025-08-26T20:42:06.5844696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5845065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5845412Z return mod(**inputs) 2025-08-26T20:42:06.5845776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5846170Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5846536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5846919Z layer_outputs = layer_module( 2025-08-26T20:42:06.5847290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5847680Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5848077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5848471Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5848870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5849272Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5849662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5850033Z key_states = self.k(current_states) 2025-08-26T20:42:06.5850177Z 2025-08-26T20:42:06.5850283Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5850651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5850985Z return mod(**inputs) 2025-08-26T20:42:06.5851332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5851698Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5852065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5852438Z layer_outputs = layer_module( 2025-08-26T20:42:06.5852788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5853151Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5853552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5853933Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5854324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5854708Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5855073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5855500Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5855692Z 2025-08-26T20:42:06.5855798Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5856160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5856487Z return mod(**inputs) 2025-08-26T20:42:06.5856827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5857207Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5857577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5857969Z layer_outputs = layer_module( 2025-08-26T20:42:06.5858315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5858702Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5859107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5859486Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5859863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5860263Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5860654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5861115Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5861331Z 2025-08-26T20:42:06.5861451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5861825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5862153Z return mod(**inputs) 2025-08-26T20:42:06.5862510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5862891Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5863269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5863640Z layer_outputs = layer_module( 2025-08-26T20:42:06.5864001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5864376Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5864759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5865147Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5865546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5865957Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5866365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5866848Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5867073Z 2025-08-26T20:42:06.5867198Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5867613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5867962Z return mod(**inputs) 2025-08-26T20:42:06.5868349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5868746Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5869128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5869535Z layer_outputs = layer_module( 2025-08-26T20:42:06.5869906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5870292Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5870686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5871090Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5871491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5871892Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5872292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5872791Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5873014Z 2025-08-26T20:42:06.5873125Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5873521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5873869Z return mod(**inputs) 2025-08-26T20:42:06.5874242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5874654Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5875057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5875465Z layer_outputs = layer_module( 2025-08-26T20:42:06.5875852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5876250Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5876654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5877067Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5877480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5877900Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5878309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.5878712Z value_states = self.v(current_states) 2025-08-26T20:42:06.5878869Z 2025-08-26T20:42:06.5878984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5879481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5879874Z return mod(**inputs) 2025-08-26T20:42:06.5880255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5880688Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5881093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5881494Z layer_outputs = layer_module( 2025-08-26T20:42:06.5881867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5882252Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5882662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5883105Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5883510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5883884Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5884265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5884387Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5884391Z 2025-08-26T20:42:06.5884498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5884708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5884777Z return mod(**inputs) 2025-08-26T20:42:06.5885016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5885101Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5885337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5885419Z layer_outputs = layer_module( 2025-08-26T20:42:06.5885663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5885745Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5885987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5886070Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5886311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5886414Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5886651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5886770Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5886774Z 2025-08-26T20:42:06.5886883Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5887100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5887167Z return mod(**inputs) 2025-08-26T20:42:06.5887407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5887481Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5887711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5887790Z layer_outputs = layer_module( 2025-08-26T20:42:06.5888007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5888093Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5888320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5888400Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5888638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5888723Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5888963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.5889072Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.5889075Z 2025-08-26T20:42:06.5889187Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5889439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5889518Z return mod(**inputs) 2025-08-26T20:42:06.5889805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5889885Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5890161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5890241Z layer_outputs = layer_module( 2025-08-26T20:42:06.5890480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5890573Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5890819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5890906Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5891145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5891230Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5891471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.5891553Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.5891572Z 2025-08-26T20:42:06.5891687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5891890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5891964Z return mod(**inputs) 2025-08-26T20:42:06.5892203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5892279Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5892522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5892623Z layer_outputs = layer_module( 2025-08-26T20:42:06.5892860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5892942Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5893181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5893279Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5893533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-26T20:42:06.5893682Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:42:06.5893686Z 2025-08-26T20:42:06.5893773Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5893881Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5894101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5894171Z return mod(**inputs) 2025-08-26T20:42:06.5894435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5894516Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5894781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5894858Z layer_outputs = layer_module( 2025-08-26T20:42:06.5895097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5895189Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5895443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5895550Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5895810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.5895926Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5896281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5896420Z return self.weight * hidden_states 2025-08-26T20:42:06.5896426Z 2025-08-26T20:42:06.5896546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5896762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5896834Z return mod(**inputs) 2025-08-26T20:42:06.5897094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5897174Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5897432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5897512Z layer_outputs = layer_module( 2025-08-26T20:42:06.5897755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5897843Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5898094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5898236Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5898473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5898601Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5898836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.5898918Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.5898943Z 2025-08-26T20:42:06.5899061Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5899260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5899334Z return mod(**inputs) 2025-08-26T20:42:06.5899572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5899647Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5899893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5899966Z layer_outputs = layer_module( 2025-08-26T20:42:06.5900196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5900276Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5900517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5900613Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5900849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5900979Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5901216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.5901304Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.5901307Z 2025-08-26T20:42:06.5901411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5901610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5901684Z return mod(**inputs) 2025-08-26T20:42:06.5901922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5902008Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5902284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5902365Z layer_outputs = layer_module( 2025-08-26T20:42:06.5902608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5902690Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5902934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5903023Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5903263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5903381Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5903615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.5903706Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.5903709Z 2025-08-26T20:42:06.5903792Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5903903Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5904101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5904184Z return mod(**inputs) 2025-08-26T20:42:06.5904433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5904508Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5904757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5904830Z layer_outputs = layer_module( 2025-08-26T20:42:06.5905076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5905158Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5905395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5905490Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5905726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.5905841Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5906075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5906153Z return self.weight * hidden_states 2025-08-26T20:42:06.5906156Z 2025-08-26T20:42:06.5906266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5906466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5906543Z return mod(**inputs) 2025-08-26T20:42:06.5906782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5906856Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5907102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5907175Z layer_outputs = layer_module( 2025-08-26T20:42:06.5907405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5907490Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5907748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5907834Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5908084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5908198Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5908448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5908562Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5908567Z 2025-08-26T20:42:06.5908676Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5908888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5908965Z return mod(**inputs) 2025-08-26T20:42:06.5909214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5909300Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5909551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5909624Z layer_outputs = layer_module( 2025-08-26T20:42:06.5909855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5909935Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5910181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5910280Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5910519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5910602Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5910837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5910922Z key_states = self.k(current_states) 2025-08-26T20:42:06.5910941Z 2025-08-26T20:42:06.5911046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5911253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5911321Z return mod(**inputs) 2025-08-26T20:42:06.5911560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5911641Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5911881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5911963Z layer_outputs = layer_module( 2025-08-26T20:42:06.5912201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5912285Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5912539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5912628Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5912887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5912974Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5913230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5913374Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5913378Z 2025-08-26T20:42:06.5913486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5913707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5913777Z return mod(**inputs) 2025-08-26T20:42:06.5914036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5914115Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5914383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5914469Z layer_outputs = layer_module( 2025-08-26T20:42:06.5914724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5914820Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5915070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5915165Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5915415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5915501Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5915762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5915932Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5915938Z 2025-08-26T20:42:06.5916055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5916269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5916368Z return mod(**inputs) 2025-08-26T20:42:06.5916631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5916710Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5916970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5917046Z layer_outputs = layer_module( 2025-08-26T20:42:06.5917288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5917398Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5917649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5917742Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5917995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5918088Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5918339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5918502Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5918506Z 2025-08-26T20:42:06.5918625Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5918842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5918924Z return mod(**inputs) 2025-08-26T20:42:06.5919188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5919269Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5919603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5919690Z layer_outputs = layer_module( 2025-08-26T20:42:06.5919954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5920039Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5920310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5920398Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5920658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5920757Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5921032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5921205Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5921224Z 2025-08-26T20:42:06.5921337Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5921552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5921630Z return mod(**inputs) 2025-08-26T20:42:06.5921892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5921979Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5922243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5922323Z layer_outputs = layer_module( 2025-08-26T20:42:06.5922569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5922654Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5922918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5923024Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5923294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5923382Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5923640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.5923734Z value_states = self.v(current_states) 2025-08-26T20:42:06.5923738Z 2025-08-26T20:42:06.5923848Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5924093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5924165Z return mod(**inputs) 2025-08-26T20:42:06.5924417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5924504Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5924757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5924839Z layer_outputs = layer_module( 2025-08-26T20:42:06.5925074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5925165Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5925424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5925511Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5925772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5925858Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5926123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5926242Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5926245Z 2025-08-26T20:42:06.5926355Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5926574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5926645Z return mod(**inputs) 2025-08-26T20:42:06.5926911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5926990Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5927243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5927349Z layer_outputs = layer_module( 2025-08-26T20:42:06.5927585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5927691Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5927943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5928036Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5928297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5928384Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5928644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5928763Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5928767Z 2025-08-26T20:42:06.5928885Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5929097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5929168Z return mod(**inputs) 2025-08-26T20:42:06.5929437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5929529Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5929778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5929850Z layer_outputs = layer_module( 2025-08-26T20:42:06.5930076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5930164Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5930417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5930506Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5930737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5930828Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5931063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.5931173Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.5931177Z 2025-08-26T20:42:06.5931289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5931486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5931559Z return mod(**inputs) 2025-08-26T20:42:06.5931795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5931870Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5932113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5932185Z layer_outputs = layer_module( 2025-08-26T20:42:06.5932416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5932497Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5932747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5932829Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5933060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5933148Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5933382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.5933489Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.5933493Z 2025-08-26T20:42:06.5933576Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5933696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5933916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5933985Z return mod(**inputs) 2025-08-26T20:42:06.5934244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5934321Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5934572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5934657Z layer_outputs = layer_module( 2025-08-26T20:42:06.5934895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5934989Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5935238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5935344Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5935612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.5935719Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5935973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5936055Z return self.weight * hidden_states 2025-08-26T20:42:06.5936059Z 2025-08-26T20:42:06.5936174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5936477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5936544Z return mod(**inputs) 2025-08-26T20:42:06.5936793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5936867Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5937116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5937191Z layer_outputs = layer_module( 2025-08-26T20:42:06.5937414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5937500Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5937747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5937849Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5938098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5938230Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5938481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.5938566Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.5938571Z 2025-08-26T20:42:06.5938686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5938893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5938970Z return mod(**inputs) 2025-08-26T20:42:06.5939218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5939295Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5939553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5939630Z layer_outputs = layer_module( 2025-08-26T20:42:06.5939890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5939975Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5940285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5940385Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5940620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5940745Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5940978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.5941071Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.5941075Z 2025-08-26T20:42:06.5941178Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5941377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5941451Z return mod(**inputs) 2025-08-26T20:42:06.5941690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5941793Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5942033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5942106Z layer_outputs = layer_module( 2025-08-26T20:42:06.5942335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5942415Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5942676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5942768Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5943017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5943141Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5943396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.5943489Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.5943493Z 2025-08-26T20:42:06.5943579Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5943696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5943909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5943979Z return mod(**inputs) 2025-08-26T20:42:06.5944242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5944322Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5944582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5944660Z layer_outputs = layer_module( 2025-08-26T20:42:06.5944900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5944992Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5945253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5945348Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5945604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.5945727Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5946003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5946089Z return self.weight * hidden_states 2025-08-26T20:42:06.5946093Z 2025-08-26T20:42:06.5946213Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5946438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5946517Z return mod(**inputs) 2025-08-26T20:42:06.5946757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5946835Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5947102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5947178Z layer_outputs = layer_module( 2025-08-26T20:42:06.5947419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5947504Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5947752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5947848Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5948132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5948226Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5948482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.5948573Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.5948577Z 2025-08-26T20:42:06.5948687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5948901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5949000Z return mod(**inputs) 2025-08-26T20:42:06.5949270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5949358Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5949628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5949706Z layer_outputs = layer_module( 2025-08-26T20:42:06.5949960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5950047Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5950315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5950403Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5950665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5950761Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5951030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5951121Z key_states = self.k(current_states) 2025-08-26T20:42:06.5951126Z 2025-08-26T20:42:06.5951236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5951456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5951525Z return mod(**inputs) 2025-08-26T20:42:06.5951777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5951862Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5952114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5952199Z layer_outputs = layer_module( 2025-08-26T20:42:06.5952451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5952536Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5952810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5952901Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5953156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5953244Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5953491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5953639Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5953645Z 2025-08-26T20:42:06.5953758Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5953986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5954060Z return mod(**inputs) 2025-08-26T20:42:06.5954328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5954424Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5954682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5954767Z layer_outputs = layer_module( 2025-08-26T20:42:06.5955007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5955099Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5955357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5955462Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5955726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5955815Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5956076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5956247Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5956251Z 2025-08-26T20:42:06.5956369Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5956586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5956657Z return mod(**inputs) 2025-08-26T20:42:06.5956920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5957000Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5957266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5957345Z layer_outputs = layer_module( 2025-08-26T20:42:06.5957589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5957685Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5957940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5958033Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5958288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5958375Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5958641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5958829Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5958834Z 2025-08-26T20:42:06.5958956Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5959191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5959277Z return mod(**inputs) 2025-08-26T20:42:06.5959616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5959702Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5959975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5960053Z layer_outputs = layer_module( 2025-08-26T20:42:06.5960310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5960401Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5960663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5960763Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5961039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5961156Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5961413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.5961580Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.5961584Z 2025-08-26T20:42:06.5961693Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5961906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5962005Z return mod(**inputs) 2025-08-26T20:42:06.5962272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5962359Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5962623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5962699Z layer_outputs = layer_module( 2025-08-26T20:42:06.5962946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5963029Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5963290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5963376Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5963633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5963729Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5963984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.5964076Z value_states = self.v(current_states) 2025-08-26T20:42:06.5964080Z 2025-08-26T20:42:06.5964190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5964411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5964482Z return mod(**inputs) 2025-08-26T20:42:06.5964741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5964826Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5965088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5965173Z layer_outputs = layer_module( 2025-08-26T20:42:06.5965436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5965520Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5965798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5965886Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5966142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5966228Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5966482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5966608Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5966612Z 2025-08-26T20:42:06.5966722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5966941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5967013Z return mod(**inputs) 2025-08-26T20:42:06.5967267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5967346Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5967637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5967721Z layer_outputs = layer_module( 2025-08-26T20:42:06.5967956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5968048Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5968308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5968412Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5968677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5968763Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5969021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.5969141Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.5969145Z 2025-08-26T20:42:06.5969262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5969476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5969549Z return mod(**inputs) 2025-08-26T20:42:06.5969817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5969895Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5970186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5970263Z layer_outputs = layer_module( 2025-08-26T20:42:06.5970497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5970587Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5970840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5970931Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5971177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5971263Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5971520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.5971638Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.5971642Z 2025-08-26T20:42:06.5971774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5971989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5972064Z return mod(**inputs) 2025-08-26T20:42:06.5972332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5972412Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5972672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5972747Z layer_outputs = layer_module( 2025-08-26T20:42:06.5972993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5973077Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5973329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5973425Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5973674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.5973767Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.5975019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.5975104Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.5975115Z 2025-08-26T20:42:06.5975227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5975440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5975519Z return mod(**inputs) 2025-08-26T20:42:06.5975769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5975875Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5976128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5976205Z layer_outputs = layer_module( 2025-08-26T20:42:06.5976448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5976534Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5976790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.5976876Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.5977121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-26T20:42:06.5977270Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:42:06.5977276Z 2025-08-26T20:42:06.5977361Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5977479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5977690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5977768Z return mod(**inputs) 2025-08-26T20:42:06.5978022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5978101Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5978363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5978441Z layer_outputs = layer_module( 2025-08-26T20:42:06.5978692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5978778Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5979035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5979163Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5979420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.5979557Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.5979823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5979907Z return self.weight * hidden_states 2025-08-26T20:42:06.5979918Z 2025-08-26T20:42:06.5980026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5980236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5980314Z return mod(**inputs) 2025-08-26T20:42:06.5980564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5980650Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5980905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5980983Z layer_outputs = layer_module( 2025-08-26T20:42:06.5981232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5981338Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5981605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5981704Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5981961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5982099Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5982379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.5982476Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.5982480Z 2025-08-26T20:42:06.5982591Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5982818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5982892Z return mod(**inputs) 2025-08-26T20:42:06.5983154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5983242Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5983502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5983587Z layer_outputs = layer_module( 2025-08-26T20:42:06.5983831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5983917Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5984185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5984285Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5984550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5984678Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5984933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.5985027Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.5985031Z 2025-08-26T20:42:06.5985143Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5985369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5985440Z return mod(**inputs) 2025-08-26T20:42:06.5985721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5985806Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5986082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5986171Z layer_outputs = layer_module( 2025-08-26T20:42:06.5986415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5986507Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5986767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.5986865Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.5987133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.5987262Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.5987526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.5987612Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.5987633Z 2025-08-26T20:42:06.5987722Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.5987841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5988056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5988136Z return mod(**inputs) 2025-08-26T20:42:06.5988395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-26T20:42:06.5988482Z encoder_outputs = self.encoder( 2025-08-26T20:42:06.5988767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward 2025-08-26T20:42:06.5988884Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-26T20:42:06.5989154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.5989240Z return self.weight * hidden_states 2025-08-26T20:42:06.5989245Z 2025-08-26T20:42:06.5989365Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5989581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5989654Z return mod(**inputs) 2025-08-26T20:42:06.5989919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5989999Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5990263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5990345Z layer_outputs = layer_module( 2025-08-26T20:42:06.5990587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5990683Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5990943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.5991042Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.5991299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.5991401Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.5991655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.5991740Z key_states = self.k(current_states) 2025-08-26T20:42:06.5991746Z 2025-08-26T20:42:06.5991866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5992098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5992181Z return mod(**inputs) 2025-08-26T20:42:06.5992462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5992546Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5992817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5992897Z layer_outputs = layer_module( 2025-08-26T20:42:06.5993149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5993236Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5993497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.5993593Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.5993864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.5993963Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.5994217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.5997840Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.5997846Z 2025-08-26T20:42:06.5997976Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.5998199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.5998282Z return mod(**inputs) 2025-08-26T20:42:06.5998549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.5998685Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.5998960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.5999039Z layer_outputs = layer_module( 2025-08-26T20:42:06.5999296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.5999382Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.5999751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.5999856Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6000119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6000211Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6000480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6000654Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6000658Z 2025-08-26T20:42:06.6000781Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6001002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6001083Z return mod(**inputs) 2025-08-26T20:42:06.6001350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6001433Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6001715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6001792Z layer_outputs = layer_module( 2025-08-26T20:42:06.6002041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6002126Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6002414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6002508Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6002779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6002875Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6003128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6003220Z value_states = self.v(current_states) 2025-08-26T20:42:06.6003223Z 2025-08-26T20:42:06.6003335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6003548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6003628Z return mod(**inputs) 2025-08-26T20:42:06.6003887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6003973Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6004230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6004307Z layer_outputs = layer_module( 2025-08-26T20:42:06.6004554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6004715Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6004973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6005059Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6005311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6005425Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6005674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6005800Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6005804Z 2025-08-26T20:42:06.6005915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6006133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6006205Z return mod(**inputs) 2025-08-26T20:42:06.6006457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6006545Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6006797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6006883Z layer_outputs = layer_module( 2025-08-26T20:42:06.6007120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6007205Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6007466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6007551Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6007807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6007899Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6008154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6008271Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6008274Z 2025-08-26T20:42:06.6008386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6008609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6008692Z return mod(**inputs) 2025-08-26T20:42:06.6008938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6009027Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6009267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6009349Z layer_outputs = layer_module( 2025-08-26T20:42:06.6009574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6009659Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6009897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6009978Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6010223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6010307Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6010550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6010659Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6010692Z 2025-08-26T20:42:06.6010805Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6011004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6011071Z return mod(**inputs) 2025-08-26T20:42:06.6011317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6011392Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6011654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6011729Z layer_outputs = layer_module( 2025-08-26T20:42:06.6011954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6012040Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6012277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6012367Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6012603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6012686Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6012929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6013010Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6013014Z 2025-08-26T20:42:06.6013104Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6013208Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6013417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6013488Z return mod(**inputs) 2025-08-26T20:42:06.6013740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6013827Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6014078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6014158Z layer_outputs = layer_module( 2025-08-26T20:42:06.6014392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6014475Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6014755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6014850Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6015114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.6015214Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6015449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6015534Z return self.weight * hidden_states 2025-08-26T20:42:06.6015538Z 2025-08-26T20:42:06.6015639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6015845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6015910Z return mod(**inputs) 2025-08-26T20:42:06.6016152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6016228Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6016462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6016540Z layer_outputs = layer_module( 2025-08-26T20:42:06.6016762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6016866Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6017103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6017194Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6017439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6017576Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6017819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.6017902Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.6017906Z 2025-08-26T20:42:06.6018016Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6018215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6018284Z return mod(**inputs) 2025-08-26T20:42:06.6018530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6018604Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6018846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6018918Z layer_outputs = layer_module( 2025-08-26T20:42:06.6019152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6019247Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6019496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6019601Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6019851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6019977Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6020234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.6020320Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.6020324Z 2025-08-26T20:42:06.6020441Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6020657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6020750Z return mod(**inputs) 2025-08-26T20:42:06.6021009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6021098Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6021347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6021422Z layer_outputs = layer_module( 2025-08-26T20:42:06.6021655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6021733Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6021970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6022072Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6022307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6022433Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6022668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.6022748Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.6022759Z 2025-08-26T20:42:06.6022881Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6023080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6023153Z return mod(**inputs) 2025-08-26T20:42:06.6023403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6023487Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6023754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6023830Z layer_outputs = layer_module( 2025-08-26T20:42:06.6024074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6024160Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6024419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6024509Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6024756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.6024877Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6025125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6025219Z return self.weight * hidden_states 2025-08-26T20:42:06.6025223Z 2025-08-26T20:42:06.6025333Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6025551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6025621Z return mod(**inputs) 2025-08-26T20:42:06.6025875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6025960Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6026209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6026294Z layer_outputs = layer_module( 2025-08-26T20:42:06.6026531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6026615Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6026871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6026975Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6027232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6027337Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6027585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6027679Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6027682Z 2025-08-26T20:42:06.6027792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6028006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6028076Z return mod(**inputs) 2025-08-26T20:42:06.6028333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6028412Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6028664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6028746Z layer_outputs = layer_module( 2025-08-26T20:42:06.6028983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6029071Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6029337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6029423Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6029680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6029768Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6030043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6030127Z key_states = self.k(current_states) 2025-08-26T20:42:06.6030131Z 2025-08-26T20:42:06.6030242Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6030461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6030530Z return mod(**inputs) 2025-08-26T20:42:06.6030788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6030867Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6031123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6031198Z layer_outputs = layer_module( 2025-08-26T20:42:06.6031433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6031526Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6031776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6031869Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6032121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6032207Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6032465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6032603Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6032608Z 2025-08-26T20:42:06.6032724Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6032934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6033013Z return mod(**inputs) 2025-08-26T20:42:06.6033284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6033363Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6033637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6033715Z layer_outputs = layer_module( 2025-08-26T20:42:06.6033961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6034047Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6034298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6034392Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6034640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6034736Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6034989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6035154Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6035169Z 2025-08-26T20:42:06.6035279Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6035490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6035587Z return mod(**inputs) 2025-08-26T20:42:06.6035844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6035929Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6036179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6036273Z layer_outputs = layer_module( 2025-08-26T20:42:06.6036522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6036607Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6036871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6036960Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6037220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6037318Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6037574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6037671Z value_states = self.v(current_states) 2025-08-26T20:42:06.6037676Z 2025-08-26T20:42:06.6037791Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6038015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6038090Z return mod(**inputs) 2025-08-26T20:42:06.6038347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6038435Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6038693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6038780Z layer_outputs = layer_module( 2025-08-26T20:42:06.6039024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6039111Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6039378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6039543Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6039854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6039946Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6040222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6040357Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6040363Z 2025-08-26T20:42:06.6040478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6040715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6040786Z return mod(**inputs) 2025-08-26T20:42:06.6041049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6041127Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6041382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6041469Z layer_outputs = layer_module( 2025-08-26T20:42:06.6041709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6041803Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6042056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6042162Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6042419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6042505Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6042757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6042915Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6042920Z 2025-08-26T20:42:06.6043033Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6043249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6043321Z return mod(**inputs) 2025-08-26T20:42:06.6043579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6043665Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6043907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6043980Z layer_outputs = layer_module( 2025-08-26T20:42:06.6044204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6044290Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6044529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6044618Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6044852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6044933Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6045176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6045285Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6045290Z 2025-08-26T20:42:06.6045401Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6045605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6045677Z return mod(**inputs) 2025-08-26T20:42:06.6045940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6046018Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6046297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6046391Z layer_outputs = layer_module( 2025-08-26T20:42:06.6046630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6046715Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6046968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6047058Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6047314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6047410Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6047664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6047744Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6047748Z 2025-08-26T20:42:06.6047839Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6047944Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6048154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6048240Z return mod(**inputs) 2025-08-26T20:42:06.6048480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6048561Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6048799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6048879Z layer_outputs = layer_module( 2025-08-26T20:42:06.6049120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6049210Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6049452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6049535Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6049778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-26T20:42:06.6049891Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6050137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6050214Z return self.weight * hidden_states 2025-08-26T20:42:06.6050218Z 2025-08-26T20:42:06.6050322Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6050532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6050598Z return mod(**inputs) 2025-08-26T20:42:06.6050844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6050918Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6051171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6051251Z layer_outputs = layer_module( 2025-08-26T20:42:06.6051470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6051554Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6051792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6051882Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6052117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6052219Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6052462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6052557Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6052561Z 2025-08-26T20:42:06.6052672Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6052874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6052940Z return mod(**inputs) 2025-08-26T20:42:06.6053187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6053260Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6053505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6053579Z layer_outputs = layer_module( 2025-08-26T20:42:06.6053809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6053888Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6054141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6054253Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6054507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6054605Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6054855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6054936Z key_states = self.k(current_states) 2025-08-26T20:42:06.6054957Z 2025-08-26T20:42:06.6055075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6055287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6055362Z return mod(**inputs) 2025-08-26T20:42:06.6055612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6055689Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6055946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6056021Z layer_outputs = layer_module( 2025-08-26T20:42:06.6056265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6056346Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6056601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6056687Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6056949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6057047Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6057304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6057443Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6057447Z 2025-08-26T20:42:06.6057550Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6057753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6057823Z return mod(**inputs) 2025-08-26T20:42:06.6058074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6058162Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6058439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6058526Z layer_outputs = layer_module( 2025-08-26T20:42:06.6058779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6058865Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6059126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6059213Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6059470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6059559Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6059813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6059981Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6059985Z 2025-08-26T20:42:06.6060095Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6060315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6060385Z return mod(**inputs) 2025-08-26T20:42:06.6060645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6060741Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6060990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6061075Z layer_outputs = layer_module( 2025-08-26T20:42:06.6061310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6061427Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6061678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6061763Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6062019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6062107Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6062367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6062451Z value_states = self.v(current_states) 2025-08-26T20:42:06.6062455Z 2025-08-26T20:42:06.6062570Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6062779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6062851Z return mod(**inputs) 2025-08-26T20:42:06.6063108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6063184Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6063444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6063520Z layer_outputs = layer_module( 2025-08-26T20:42:06.6063753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6063845Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6064094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6064185Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6064430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6064519Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6064791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6064907Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6064927Z 2025-08-26T20:42:06.6065044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6065257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6065336Z return mod(**inputs) 2025-08-26T20:42:06.6065598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6065677Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6065961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6066038Z layer_outputs = layer_module( 2025-08-26T20:42:06.6066282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6066367Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6066625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6066718Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6066977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6067094Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6067350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6067469Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6067473Z 2025-08-26T20:42:06.6067600Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6067812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6067891Z return mod(**inputs) 2025-08-26T20:42:06.6068139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6068226Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6068478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6068556Z layer_outputs = layer_module( 2025-08-26T20:42:06.6068800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6068884Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6069146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6069233Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6069492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6069587Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6069848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6069969Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6069975Z 2025-08-26T20:42:06.6070084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6070305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6070375Z return mod(**inputs) 2025-08-26T20:42:06.6070641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6070729Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6070979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6071082Z layer_outputs = layer_module( 2025-08-26T20:42:06.6071323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6071424Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6071694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6071780Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6072041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6072128Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6072385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6072478Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6072482Z 2025-08-26T20:42:06.6072567Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6072686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6072901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6072977Z return mod(**inputs) 2025-08-26T20:42:06.6073235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6073331Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6073592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6073667Z layer_outputs = layer_module( 2025-08-26T20:42:06.6073909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6074009Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6074274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6074385Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6074653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.6074770Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6075037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6075121Z return self.weight * hidden_states 2025-08-26T20:42:06.6075134Z 2025-08-26T20:42:06.6075246Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6075465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6075549Z return mod(**inputs) 2025-08-26T20:42:06.6075808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6075897Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6076155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6076234Z layer_outputs = layer_module( 2025-08-26T20:42:06.6076483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6076568Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6076832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6076931Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6077185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6077325Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6077605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.6077701Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.6077706Z 2025-08-26T20:42:06.6077835Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6078060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6078133Z return mod(**inputs) 2025-08-26T20:42:06.6078393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6078479Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6078736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6078822Z layer_outputs = layer_module( 2025-08-26T20:42:06.6079064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6079150Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6079645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6079761Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6080025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6080180Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6080439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.6080538Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.6080542Z 2025-08-26T20:42:06.6080656Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6080903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6080977Z return mod(**inputs) 2025-08-26T20:42:06.6081246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6081329Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6081589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6081679Z layer_outputs = layer_module( 2025-08-26T20:42:06.6081921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6082014Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6082272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6082393Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6082750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6082879Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6083144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.6083233Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.6083237Z 2025-08-26T20:42:06.6083337Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6083451Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6083670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6083752Z return mod(**inputs) 2025-08-26T20:42:06.6084012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6084104Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6084390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6084472Z layer_outputs = layer_module( 2025-08-26T20:42:06.6084738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6084826Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6085089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6085181Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6085435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.6085561Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6085815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6085909Z return self.weight * hidden_states 2025-08-26T20:42:06.6085912Z 2025-08-26T20:42:06.6086026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6086248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6086321Z return mod(**inputs) 2025-08-26T20:42:06.6086576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6086699Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6086957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6087044Z layer_outputs = layer_module( 2025-08-26T20:42:06.6087286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6087389Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6087658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6087743Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6087996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6088085Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6088330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6088423Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6088427Z 2025-08-26T20:42:06.6088536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6088751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6088820Z return mod(**inputs) 2025-08-26T20:42:06.6089074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6089154Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6089401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6089486Z layer_outputs = layer_module( 2025-08-26T20:42:06.6089721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6089812Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6090061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6090146Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6090401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6090492Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6090761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6090846Z key_states = self.k(current_states) 2025-08-26T20:42:06.6090850Z 2025-08-26T20:42:06.6090960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6091195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6091266Z return mod(**inputs) 2025-08-26T20:42:06.6091528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6091605Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6091866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6091943Z layer_outputs = layer_module( 2025-08-26T20:42:06.6092182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6092272Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6092522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6092615Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6092862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6092967Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6093224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6093362Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6093366Z 2025-08-26T20:42:06.6093482Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6093713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6093783Z return mod(**inputs) 2025-08-26T20:42:06.6094046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6094123Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6094382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6094456Z layer_outputs = layer_module( 2025-08-26T20:42:06.6094701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6094785Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6095035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6095127Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6095376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6095468Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6095717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6095884Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6095895Z 2025-08-26T20:42:06.6096003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6096398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6096486Z return mod(**inputs) 2025-08-26T20:42:06.6096743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6096831Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6097082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6097164Z layer_outputs = layer_module( 2025-08-26T20:42:06.6097456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6097541Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6097822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6097911Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6098160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6098255Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6098508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6098602Z value_states = self.v(current_states) 2025-08-26T20:42:06.6098608Z 2025-08-26T20:42:06.6098721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6098949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6099028Z return mod(**inputs) 2025-08-26T20:42:06.6099280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6099367Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6099659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6099745Z layer_outputs = layer_module( 2025-08-26T20:42:06.6099990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6100077Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6100349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6100462Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6100720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6100807Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6101058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6101183Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6101187Z 2025-08-26T20:42:06.6101295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6101511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6101580Z return mod(**inputs) 2025-08-26T20:42:06.6101831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6101918Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6102169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6102254Z layer_outputs = layer_module( 2025-08-26T20:42:06.6102492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6102586Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6102838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6102924Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6103180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6103268Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6103520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6103638Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6103661Z 2025-08-26T20:42:06.6103770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6104004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6104076Z return mod(**inputs) 2025-08-26T20:42:06.6104334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6104414Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6104672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6104747Z layer_outputs = layer_module( 2025-08-26T20:42:06.6104984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6105077Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6105335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6105427Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6105683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6105770Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6106073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6106189Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6106192Z 2025-08-26T20:42:06.6106308Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6106522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6106613Z return mod(**inputs) 2025-08-26T20:42:06.6106882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6106961Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6107233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6107314Z layer_outputs = layer_module( 2025-08-26T20:42:06.6107563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6107653Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6107920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6108016Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6108280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6108381Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6108644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6108729Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6108733Z 2025-08-26T20:42:06.6108854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6109081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6109159Z return mod(**inputs) 2025-08-26T20:42:06.6109421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6109500Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6109769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6109847Z layer_outputs = layer_module( 2025-08-26T20:42:06.6110090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6110196Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6110474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6110587Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6110852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-26T20:42:06.6111012Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:42:06.6111016Z 2025-08-26T20:42:06.6111104Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6111224Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6111439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6111512Z return mod(**inputs) 2025-08-26T20:42:06.6111788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6111867Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6112138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6112216Z layer_outputs = layer_module( 2025-08-26T20:42:06.6112461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6112573Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6112841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6112939Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6113202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-26T20:42:06.6113350Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6113618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6113704Z return self.weight * hidden_states 2025-08-26T20:42:06.6113708Z 2025-08-26T20:42:06.6113829Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6114045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6114125Z return mod(**inputs) 2025-08-26T20:42:06.6114394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6114474Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6114749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6114829Z layer_outputs = layer_module( 2025-08-26T20:42:06.6115078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6115165Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6115426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6115524Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6115789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6115892Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6116154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6116246Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6116250Z 2025-08-26T20:42:06.6116360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6116578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6116655Z return mod(**inputs) 2025-08-26T20:42:06.6116941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6117045Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6117319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6117399Z layer_outputs = layer_module( 2025-08-26T20:42:06.6117649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6117733Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6117998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6118088Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6118358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6118451Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6118708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6118800Z key_states = self.k(current_states) 2025-08-26T20:42:06.6118804Z 2025-08-26T20:42:06.6118917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6119159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6119231Z return mod(**inputs) 2025-08-26T20:42:06.6119574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6119667Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6119951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6120038Z layer_outputs = layer_module( 2025-08-26T20:42:06.6120284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6120370Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6120639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6120728Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6120990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6121081Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6121342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6121486Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6121490Z 2025-08-26T20:42:06.6121603Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6121827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6121902Z return mod(**inputs) 2025-08-26T20:42:06.6122169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6122248Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6122509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6122599Z layer_outputs = layer_module( 2025-08-26T20:42:06.6122840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6122933Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6123190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6123300Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6123568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6123685Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6123950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6124122Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6124126Z 2025-08-26T20:42:06.6124242Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6124466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6124536Z return mod(**inputs) 2025-08-26T20:42:06.6124794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6124872Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6125130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6125207Z layer_outputs = layer_module( 2025-08-26T20:42:06.6125445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6125534Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6125804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6125897Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6126146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6126241Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6126509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6126594Z value_states = self.v(current_states) 2025-08-26T20:42:06.6126598Z 2025-08-26T20:42:06.6126713Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6126924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6127000Z return mod(**inputs) 2025-08-26T20:42:06.6127253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6127331Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6127587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6127661Z layer_outputs = layer_module( 2025-08-26T20:42:06.6127904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6127988Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6128241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6128333Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6128584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6128681Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6128928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6129053Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6129057Z 2025-08-26T20:42:06.6129165Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6129374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6129455Z return mod(**inputs) 2025-08-26T20:42:06.6129726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6129813Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6130081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6130158Z layer_outputs = layer_module( 2025-08-26T20:42:06.6130407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6130490Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6130747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6130831Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6131078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6131175Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6131424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6131548Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6131552Z 2025-08-26T20:42:06.6131661Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6131876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6131964Z return mod(**inputs) 2025-08-26T20:42:06.6132220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6132306Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6132561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6132665Z layer_outputs = layer_module( 2025-08-26T20:42:06.6132904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6132989Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6133249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6133337Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6133596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6133683Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6133939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6134053Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6134058Z 2025-08-26T20:42:06.6134167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6134387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6134456Z return mod(**inputs) 2025-08-26T20:42:06.6134714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6134793Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6135046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6135127Z layer_outputs = layer_module( 2025-08-26T20:42:06.6135351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6135435Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6135671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6135754Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6136025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6136110Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6136367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6136446Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6136451Z 2025-08-26T20:42:06.6136540Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6136645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6136844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6136918Z return mod(**inputs) 2025-08-26T20:42:06.6137160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6137242Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6137486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6137557Z layer_outputs = layer_module( 2025-08-26T20:42:06.6137792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6137869Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6138130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6138223Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6138458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.6138565Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6138817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6138903Z return self.weight * hidden_states 2025-08-26T20:42:06.6138908Z 2025-08-26T20:42:06.6139010Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6139218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6139283Z return mod(**inputs) 2025-08-26T20:42:06.6139518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6139601Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6139839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6139915Z layer_outputs = layer_module( 2025-08-26T20:42:06.6140134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6140212Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6140457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6140548Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6140789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6140910Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6141147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.6141236Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.6141239Z 2025-08-26T20:42:06.6141340Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6141547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6141616Z return mod(**inputs) 2025-08-26T20:42:06.6141875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6141951Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6142205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6142286Z layer_outputs = layer_module( 2025-08-26T20:42:06.6142518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6142609Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6142856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6142952Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6143212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6143339Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6143605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.6143686Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.6143692Z 2025-08-26T20:42:06.6143803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6144002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6144111Z return mod(**inputs) 2025-08-26T20:42:06.6144355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6144428Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6144672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6144763Z layer_outputs = layer_module( 2025-08-26T20:42:06.6144990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6145077Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6145318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6145417Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6145654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6145771Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6146014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.6146095Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.6146099Z 2025-08-26T20:42:06.6146191Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6146295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6146503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6146571Z return mod(**inputs) 2025-08-26T20:42:06.6146811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6146893Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6147131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6147209Z layer_outputs = layer_module( 2025-08-26T20:42:06.6147431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6147509Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6147753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6147841Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6148116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.6148231Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6148507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6148601Z return self.weight * hidden_states 2025-08-26T20:42:06.6148605Z 2025-08-26T20:42:06.6148714Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6149124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6149200Z return mod(**inputs) 2025-08-26T20:42:06.6149472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6149553Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6149814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6149899Z layer_outputs = layer_module( 2025-08-26T20:42:06.6150136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6150228Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6150487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6150599Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6150867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6150958Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6151218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6151320Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6151324Z 2025-08-26T20:42:06.6151439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6151649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6151721Z return mod(**inputs) 2025-08-26T20:42:06.6151981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6152060Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6152327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6152402Z layer_outputs = layer_module( 2025-08-26T20:42:06.6152636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6152728Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6152985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6153079Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6153336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6153424Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6153690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6153773Z key_states = self.k(current_states) 2025-08-26T20:42:06.6153777Z 2025-08-26T20:42:06.6153892Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6154103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6154181Z return mod(**inputs) 2025-08-26T20:42:06.6154433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6154535Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6154801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6154892Z layer_outputs = layer_module( 2025-08-26T20:42:06.6155138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6155225Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6155480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6155573Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6155828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6155924Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6156176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6156314Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6156324Z 2025-08-26T20:42:06.6156435Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6156642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6156736Z return mod(**inputs) 2025-08-26T20:42:06.6156999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6157083Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6157349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6157425Z layer_outputs = layer_module( 2025-08-26T20:42:06.6157696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6157781Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6158037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6158125Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6158375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6158473Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6158736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6158912Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6158916Z 2025-08-26T20:42:06.6159029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6159256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6159330Z return mod(**inputs) 2025-08-26T20:42:06.6159650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6159747Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6160007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6160095Z layer_outputs = layer_module( 2025-08-26T20:42:06.6160349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6160432Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6160687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6160773Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6161030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6161142Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6161414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6161507Z value_states = self.v(current_states) 2025-08-26T20:42:06.6161511Z 2025-08-26T20:42:06.6161622Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6161842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6161914Z return mod(**inputs) 2025-08-26T20:42:06.6162173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6162254Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6162509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6162593Z layer_outputs = layer_module( 2025-08-26T20:42:06.6162827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6162917Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6163162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6163266Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6163524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6163611Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6163866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6163998Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6164002Z 2025-08-26T20:42:06.6164110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6164329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6164401Z return mod(**inputs) 2025-08-26T20:42:06.6164659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6164736Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6164997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6165073Z layer_outputs = layer_module( 2025-08-26T20:42:06.6165310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6165401Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6165652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6165745Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6165995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6166084Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6166340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6166456Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6166460Z 2025-08-26T20:42:06.6166576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6166786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6166864Z return mod(**inputs) 2025-08-26T20:42:06.6167116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6167196Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6167469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6167547Z layer_outputs = layer_module( 2025-08-26T20:42:06.6167807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6167892Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6168147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6168241Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6168495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6168590Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6168843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6168959Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6168970Z 2025-08-26T20:42:06.6169079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6169299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6169374Z return mod(**inputs) 2025-08-26T20:42:06.6169614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6169711Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6169950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6170021Z layer_outputs = layer_module( 2025-08-26T20:42:06.6170249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6170342Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6170587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6170667Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6170907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6171000Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6171249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6171337Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6171341Z 2025-08-26T20:42:06.6171425Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6171534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6171750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6171820Z return mod(**inputs) 2025-08-26T20:42:06.6172063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6172135Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6172387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6172463Z layer_outputs = layer_module( 2025-08-26T20:42:06.6172699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6172789Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6173037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6173126Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6173360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-26T20:42:06.6173483Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6173729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6173825Z return self.weight * hidden_states 2025-08-26T20:42:06.6173829Z 2025-08-26T20:42:06.6173940Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6174143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6174211Z return mod(**inputs) 2025-08-26T20:42:06.6174456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6174530Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6174774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6174849Z layer_outputs = layer_module( 2025-08-26T20:42:06.6175083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6175161Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6175399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6175488Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6175740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6175834Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6176067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6176146Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6176164Z 2025-08-26T20:42:06.6176277Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6176476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6176551Z return mod(**inputs) 2025-08-26T20:42:06.6176788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6176860Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6177101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6177175Z layer_outputs = layer_module( 2025-08-26T20:42:06.6177403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6177481Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6177725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6177809Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6178044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6178134Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6178366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6178451Z key_states = self.k(current_states) 2025-08-26T20:42:06.6178456Z 2025-08-26T20:42:06.6178560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6178758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6178831Z return mod(**inputs) 2025-08-26T20:42:06.6179071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6179154Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6179388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6179476Z layer_outputs = layer_module( 2025-08-26T20:42:06.6179720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6179816Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6180074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6180159Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6180413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6180500Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6180749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6180896Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6180900Z 2025-08-26T20:42:06.6181012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6181229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6181300Z return mod(**inputs) 2025-08-26T20:42:06.6181551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6181667Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6181920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6182004Z layer_outputs = layer_module( 2025-08-26T20:42:06.6182240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6182350Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6182603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6182689Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6182944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6183032Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6183282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6183446Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6183450Z 2025-08-26T20:42:06.6183557Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6183774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6183844Z return mod(**inputs) 2025-08-26T20:42:06.6184101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6184179Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6184429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6184513Z layer_outputs = layer_module( 2025-08-26T20:42:06.6184753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6184839Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6185070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6185157Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6185391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6185474Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6185732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6185813Z value_states = self.v(current_states) 2025-08-26T20:42:06.6185817Z 2025-08-26T20:42:06.6185941Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6186143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6186210Z return mod(**inputs) 2025-08-26T20:42:06.6186458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6186536Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6186791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6186866Z layer_outputs = layer_module( 2025-08-26T20:42:06.6187110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6187197Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6187435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6187523Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6187762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6187877Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6188123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6188237Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6188241Z 2025-08-26T20:42:06.6188359Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6188588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6188663Z return mod(**inputs) 2025-08-26T20:42:06.6188903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6188980Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6189241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6189319Z layer_outputs = layer_module( 2025-08-26T20:42:06.6189563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6189646Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6189904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6189989Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6190236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6190332Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6190579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6190704Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6190709Z 2025-08-26T20:42:06.6190818Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6191031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6191110Z return mod(**inputs) 2025-08-26T20:42:06.6191360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6191445Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6191695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6191771Z layer_outputs = layer_module( 2025-08-26T20:42:06.6192032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6192115Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6192384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6192471Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6192727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6192814Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6193060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6193183Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6193187Z 2025-08-26T20:42:06.6193296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6193518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6193587Z return mod(**inputs) 2025-08-26T20:42:06.6193840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6193926Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6194194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6194276Z layer_outputs = layer_module( 2025-08-26T20:42:06.6194513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6194594Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6194867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6194954Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6195210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6195298Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6195553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6195638Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6195642Z 2025-08-26T20:42:06.6195752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6195968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6196039Z return mod(**inputs) 2025-08-26T20:42:06.6196459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6196547Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6196804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6196889Z layer_outputs = layer_module( 2025-08-26T20:42:06.6197131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6197224Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6197486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6197582Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6197841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward 2025-08-26T20:42:06.6197985Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-26T20:42:06.6197991Z 2025-08-26T20:42:06.6198088Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6198203Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6198472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6198545Z return mod(**inputs) 2025-08-26T20:42:06.6198827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6198918Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6199180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6199265Z layer_outputs = layer_module( 2025-08-26T20:42:06.6199560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6199649Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6199914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6200017Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6200280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.6200388Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6200650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6200772Z return self.weight * hidden_states 2025-08-26T20:42:06.6200776Z 2025-08-26T20:42:06.6200890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6201130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6201203Z return mod(**inputs) 2025-08-26T20:42:06.6201469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6201575Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6201836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6201920Z layer_outputs = layer_module( 2025-08-26T20:42:06.6202162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6202253Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6202510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6202608Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6202871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6202996Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6203261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.6203350Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.6203354Z 2025-08-26T20:42:06.6203471Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6203690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6203760Z return mod(**inputs) 2025-08-26T20:42:06.6204031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6204109Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6204375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6204450Z layer_outputs = layer_module( 2025-08-26T20:42:06.6204703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6204792Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6205051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6205153Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6205413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6205541Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6205782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.6205864Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.6205867Z 2025-08-26T20:42:06.6205979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6206184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6206259Z return mod(**inputs) 2025-08-26T20:42:06.6206503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6206576Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6206822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6206893Z layer_outputs = layer_module( 2025-08-26T20:42:06.6207144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6207224Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6207459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6207558Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6207788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6207930Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6208162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.6208252Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.6208255Z 2025-08-26T20:42:06.6208337Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6208440Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6208648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6208713Z return mod(**inputs) 2025-08-26T20:42:06.6208957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6209031Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6209266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6209345Z layer_outputs = layer_module( 2025-08-26T20:42:06.6209565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6209653Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6209901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6209987Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6210243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.6210358Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6210610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6210694Z return self.weight * hidden_states 2025-08-26T20:42:06.6210698Z 2025-08-26T20:42:06.6210814Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6211044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6211116Z return mod(**inputs) 2025-08-26T20:42:06.6211394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6211483Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6211732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6211804Z layer_outputs = layer_module( 2025-08-26T20:42:06.6212031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6212118Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6212355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6212446Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6212686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6212769Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6213012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6213109Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6213113Z 2025-08-26T20:42:06.6213223Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6213422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6213496Z return mod(**inputs) 2025-08-26T20:42:06.6213731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6213819Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6214066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6214139Z layer_outputs = layer_module( 2025-08-26T20:42:06.6214369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6214448Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6214690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6214780Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6215013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6215102Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6215337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6215417Z key_states = self.k(current_states) 2025-08-26T20:42:06.6215428Z 2025-08-26T20:42:06.6215531Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6215730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6215804Z return mod(**inputs) 2025-08-26T20:42:06.6216040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6216121Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6216357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6216429Z layer_outputs = layer_module( 2025-08-26T20:42:06.6216657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6216740Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6216998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6217080Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6217329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6217419Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6217655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6217793Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6217797Z 2025-08-26T20:42:06.6217898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6218104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6218172Z return mod(**inputs) 2025-08-26T20:42:06.6218408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6218491Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6218729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6218808Z layer_outputs = layer_module( 2025-08-26T20:42:06.6219028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6219123Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6219364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6219445Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6219687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6219787Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6220024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6220188Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6220192Z 2025-08-26T20:42:06.6220298Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6220506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6220575Z return mod(**inputs) 2025-08-26T20:42:06.6220822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6220895Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6221135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6221218Z layer_outputs = layer_module( 2025-08-26T20:42:06.6221442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6221529Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6221765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6221847Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6222087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6222172Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6222413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6222491Z value_states = self.v(current_states) 2025-08-26T20:42:06.6222495Z 2025-08-26T20:42:06.6222608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6222826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6223234Z return mod(**inputs) 2025-08-26T20:42:06.6223500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6223594Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6223854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6223934Z layer_outputs = layer_module( 2025-08-26T20:42:06.6224169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6224260Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6224513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6224618Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6224854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6224937Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6225184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6225302Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6225306Z 2025-08-26T20:42:06.6225443Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6225654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6225733Z return mod(**inputs) 2025-08-26T20:42:06.6225985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6226063Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6226344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6226422Z layer_outputs = layer_module( 2025-08-26T20:42:06.6226667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6226748Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6226987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6227088Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6227317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6227404Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6227632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6227741Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6227751Z 2025-08-26T20:42:06.6227855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6228056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6228130Z return mod(**inputs) 2025-08-26T20:42:06.6228368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6228449Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6228683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6228755Z layer_outputs = layer_module( 2025-08-26T20:42:06.6228981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6229059Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6229300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6229408Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6229651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6229760Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6230008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6230132Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6230136Z 2025-08-26T20:42:06.6230423Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6230627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6230694Z return mod(**inputs) 2025-08-26T20:42:06.6230929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6231011Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6231246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6231324Z layer_outputs = layer_module( 2025-08-26T20:42:06.6231556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6231635Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6231889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6231969Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6232213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6232296Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6232550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6232641Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6232645Z 2025-08-26T20:42:06.6232726Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6232842Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6233041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6233117Z return mod(**inputs) 2025-08-26T20:42:06.6233352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6233425Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6233670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6233742Z layer_outputs = layer_module( 2025-08-26T20:42:06.6233970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6234052Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6234287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6234379Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6234613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-26T20:42:06.6234731Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6234964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6235042Z return self.weight * hidden_states 2025-08-26T20:42:06.6235053Z 2025-08-26T20:42:06.6235157Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6235358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6235432Z return mod(**inputs) 2025-08-26T20:42:06.6235686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6235769Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6236021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6236094Z layer_outputs = layer_module( 2025-08-26T20:42:06.6236331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6236412Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6236670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6236756Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6237011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6237109Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6237362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6237456Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6237460Z 2025-08-26T20:42:06.6237569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6237803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6237873Z return mod(**inputs) 2025-08-26T20:42:06.6238124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6238214Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6238463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6238562Z layer_outputs = layer_module( 2025-08-26T20:42:06.6238801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6238883Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6239144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6239230Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6239557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6239653Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6239900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6239992Z key_states = self.k(current_states) 2025-08-26T20:42:06.6239999Z 2025-08-26T20:42:06.6240109Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6240335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6240408Z return mod(**inputs) 2025-08-26T20:42:06.6240680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6240762Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6241025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6241115Z layer_outputs = layer_module( 2025-08-26T20:42:06.6241357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6241449Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6241720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6241808Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6242089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6242179Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6242474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6242612Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6242617Z 2025-08-26T20:42:06.6242731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6242944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6243013Z return mod(**inputs) 2025-08-26T20:42:06.6243274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6243353Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6243616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6243693Z layer_outputs = layer_module( 2025-08-26T20:42:06.6243932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6244022Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6244276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6244394Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6244642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6244729Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6244980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6245161Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6245167Z 2025-08-26T20:42:06.6245283Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6245493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6245570Z return mod(**inputs) 2025-08-26T20:42:06.6245822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6245901Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6246160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6246236Z layer_outputs = layer_module( 2025-08-26T20:42:06.6246476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6246559Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6246809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6246904Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6247154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6247252Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6247500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6247584Z value_states = self.v(current_states) 2025-08-26T20:42:06.6247595Z 2025-08-26T20:42:06.6247706Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6247915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6247995Z return mod(**inputs) 2025-08-26T20:42:06.6248248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6248351Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6248617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6248697Z layer_outputs = layer_module( 2025-08-26T20:42:06.6248940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6249026Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6249287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6249374Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6249631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6249722Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6249950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6250065Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6250069Z 2025-08-26T20:42:06.6250170Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6250374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6250458Z return mod(**inputs) 2025-08-26T20:42:06.6250692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6250775Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6251014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6251119Z layer_outputs = layer_module( 2025-08-26T20:42:06.6251341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6251420Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6251669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6251751Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6251994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6252080Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6252313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6252429Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6252433Z 2025-08-26T20:42:06.6252537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6252744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6252814Z return mod(**inputs) 2025-08-26T20:42:06.6253062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6253137Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6253375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6253456Z layer_outputs = layer_module( 2025-08-26T20:42:06.6253682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6253768Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6254016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6254097Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6254353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6254437Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6254689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6254797Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6254801Z 2025-08-26T20:42:06.6254903Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6255103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6255169Z return mod(**inputs) 2025-08-26T20:42:06.6255415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6255491Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6255733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6255806Z layer_outputs = layer_module( 2025-08-26T20:42:06.6256029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6256116Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6256355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6256460Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6256712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6256800Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6257100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6257207Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6257211Z 2025-08-26T20:42:06.6257301Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6257405Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6257603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6257678Z return mod(**inputs) 2025-08-26T20:42:06.6257913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6257995Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6258288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6258362Z layer_outputs = layer_module( 2025-08-26T20:42:06.6258581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6258657Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6258896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6258985Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6259223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.6259320Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6259550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6259637Z return self.weight * hidden_states 2025-08-26T20:42:06.6259640Z 2025-08-26T20:42:06.6259742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6259947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6260013Z return mod(**inputs) 2025-08-26T20:42:06.6260252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6260352Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6260591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6260687Z layer_outputs = layer_module( 2025-08-26T20:42:06.6260912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6260999Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6261236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6261327Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6261567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6261687Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6261930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.6262012Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.6262015Z 2025-08-26T20:42:06.6262120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6262325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6262414Z return mod(**inputs) 2025-08-26T20:42:06.6262661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6262736Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6262981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6263053Z layer_outputs = layer_module( 2025-08-26T20:42:06.6263294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6263384Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6263617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6263716Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6263950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6264069Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6264309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.6264389Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.6264393Z 2025-08-26T20:42:06.6264514Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6264711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6264775Z return mod(**inputs) 2025-08-26T20:42:06.6265013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6265084Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6265320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6265393Z layer_outputs = layer_module( 2025-08-26T20:42:06.6265620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6265699Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6265931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6266029Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6266267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6266410Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6266662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.6266744Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.6266748Z 2025-08-26T20:42:06.6266858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6267057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6267131Z return mod(**inputs) 2025-08-26T20:42:06.6267366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6267440Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6267690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6267762Z layer_outputs = layer_module( 2025-08-26T20:42:06.6267990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6268070Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6268314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6268423Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6268656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-08-26T20:42:06.6268795Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-08-26T20:42:06.6268799Z 2025-08-26T20:42:06.6268882Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6268994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6269211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6269280Z return mod(**inputs) 2025-08-26T20:42:06.6269526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6269602Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6269846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6269921Z layer_outputs = layer_module( 2025-08-26T20:42:06.6270154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6270247Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6270497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6270591Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6270839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-26T20:42:06.6270960Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6271209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6271291Z return self.weight * hidden_states 2025-08-26T20:42:06.6271294Z 2025-08-26T20:42:06.6271412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6271624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6271701Z return mod(**inputs) 2025-08-26T20:42:06.6271950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6272027Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6272288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6272386Z layer_outputs = layer_module( 2025-08-26T20:42:06.6272635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6272730Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6272973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6273056Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6273300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6273398Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6273645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6273737Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6273741Z 2025-08-26T20:42:06.6273850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6274062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6274141Z return mod(**inputs) 2025-08-26T20:42:06.6274389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6274472Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6274763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6274840Z layer_outputs = layer_module( 2025-08-26T20:42:06.6275089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6275174Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6275461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6275548Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6275825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6275919Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6276177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6276270Z key_states = self.k(current_states) 2025-08-26T20:42:06.6276274Z 2025-08-26T20:42:06.6276387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6276611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6276682Z return mod(**inputs) 2025-08-26T20:42:06.6276949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6277040Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6277309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6277393Z layer_outputs = layer_module( 2025-08-26T20:42:06.6277636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6277722Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6277991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6278080Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6278344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6278436Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6278702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6278861Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6278865Z 2025-08-26T20:42:06.6278979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6279218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6279291Z return mod(**inputs) 2025-08-26T20:42:06.6279641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6279730Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6279998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6280087Z layer_outputs = layer_module( 2025-08-26T20:42:06.6280332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6280430Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6280696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6280795Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6281059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6281151Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6281448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6281615Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6281619Z 2025-08-26T20:42:06.6281738Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6281951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6282039Z return mod(**inputs) 2025-08-26T20:42:06.6282330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6282410Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6282679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6282754Z layer_outputs = layer_module( 2025-08-26T20:42:06.6282997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6283087Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6283351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6283442Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6283699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6283795Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6284055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6284138Z value_states = self.v(current_states) 2025-08-26T20:42:06.6284144Z 2025-08-26T20:42:06.6284263Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6284477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6284556Z return mod(**inputs) 2025-08-26T20:42:06.6284810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6284887Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6285149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6285226Z layer_outputs = layer_module( 2025-08-26T20:42:06.6285487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6285572Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6285838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6285931Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6286178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6286275Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6286523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6286649Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6286652Z 2025-08-26T20:42:06.6286763Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6286971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6287052Z return mod(**inputs) 2025-08-26T20:42:06.6287301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6287388Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6287642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6287738Z layer_outputs = layer_module( 2025-08-26T20:42:06.6287980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6288065Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6288320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6288425Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6288679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6288766Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6289013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6289134Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6289140Z 2025-08-26T20:42:06.6289250Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6289467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6289536Z return mod(**inputs) 2025-08-26T20:42:06.6289786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6289872Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6290121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6290206Z layer_outputs = layer_module( 2025-08-26T20:42:06.6290440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6290525Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6290779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6290866Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6291120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6291205Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6291460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6291579Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6291583Z 2025-08-26T20:42:06.6291712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6291934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6292021Z return mod(**inputs) 2025-08-26T20:42:06.6292283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6292362Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6292612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6292697Z layer_outputs = layer_module( 2025-08-26T20:42:06.6292933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6293025Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6293275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-26T20:42:06.6293362Z self_attention_outputs = self.layer[0]( 2025-08-26T20:42:06.6293620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-26T20:42:06.6293710Z attention_output = self.SelfAttention( 2025-08-26T20:42:06.6293965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6294068Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6294072Z 2025-08-26T20:42:06.6294163Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6294275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6294487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6294582Z return mod(**inputs) 2025-08-26T20:42:06.6294833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6294920Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6295173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6295249Z layer_outputs = layer_module( 2025-08-26T20:42:06.6295502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6295588Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6295848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6295936Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6296293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-26T20:42:06.6296429Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6296685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6296777Z return self.weight * hidden_states 2025-08-26T20:42:06.6296782Z 2025-08-26T20:42:06.6296891Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6297112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6297184Z return mod(**inputs) 2025-08-26T20:42:06.6297438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6297525Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6297777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6297861Z layer_outputs = layer_module( 2025-08-26T20:42:06.6298102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6298234Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6298494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6298610Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6298868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6298961Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6299213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-26T20:42:06.6299306Z query_states = self.q(hidden_states) 2025-08-26T20:42:06.6299310Z 2025-08-26T20:42:06.6299421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6299647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6299718Z return mod(**inputs) 2025-08-26T20:42:06.6299979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6300058Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6300311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6300425Z layer_outputs = layer_module( 2025-08-26T20:42:06.6300660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6300753Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6301000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6301085Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6301368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6301459Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6301712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-26T20:42:06.6301795Z key_states = self.k(current_states) 2025-08-26T20:42:06.6301799Z 2025-08-26T20:42:06.6301911Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6302121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6302190Z return mod(**inputs) 2025-08-26T20:42:06.6302446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6302523Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6302778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6302854Z layer_outputs = layer_module( 2025-08-26T20:42:06.6303089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6303180Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6303416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6303500Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6303727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6303808Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6304048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-26T20:42:06.6304179Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-26T20:42:06.6304184Z 2025-08-26T20:42:06.6304292Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6304519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6304594Z return mod(**inputs) 2025-08-26T20:42:06.6304840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6304912Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6305156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6305225Z layer_outputs = layer_module( 2025-08-26T20:42:06.6305447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6305523Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6305750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6305837Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6306064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6306153Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6306383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-26T20:42:06.6306558Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-26T20:42:06.6306568Z 2025-08-26T20:42:06.6306670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6306866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6306939Z return mod(**inputs) 2025-08-26T20:42:06.6307172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6307268Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6307502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6307572Z layer_outputs = layer_module( 2025-08-26T20:42:06.6307797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6307872Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6308112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6308194Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6308431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6308523Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6308761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-26T20:42:06.6308849Z value_states = self.v(current_states) 2025-08-26T20:42:06.6308853Z 2025-08-26T20:42:06.6308955Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6309161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6309227Z return mod(**inputs) 2025-08-26T20:42:06.6309464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6309547Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6309784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6309863Z layer_outputs = layer_module( 2025-08-26T20:42:06.6310086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6310165Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6310422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6310504Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6310776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6310865Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6311117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6311248Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6311251Z 2025-08-26T20:42:06.6311355Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6311562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6311629Z return mod(**inputs) 2025-08-26T20:42:06.6311874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6311948Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6312186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6312267Z layer_outputs = layer_module( 2025-08-26T20:42:06.6312490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6312597Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6312840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6312924Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6313181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6313288Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6313549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-26T20:42:06.6313664Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-26T20:42:06.6313670Z 2025-08-26T20:42:06.6313787Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6314004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6314075Z return mod(**inputs) 2025-08-26T20:42:06.6314339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6314417Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6314679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6314758Z layer_outputs = layer_module( 2025-08-26T20:42:06.6314998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6315088Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6315343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6315435Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6315700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6315790Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6316059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-26T20:42:06.6316173Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-26T20:42:06.6316176Z 2025-08-26T20:42:06.6316296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6316510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6316607Z return mod(**inputs) 2025-08-26T20:42:06.6316875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6316970Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6317232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6317310Z layer_outputs = layer_module( 2025-08-26T20:42:06.6317550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6317633Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6317892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-26T20:42:06.6317988Z cross_attention_outputs = self.layer[1]( 2025-08-26T20:42:06.6318242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-26T20:42:06.6318336Z attention_output = self.EncDecAttention( 2025-08-26T20:42:06.6318595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-26T20:42:06.6318678Z attn_output = self.o(attn_output) 2025-08-26T20:42:06.6318707Z 2025-08-26T20:42:06.6318795Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6318904Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6319126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6319197Z return mod(**inputs) 2025-08-26T20:42:06.6319532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6319638Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6319912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6319997Z layer_outputs = layer_module( 2025-08-26T20:42:06.6320241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6320331Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6320600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6320703Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6320968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-26T20:42:06.6321076Z forwarded_states = self.layer_norm(hidden_states) 2025-08-26T20:42:06.6321344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-26T20:42:06.6321432Z return self.weight * hidden_states 2025-08-26T20:42:06.6321437Z 2025-08-26T20:42:06.6321549Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6321776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6321850Z return mod(**inputs) 2025-08-26T20:42:06.6322119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6322201Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6322466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6322545Z layer_outputs = layer_module( 2025-08-26T20:42:06.6322789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6322884Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6323171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6323280Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6323559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6323691Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6323978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-26T20:42:06.6324066Z hidden_states = self.wi(hidden_states) 2025-08-26T20:42:06.6324070Z 2025-08-26T20:42:06.6324190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6324409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6324491Z return mod(**inputs) 2025-08-26T20:42:06.6324760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6324840Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6325110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6325187Z layer_outputs = layer_module( 2025-08-26T20:42:06.6325438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6325543Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6325807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6325912Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6326165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6326322Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6326579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-26T20:42:06.6326668Z hidden_states = self.act(hidden_states) 2025-08-26T20:42:06.6326680Z 2025-08-26T20:42:06.6326794Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6327011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6327092Z return mod(**inputs) 2025-08-26T20:42:06.6327351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-26T20:42:06.6327437Z decoder_outputs = self.decoder( 2025-08-26T20:42:06.6327697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-26T20:42:06.6327768Z layer_outputs = layer_module( 2025-08-26T20:42:06.6327991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:06.6328070Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:06.6328310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-26T20:42:06.6328401Z hidden_states = self.layer[-1](hidden_states) 2025-08-26T20:42:06.6328637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-26T20:42:06.6328766Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-26T20:42:06.6329003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-26T20:42:06.6329090Z hidden_states = self.wo(hidden_states) 2025-08-26T20:42:06.6329094Z 2025-08-26T20:42:06.6329175Z cudagraph partition due to non gpu ops 2025-08-26T20:42:06.6329280Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6329503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6329572Z return mod(**inputs) 2025-08-26T20:42:06.6329831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward 2025-08-26T20:42:06.6329957Z sequence_output = sequence_output * (self.model_dim**-0.5) 2025-08-26T20:42:06.6329962Z 2025-08-26T20:42:06.6330071Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6330272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6330343Z return mod(**inputs) 2025-08-26T20:42:06.6330602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward 2025-08-26T20:42:06.6330694Z lm_logits = self.lm_head(sequence_output) 2025-08-26T20:42:06.6330699Z 2025-08-26T20:42:06.6330816Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6331026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6331096Z return mod(**inputs) 2025-08-26T20:42:06.6331361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-26T20:42:06.6331511Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-26T20:42:06.6331542Z 2025-08-26T20:42:06.6331653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6331851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6331924Z return mod(**inputs) 2025-08-26T20:42:06.6332177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-26T20:42:06.6332329Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-26T20:42:06.6332332Z 2025-08-26T20:42:06.6332441Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:06.6332634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:06.6332705Z return mod(**inputs) 2025-08-26T20:42:06.6332936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-26T20:42:06.6333065Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-26T20:42:06.6333076Z 2025-08-26T20:42:16.8165799Z Compilation time (from dynamo_timed): 18.865503961 2025-08-26T20:42:16.8332803Z pass 2025-08-26T20:42:16.8333514Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:16.8334445Z TIMING: _recursive_pre_grad_passes:0.012 _recursive_joint_graph_passes:0.60287 _recursive_post_grad_passes:0.19883 async_compile.wait:0.79898 code_gen:9.80707 inductor_compile:11.57387 backend_compile:15.83149 gc:0.00122 entire_frame_compile:18.8655 total_wall_time:18.8655 2025-08-26T20:42:16.8335474Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:20423 | FakeTensor.__torch_dispatch__:5324 | ProxyTorchDispatchMode.__torch_dispatch__:7292 2025-08-26T20:42:16.8336052Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-08-26T20:42:22.5033525Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:42:22.5034598Z from pkg_resources import resource_filename 2025-08-26T20:42:23.1266980Z 2025-08-26T20:42:24.3381967Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:42:24.3386941Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:42:24.3392306Z cpu eval T5Small 2025-08-26T20:42:25.5006438Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:25.8979190Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:26.3050705Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:38.1636010Z Compilation time (from dynamo_timed): 10.202766968 2025-08-26T20:42:38.1743954Z pass 2025-08-26T20:42:38.1744393Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:38.1745033Z TIMING: _recursive_pre_grad_passes:0.01383 async_compile.wait:0.00642 backend_compile:6.87905 gc:0.00027 entire_frame_compile:10.20277 total_wall_time:10.20277 2025-08-26T20:42:38.1745659Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:2289 | FakeTensor.__torch_dispatch__:17 2025-08-26T20:42:38.1746138Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-08-26T20:42:43.4221960Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:42:43.4222955Z from pkg_resources import resource_filename 2025-08-26T20:42:44.0236513Z 2025-08-26T20:42:46.7390823Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:42:46.7391143Z loading model: 0it [00:02, ?it/s] 2025-08-26T20:42:46.7401380Z cpu eval TrOCRForCausalLM 2025-08-26T20:42:46.8833492Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-08-26T20:42:46.9134914Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:47.1880181Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:47.4556959Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:42:55.5655872Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5656464Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5656798Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5657105Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5657465Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5657797Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5658097Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5658486Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5658827Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5659689Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5662289Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5662758Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5663211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5663773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5664257Z return mod(**inputs) 2025-08-26T20:42:55.5664737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5665189Z outputs = self.model.decoder( 2025-08-26T20:42:55.5665658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5666094Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5666535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5666941Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5667364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5668125Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5668581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5669143Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5669340Z 2025-08-26T20:42:55.5669460Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5669882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5670257Z return mod(**inputs) 2025-08-26T20:42:55.5670662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5671103Z outputs = self.model.decoder( 2025-08-26T20:42:55.5671535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5671982Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5672388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5672809Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5673258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5673819Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5674282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5674732Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5674884Z 2025-08-26T20:42:55.5675003Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5675417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5675863Z return mod(**inputs) 2025-08-26T20:42:55.5676271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5676689Z outputs = self.model.decoder( 2025-08-26T20:42:55.5677092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5677516Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5677904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5678294Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5678719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5679173Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5679763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5680218Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5680378Z 2025-08-26T20:42:55.5680480Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5680721Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5680965Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5681236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5681648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5682029Z return mod(**inputs) 2025-08-26T20:42:55.5682455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5682899Z outputs = self.model.decoder( 2025-08-26T20:42:55.5683447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5683885Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5684300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5684711Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5685179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5685638Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5686102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5686543Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5686703Z 2025-08-26T20:42:55.5686821Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5687219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5687577Z return mod(**inputs) 2025-08-26T20:42:55.5687994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5688437Z outputs = self.model.decoder( 2025-08-26T20:42:55.5688858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5689272Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5689670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5690059Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5690480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5690950Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5691161Z 2025-08-26T20:42:55.5691282Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5691675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5692018Z return mod(**inputs) 2025-08-26T20:42:55.5692413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5692837Z outputs = self.model.decoder( 2025-08-26T20:42:55.5693254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5693676Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5694052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5694445Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5694868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5695338Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5695755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5696136Z return self.act(input) 2025-08-26T20:42:55.5696439Z 2025-08-26T20:42:55.5696560Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5696954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5697312Z return mod(**inputs) 2025-08-26T20:42:55.5697698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5698117Z outputs = self.model.decoder( 2025-08-26T20:42:55.5698530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5698950Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5699319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5699750Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5700196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5700623Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5700770Z 2025-08-26T20:42:55.5700888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5701270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5701618Z return mod(**inputs) 2025-08-26T20:42:55.5702002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5702418Z outputs = self.model.decoder( 2025-08-26T20:42:55.5702827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5703235Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5703610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5703996Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5704410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5704883Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5705301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5705755Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5705934Z 2025-08-26T20:42:55.5706054Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5706477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5706820Z return mod(**inputs) 2025-08-26T20:42:55.5707213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5707630Z outputs = self.model.decoder( 2025-08-26T20:42:55.5708043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5708458Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5708829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5709223Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5709646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5710090Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5710529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5710959Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5711114Z 2025-08-26T20:42:55.5711225Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5711615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5711989Z return mod(**inputs) 2025-08-26T20:42:55.5712374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5712790Z outputs = self.model.decoder( 2025-08-26T20:42:55.5713198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5713617Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5713999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5714382Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5714828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5715286Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5715730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5716157Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5716317Z 2025-08-26T20:42:55.5716404Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5716635Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5716863Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5717114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5717492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5717845Z return mod(**inputs) 2025-08-26T20:42:55.5718249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5718677Z outputs = self.model.decoder( 2025-08-26T20:42:55.5719076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5719700Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5720127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5720532Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5720966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5721399Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5721840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5722243Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5722385Z 2025-08-26T20:42:55.5722497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5722867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5723204Z return mod(**inputs) 2025-08-26T20:42:55.5723601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5724008Z outputs = self.model.decoder( 2025-08-26T20:42:55.5724405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5724817Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5725186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5725556Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5725955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5726400Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5726574Z 2025-08-26T20:42:55.5726681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5727057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5727391Z return mod(**inputs) 2025-08-26T20:42:55.5727759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5728159Z outputs = self.model.decoder( 2025-08-26T20:42:55.5728536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5728926Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5729300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5729668Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5730094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5730545Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5730967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5731340Z return self.act(input) 2025-08-26T20:42:55.5731460Z 2025-08-26T20:42:55.5731579Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5731952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5732291Z return mod(**inputs) 2025-08-26T20:42:55.5732660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5733073Z outputs = self.model.decoder( 2025-08-26T20:42:55.5733488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5733881Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5734236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5734624Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5735039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5735462Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5735619Z 2025-08-26T20:42:55.5735730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5736142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5736493Z return mod(**inputs) 2025-08-26T20:42:55.5736877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5737278Z outputs = self.model.decoder( 2025-08-26T20:42:55.5737690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5738108Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5738479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5738866Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5739290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5739737Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5740187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5740645Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5740822Z 2025-08-26T20:42:55.5740935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5741327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5741680Z return mod(**inputs) 2025-08-26T20:42:55.5742076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5742500Z outputs = self.model.decoder( 2025-08-26T20:42:55.5742902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5743319Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5743704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5744120Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5744540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5745003Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5745445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5745882Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5746026Z 2025-08-26T20:42:55.5746148Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5746531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5746878Z return mod(**inputs) 2025-08-26T20:42:55.5747263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5747685Z outputs = self.model.decoder( 2025-08-26T20:42:55.5748100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5748509Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5748884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5749293Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5749749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5750165Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5750607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5751058Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5751203Z 2025-08-26T20:42:55.5751296Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5751516Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5751728Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5751973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5752344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5752672Z return mod(**inputs) 2025-08-26T20:42:55.5753033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5753428Z outputs = self.model.decoder( 2025-08-26T20:42:55.5753811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5754234Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5754610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5754998Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5755417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5755862Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5756296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5756716Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5756863Z 2025-08-26T20:42:55.5756979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5757365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5757714Z return mod(**inputs) 2025-08-26T20:42:55.5758101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5758512Z outputs = self.model.decoder( 2025-08-26T20:42:55.5758937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5759435Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5759922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5760341Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5760768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5761248Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5761430Z 2025-08-26T20:42:55.5761534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5761903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5762231Z return mod(**inputs) 2025-08-26T20:42:55.5762607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5763000Z outputs = self.model.decoder( 2025-08-26T20:42:55.5763393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5763787Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5764153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5764521Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5764915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5765358Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5765770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5766116Z return self.act(input) 2025-08-26T20:42:55.5766238Z 2025-08-26T20:42:55.5766346Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5766719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5767053Z return mod(**inputs) 2025-08-26T20:42:55.5767446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5767926Z outputs = self.model.decoder( 2025-08-26T20:42:55.5768320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5768719Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5769080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5769447Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5769883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5770322Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5770463Z 2025-08-26T20:42:55.5770576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5770958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5771309Z return mod(**inputs) 2025-08-26T20:42:55.5771697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5772135Z outputs = self.model.decoder( 2025-08-26T20:42:55.5772546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5772953Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5773317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5773733Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5774155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5774628Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5775059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5775518Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5775702Z 2025-08-26T20:42:55.5775814Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5776203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5776559Z return mod(**inputs) 2025-08-26T20:42:55.5776945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5777370Z outputs = self.model.decoder( 2025-08-26T20:42:55.5777779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5778219Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5778582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5779010Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5779432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5779893Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5780337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5780783Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5780939Z 2025-08-26T20:42:55.5781058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5781459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5781825Z return mod(**inputs) 2025-08-26T20:42:55.5782228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5782651Z outputs = self.model.decoder( 2025-08-26T20:42:55.5783084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5783504Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5783878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5784259Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5784683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5785129Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5785587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5786030Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5786184Z 2025-08-26T20:42:55.5786278Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5786519Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5786754Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5787015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5787415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5787767Z return mod(**inputs) 2025-08-26T20:42:55.5788155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5788585Z outputs = self.model.decoder( 2025-08-26T20:42:55.5789031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5789447Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5789862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5790254Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5790678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5791123Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5791569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5792006Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5792217Z 2025-08-26T20:42:55.5792397Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5792877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5793509Z return mod(**inputs) 2025-08-26T20:42:55.5794027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5794540Z outputs = self.model.decoder( 2025-08-26T20:42:55.5795139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5795627Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5796095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5796820Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5797435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5798002Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5798248Z 2025-08-26T20:42:55.5798402Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5798885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5799307Z return mod(**inputs) 2025-08-26T20:42:55.5799870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5800412Z outputs = self.model.decoder( 2025-08-26T20:42:55.5800867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5801406Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5801875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5802357Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5802881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5803442Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5803962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5804431Z return self.act(input) 2025-08-26T20:42:55.5804579Z 2025-08-26T20:42:55.5804750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5805200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5805649Z return mod(**inputs) 2025-08-26T20:42:55.5806125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5806688Z outputs = self.model.decoder( 2025-08-26T20:42:55.5807229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5807687Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5808217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5819392Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5820053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5820511Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5820668Z 2025-08-26T20:42:55.5820799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5821202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5821560Z return mod(**inputs) 2025-08-26T20:42:55.5821981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5822435Z outputs = self.model.decoder( 2025-08-26T20:42:55.5822853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5823281Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5823661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5824192Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5824617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5825078Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5825544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5826066Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5826258Z 2025-08-26T20:42:55.5826387Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5826777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5827138Z return mod(**inputs) 2025-08-26T20:42:55.5827535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5827965Z outputs = self.model.decoder( 2025-08-26T20:42:55.5828381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5828792Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5829169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5829574Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5830002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5830444Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5830894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5831320Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5831469Z 2025-08-26T20:42:55.5831593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5831984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5832330Z return mod(**inputs) 2025-08-26T20:42:55.5832721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5833139Z outputs = self.model.decoder( 2025-08-26T20:42:55.5833546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5833991Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5834369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5834799Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5835236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5835694Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5836139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5836583Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5836748Z 2025-08-26T20:42:55.5836837Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5837079Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5837309Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5837562Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5837958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5838322Z return mod(**inputs) 2025-08-26T20:42:55.5838723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5839251Z outputs = self.model.decoder( 2025-08-26T20:42:55.5839770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5840207Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5840600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5841004Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5841453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5841909Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5842367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5842810Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5842963Z 2025-08-26T20:42:55.5843092Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5843489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5843850Z return mod(**inputs) 2025-08-26T20:42:55.5844248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5844682Z outputs = self.model.decoder( 2025-08-26T20:42:55.5845100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5845530Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5845916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5846317Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5846750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5847229Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5847431Z 2025-08-26T20:42:55.5847546Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5847946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5848307Z return mod(**inputs) 2025-08-26T20:42:55.5848700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5849135Z outputs = self.model.decoder( 2025-08-26T20:42:55.5849581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5850014Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5850420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5850816Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5851249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5851732Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5852164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5852546Z return self.act(input) 2025-08-26T20:42:55.5852672Z 2025-08-26T20:42:55.5852787Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5853187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5853545Z return mod(**inputs) 2025-08-26T20:42:55.5853948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5854380Z outputs = self.model.decoder( 2025-08-26T20:42:55.5854787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5855224Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5855604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5856010Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5856444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5856921Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5857075Z 2025-08-26T20:42:55.5857190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5857574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5857921Z return mod(**inputs) 2025-08-26T20:42:55.5858306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5858724Z outputs = self.model.decoder( 2025-08-26T20:42:55.5859132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5859564Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5859934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5860324Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5860745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5861199Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5861643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5862091Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5862277Z 2025-08-26T20:42:55.5862390Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5862782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5863126Z return mod(**inputs) 2025-08-26T20:42:55.5863513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5863924Z outputs = self.model.decoder( 2025-08-26T20:42:55.5864340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5864792Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5865170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5865583Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5866022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5866481Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5866930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5867375Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5867524Z 2025-08-26T20:42:55.5867639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5868040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5868413Z return mod(**inputs) 2025-08-26T20:42:55.5868815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5869255Z outputs = self.model.decoder( 2025-08-26T20:42:55.5869675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5870120Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5870495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5870888Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5871302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5871788Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5872255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5872709Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5872864Z 2025-08-26T20:42:55.5872965Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5873196Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5873431Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5873696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5874089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5874439Z return mod(**inputs) 2025-08-26T20:42:55.5874834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5875272Z outputs = self.model.decoder( 2025-08-26T20:42:55.5875689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5876126Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5876501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5876903Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5877328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5877778Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5878220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5878663Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5878820Z 2025-08-26T20:42:55.5878936Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5879334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5879798Z return mod(**inputs) 2025-08-26T20:42:55.5880229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5880666Z outputs = self.model.decoder( 2025-08-26T20:42:55.5881109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5881544Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5881931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5882325Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5882756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5883237Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5883432Z 2025-08-26T20:42:55.5883555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5883945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5884301Z return mod(**inputs) 2025-08-26T20:42:55.5884721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5885152Z outputs = self.model.decoder( 2025-08-26T20:42:55.5885596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5886023Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5886413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5886816Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5887265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5887734Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5888162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5888540Z return self.act(input) 2025-08-26T20:42:55.5888664Z 2025-08-26T20:42:55.5888787Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5889189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5889537Z return mod(**inputs) 2025-08-26T20:42:55.5889940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5890371Z outputs = self.model.decoder( 2025-08-26T20:42:55.5890793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5891206Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5891573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5891955Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5892369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5892796Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5892945Z 2025-08-26T20:42:55.5893055Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5893443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5893787Z return mod(**inputs) 2025-08-26T20:42:55.5894169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5894583Z outputs = self.model.decoder( 2025-08-26T20:42:55.5895015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5895432Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5895848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5896476Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5896896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5897346Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5897788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5898265Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5898445Z 2025-08-26T20:42:55.5898569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5898949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5899305Z return mod(**inputs) 2025-08-26T20:42:55.5899703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5900165Z outputs = self.model.decoder( 2025-08-26T20:42:55.5900569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5901063Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5901456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5901850Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5902264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5902745Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5903182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5903607Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5903754Z 2025-08-26T20:42:55.5903882Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5904266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5904604Z return mod(**inputs) 2025-08-26T20:42:55.5904989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5905402Z outputs = self.model.decoder( 2025-08-26T20:42:55.5905810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5906224Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5906592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5906983Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5907401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5907838Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5908270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5908697Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5908852Z 2025-08-26T20:42:55.5909090Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5909321Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5909546Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5909795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5910182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5910575Z return mod(**inputs) 2025-08-26T20:42:55.5910984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5911460Z outputs = self.model.decoder( 2025-08-26T20:42:55.5911867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5912275Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5912647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5913033Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5913438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5913881Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5914325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5914765Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5914916Z 2025-08-26T20:42:55.5915040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5915430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5915806Z return mod(**inputs) 2025-08-26T20:42:55.5916201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5916640Z outputs = self.model.decoder( 2025-08-26T20:42:55.5917048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5917488Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5917892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5918302Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5918732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5919212Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5919471Z 2025-08-26T20:42:55.5919593Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5920003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5920362Z return mod(**inputs) 2025-08-26T20:42:55.5920762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5921198Z outputs = self.model.decoder( 2025-08-26T20:42:55.5921628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5922068Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5922460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5922856Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5923293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5923775Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5924208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5924589Z return self.act(input) 2025-08-26T20:42:55.5924713Z 2025-08-26T20:42:55.5924829Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5925234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5925594Z return mod(**inputs) 2025-08-26T20:42:55.5926023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5926455Z outputs = self.model.decoder( 2025-08-26T20:42:55.5926883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5927312Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5927698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5928098Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5928525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5928962Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5929126Z 2025-08-26T20:42:55.5929241Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5929640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5930006Z return mod(**inputs) 2025-08-26T20:42:55.5930387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5930804Z outputs = self.model.decoder( 2025-08-26T20:42:55.5931212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5931652Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5932026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5932404Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5932824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5933285Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5933726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5934173Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5934360Z 2025-08-26T20:42:55.5934472Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5934858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5935205Z return mod(**inputs) 2025-08-26T20:42:55.5935589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5935997Z outputs = self.model.decoder( 2025-08-26T20:42:55.5936405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5936819Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5937190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5937578Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5937991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5938429Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5938864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5939292Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5939439Z 2025-08-26T20:42:55.5939548Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5939932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5940280Z return mod(**inputs) 2025-08-26T20:42:55.5940663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5941114Z outputs = self.model.decoder( 2025-08-26T20:42:55.5941551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5941975Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5942344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5942745Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5943169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5943623Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5944075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5944502Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5944654Z 2025-08-26T20:42:55.5944749Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5944973Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5945199Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5945455Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5945839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5946223Z return mod(**inputs) 2025-08-26T20:42:55.5946632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5947060Z outputs = self.model.decoder( 2025-08-26T20:42:55.5947481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5947942Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5948322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5948720Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5949158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5949622Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5950087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5950531Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5950907Z 2025-08-26T20:42:55.5951027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5951430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5951801Z return mod(**inputs) 2025-08-26T20:42:55.5952207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5952650Z outputs = self.model.decoder( 2025-08-26T20:42:55.5953079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5953523Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5953912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5954311Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5954755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5955243Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5955432Z 2025-08-26T20:42:55.5955555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5955958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5956308Z return mod(**inputs) 2025-08-26T20:42:55.5956774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5957208Z outputs = self.model.decoder( 2025-08-26T20:42:55.5957657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5958078Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5958460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5958861Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5959288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5959840Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5960266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5960647Z return self.act(input) 2025-08-26T20:42:55.5960779Z 2025-08-26T20:42:55.5960894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5961296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5961654Z return mod(**inputs) 2025-08-26T20:42:55.5962087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5962523Z outputs = self.model.decoder( 2025-08-26T20:42:55.5962950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5963384Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5963749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5964159Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5964574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.5964998Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.5965152Z 2025-08-26T20:42:55.5965274Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5965654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5966003Z return mod(**inputs) 2025-08-26T20:42:55.5966393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5966808Z outputs = self.model.decoder( 2025-08-26T20:42:55.5967206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5967625Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5967998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5968388Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5968808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5969248Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5969690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.5970148Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.5970325Z 2025-08-26T20:42:55.5970445Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5970832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5971175Z return mod(**inputs) 2025-08-26T20:42:55.5971592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5972010Z outputs = self.model.decoder( 2025-08-26T20:42:55.5972431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5972835Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5973211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5973600Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5974016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5974459Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5974895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.5975321Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.5975475Z 2025-08-26T20:42:55.5975587Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5975975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5976318Z return mod(**inputs) 2025-08-26T20:42:55.5976697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5977132Z outputs = self.model.decoder( 2025-08-26T20:42:55.5977539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5977963Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5978331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5978740Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5979159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5979598Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5980039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.5980459Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.5980617Z 2025-08-26T20:42:55.5980705Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5980935Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5981164Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.5981409Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5981796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5982216Z return mod(**inputs) 2025-08-26T20:42:55.5982645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5983088Z outputs = self.model.decoder( 2025-08-26T20:42:55.5983485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5983896Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5984264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5984657Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5985078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.5985519Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.5985968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.5986400Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.5986547Z 2025-08-26T20:42:55.5986694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5987079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5987447Z return mod(**inputs) 2025-08-26T20:42:55.5987844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5988261Z outputs = self.model.decoder( 2025-08-26T20:42:55.5988668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5989074Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5989440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5989827Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5990244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5990705Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5990886Z 2025-08-26T20:42:55.5991000Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5991384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5991772Z return mod(**inputs) 2025-08-26T20:42:55.5992166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5992585Z outputs = self.model.decoder( 2025-08-26T20:42:55.5992993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5993413Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5993805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5994194Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.5994614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.5995093Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.5995506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.5995887Z return self.act(input) 2025-08-26T20:42:55.5996006Z 2025-08-26T20:42:55.5996125Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.5996663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.5997035Z return mod(**inputs) 2025-08-26T20:42:55.5997453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.5997891Z outputs = self.model.decoder( 2025-08-26T20:42:55.5998305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.5998736Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.5999130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.5999574Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6000016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.6000456Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.6000617Z 2025-08-26T20:42:55.6000731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6001132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6001493Z return mod(**inputs) 2025-08-26T20:42:55.6001968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6002399Z outputs = self.model.decoder( 2025-08-26T20:42:55.6002846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6003277Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6003663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6004060Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6004487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6004943Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6005392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.6005863Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.6006047Z 2025-08-26T20:42:55.6006160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6006559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6006917Z return mod(**inputs) 2025-08-26T20:42:55.6007313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6007765Z outputs = self.model.decoder( 2025-08-26T20:42:55.6008182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6008612Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6008999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6009445Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6009870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6010335Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6010774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.6011197Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.6011345Z 2025-08-26T20:42:55.6011462Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6011840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6012187Z return mod(**inputs) 2025-08-26T20:42:55.6012573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6012992Z outputs = self.model.decoder( 2025-08-26T20:42:55.6013386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6013801Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6014174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6014555Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6014970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6015406Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6015846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.6016276Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.6016427Z 2025-08-26T20:42:55.6016523Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6016754Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6016975Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6017252Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6017655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6018006Z return mod(**inputs) 2025-08-26T20:42:55.6018387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6018806Z outputs = self.model.decoder( 2025-08-26T20:42:55.6019213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6019648Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6020025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6020434Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6020850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6021295Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6021730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.6022147Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.6022332Z 2025-08-26T20:42:55.6022448Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6022845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6023205Z return mod(**inputs) 2025-08-26T20:42:55.6023623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6024051Z outputs = self.model.decoder( 2025-08-26T20:42:55.6024457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6024869Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6025243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6025635Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6026073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.6026541Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.6026732Z 2025-08-26T20:42:55.6026855Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6027250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6027600Z return mod(**inputs) 2025-08-26T20:42:55.6028001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6028439Z outputs = self.model.decoder( 2025-08-26T20:42:55.6028857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6029284Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6029660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6030060Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6030496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.6030971Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.6031388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.6031766Z return self.act(input) 2025-08-26T20:42:55.6031894Z 2025-08-26T20:42:55.6032008Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6032470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6032830Z return mod(**inputs) 2025-08-26T20:42:55.6033241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6033682Z outputs = self.model.decoder( 2025-08-26T20:42:55.6034105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6034535Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6034921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6035316Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6035752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.6036188Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.6036341Z 2025-08-26T20:42:55.6036463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6036856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6037212Z return mod(**inputs) 2025-08-26T20:42:55.6037605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6038050Z outputs = self.model.decoder( 2025-08-26T20:42:55.6038466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6038888Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6039271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6039770Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6040209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6040659Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6041114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.6041591Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.6041782Z 2025-08-26T20:42:55.6041898Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6042296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6042651Z return mod(**inputs) 2025-08-26T20:42:55.6043047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6043488Z outputs = self.model.decoder( 2025-08-26T20:42:55.6043901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6044319Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6044687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6045074Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6045499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6045973Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6046418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.6046864Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.6047023Z 2025-08-26T20:42:55.6047138Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6047565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6047914Z return mod(**inputs) 2025-08-26T20:42:55.6048306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6048727Z outputs = self.model.decoder( 2025-08-26T20:42:55.6049131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6049547Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6049917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6050322Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6050750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6051218Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6051666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.6052089Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.6052250Z 2025-08-26T20:42:55.6052337Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6052568Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6052817Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6053070Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6053465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6053839Z return mod(**inputs) 2025-08-26T20:42:55.6054249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6054712Z outputs = self.model.decoder( 2025-08-26T20:42:55.6055124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6055550Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6055924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6056317Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6056754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6057212Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6057672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.6058108Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.6058265Z 2025-08-26T20:42:55.6058386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6058773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6059125Z return mod(**inputs) 2025-08-26T20:42:55.6059528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6059959Z outputs = self.model.decoder( 2025-08-26T20:42:55.6060370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6060790Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6061160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6061548Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6061966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.6062431Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.6062613Z 2025-08-26T20:42:55.6062741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6063133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6063502Z return mod(**inputs) 2025-08-26T20:42:55.6063892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6064312Z outputs = self.model.decoder( 2025-08-26T20:42:55.6064713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6065130Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6065506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6065908Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6066321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.6066793Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.6067211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.6067578Z return self.act(input) 2025-08-26T20:42:55.6067696Z 2025-08-26T20:42:55.6067851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6068230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6068585Z return mod(**inputs) 2025-08-26T20:42:55.6068980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6069425Z outputs = self.model.decoder( 2025-08-26T20:42:55.6069849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6070286Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6070673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6071068Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6071484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.6071901Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.6072055Z 2025-08-26T20:42:55.6072167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6072550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6072898Z return mod(**inputs) 2025-08-26T20:42:55.6073281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6073687Z outputs = self.model.decoder( 2025-08-26T20:42:55.6074092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6074504Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6074874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6075253Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6075670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6076106Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6076540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-26T20:42:55.6077006Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:42:55.6077197Z 2025-08-26T20:42:55.6077307Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6077712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6078074Z return mod(**inputs) 2025-08-26T20:42:55.6078498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6078931Z outputs = self.model.decoder( 2025-08-26T20:42:55.6079405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6079854Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6080242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6080645Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6081077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6081198Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6081478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-26T20:42:55.6081581Z key_states = self.k_proj(current_states) 2025-08-26T20:42:55.6081585Z 2025-08-26T20:42:55.6081700Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6081979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6082053Z return mod(**inputs) 2025-08-26T20:42:55.6082335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6082428Z outputs = self.model.decoder( 2025-08-26T20:42:55.6082707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6082819Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6083067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6083154Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6083442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6083551Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6083841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-26T20:42:55.6083937Z value_states = self.v_proj(current_states) 2025-08-26T20:42:55.6083941Z 2025-08-26T20:42:55.6084038Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6084129Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6084214Z cudagraph partition due to non gpu ops 2025-08-26T20:42:55.6084338Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6084560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6084641Z return mod(**inputs) 2025-08-26T20:42:55.6084921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6085003Z outputs = self.model.decoder( 2025-08-26T20:42:55.6085288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6085369Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6085624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6085710Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6085984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-26T20:42:55.6086102Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:42:55.6086397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-26T20:42:55.6086496Z attn_output = self.out_proj(attn_output) 2025-08-26T20:42:55.6086517Z 2025-08-26T20:42:55.6086635Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6086853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6086935Z return mod(**inputs) 2025-08-26T20:42:55.6087216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6087305Z outputs = self.model.decoder( 2025-08-26T20:42:55.6087582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6087671Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6087917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6088003Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6088290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.6088421Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.6088444Z 2025-08-26T20:42:55.6088563Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6088784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6088857Z return mod(**inputs) 2025-08-26T20:42:55.6089141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6089242Z outputs = self.model.decoder( 2025-08-26T20:42:55.6089531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6089614Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6089866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6089952Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6090231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-26T20:42:55.6090383Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:42:55.6090611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:42:55.6090691Z return self.act(input) 2025-08-26T20:42:55.6090695Z 2025-08-26T20:42:55.6090804Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6091017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6091094Z return mod(**inputs) 2025-08-26T20:42:55.6091367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-26T20:42:55.6091451Z outputs = self.model.decoder( 2025-08-26T20:42:55.6091725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-26T20:42:55.6091804Z layer_outputs = decoder_layer( 2025-08-26T20:42:55.6092053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:42:55.6092137Z return super().__call__(*args, **kwargs) 2025-08-26T20:42:55.6092413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-26T20:42:55.6092500Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:42:55.6092504Z 2025-08-26T20:42:55.6092618Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6092845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6092916Z return mod(**inputs) 2025-08-26T20:42:55.6093211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 839, in forward 2025-08-26T20:42:55.6093315Z logits = self.output_projection(outputs[0]) 2025-08-26T20:42:55.6093320Z 2025-08-26T20:42:55.6093436Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:42:55.6093653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:42:55.6093723Z return mod(**inputs) 2025-08-26T20:42:55.6094008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 844, in forward 2025-08-26T20:42:55.6094173Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:42:55.6094177Z 2025-08-26T20:43:04.7979293Z Compilation time (from dynamo_timed): 15.534935369 2025-08-26T20:43:04.8014382Z pass 2025-08-26T20:43:04.8014796Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:43:04.8015683Z TIMING: _recursive_pre_grad_passes:0.00855 _recursive_joint_graph_passes:0.78984 _recursive_post_grad_passes:0.07851 async_compile.wait:0.82771 code_gen:8.16824 inductor_compile:9.42697 backend_compile:13.07257 gc:0.00112 entire_frame_compile:15.53494 total_wall_time:15.53494 2025-08-26T20:43:04.8017116Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:14341 | FakeTensor.__torch_dispatch__:4316 | ProxyTorchDispatchMode.__torch_dispatch__:5467 2025-08-26T20:43:04.8017637Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-08-26T20:43:10.2841341Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:43:10.2842653Z from pkg_resources import resource_filename 2025-08-26T20:43:10.9190066Z 2025-08-26T20:43:17.6597566Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:43:17.6597924Z loading model: 0it [00:06, ?it/s] 2025-08-26T20:43:17.6622285Z cpu eval XGLMForCausalLM 2025-08-26T20:43:18.0477290Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-08-26T20:43:18.1358131Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:43:18.6668174Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:43:19.2352236Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:43:34.9270820Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9271759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9272264Z return mod(**inputs) 2025-08-26T20:43:34.9272777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9273221Z outputs = self.model( 2025-08-26T20:43:34.9273646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9274344Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9274794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9275215Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9275656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9276551Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9277108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9278071Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9278295Z 2025-08-26T20:43:34.9278430Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9278940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9279720Z return mod(**inputs) 2025-08-26T20:43:34.9280188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9280704Z outputs = self.model( 2025-08-26T20:43:34.9281093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9288149Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9288721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9289149Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9289615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9290430Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9290886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9291363Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9291523Z 2025-08-26T20:43:34.9291659Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9292080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9292510Z return mod(**inputs) 2025-08-26T20:43:34.9292939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9293362Z outputs = self.model( 2025-08-26T20:43:34.9293761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9294179Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9294586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9295018Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9295436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9295897Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9296649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9297144Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9297333Z 2025-08-26T20:43:34.9297452Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9297852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9298217Z return mod(**inputs) 2025-08-26T20:43:34.9298613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9299024Z outputs = self.model( 2025-08-26T20:43:34.9299423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9299846Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9300251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9300657Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9301129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9301569Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9302057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9302533Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9302756Z 2025-08-26T20:43:34.9303150Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9303553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9303913Z return mod(**inputs) 2025-08-26T20:43:34.9304297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9304710Z outputs = self.model( 2025-08-26T20:43:34.9305100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9305522Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9305905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9306294Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9306703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9307189Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9307612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9308016Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9308161Z 2025-08-26T20:43:34.9308303Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9308671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9309007Z return mod(**inputs) 2025-08-26T20:43:34.9309377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9309762Z outputs = self.model( 2025-08-26T20:43:34.9310119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9310509Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9310867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9311253Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9311669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9312089Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9312506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9312935Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9313098Z 2025-08-26T20:43:34.9313217Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9313597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9313967Z return mod(**inputs) 2025-08-26T20:43:34.9314355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9314791Z outputs = self.model( 2025-08-26T20:43:34.9315185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9315623Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9316017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9316443Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9316891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9317344Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9317791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9318282Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9318490Z 2025-08-26T20:43:34.9318606Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9319008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9319699Z return mod(**inputs) 2025-08-26T20:43:34.9320134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9320559Z outputs = self.model( 2025-08-26T20:43:34.9320969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9321364Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9321714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9322125Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9322531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9322973Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9323401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9323862Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9324022Z 2025-08-26T20:43:34.9324144Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9324554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9324936Z return mod(**inputs) 2025-08-26T20:43:34.9325325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9325747Z outputs = self.model( 2025-08-26T20:43:34.9326140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9326567Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9326956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9327351Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9327780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9328266Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9328461Z 2025-08-26T20:43:34.9328583Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9328991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9329366Z return mod(**inputs) 2025-08-26T20:43:34.9329774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9330191Z outputs = self.model( 2025-08-26T20:43:34.9330588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9331004Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9331389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9331792Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9332248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9332737Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9333161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9333543Z return self.act(input) 2025-08-26T20:43:34.9333671Z 2025-08-26T20:43:34.9333785Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9334183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9334547Z return mod(**inputs) 2025-08-26T20:43:34.9334939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9335374Z outputs = self.model( 2025-08-26T20:43:34.9335772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9336203Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9336595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9336986Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9337396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9337850Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9338000Z 2025-08-26T20:43:34.9338121Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9338505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9338859Z return mod(**inputs) 2025-08-26T20:43:34.9339265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9339674Z outputs = self.model( 2025-08-26T20:43:34.9340044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9340457Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9340833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9341229Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9341641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9342071Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9342503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9342958Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9343134Z 2025-08-26T20:43:34.9343257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9343641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9343987Z return mod(**inputs) 2025-08-26T20:43:34.9344377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9344799Z outputs = self.model( 2025-08-26T20:43:34.9345191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9345621Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9346009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9346399Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9346809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9347297Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9347750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9348173Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9348323Z 2025-08-26T20:43:34.9348436Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9348823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9349170Z return mod(**inputs) 2025-08-26T20:43:34.9349545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9349965Z outputs = self.model( 2025-08-26T20:43:34.9350350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9350764Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9351116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9351484Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9351884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9352348Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9352788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9353236Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9353423Z 2025-08-26T20:43:34.9353537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9353943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9354329Z return mod(**inputs) 2025-08-26T20:43:34.9354727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9355139Z outputs = self.model( 2025-08-26T20:43:34.9355536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9355959Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9356350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9356745Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9357175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9357624Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9358073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9358568Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9358777Z 2025-08-26T20:43:34.9358893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9359297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9359788Z return mod(**inputs) 2025-08-26T20:43:34.9360213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9360633Z outputs = self.model( 2025-08-26T20:43:34.9361009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9361402Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9361764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9362135Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9362565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9362982Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9363407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9363818Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9363964Z 2025-08-26T20:43:34.9364074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9364434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9364766Z return mod(**inputs) 2025-08-26T20:43:34.9365129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9365518Z outputs = self.model( 2025-08-26T20:43:34.9365872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9366255Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9366608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9366975Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9367408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9367844Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9368254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9368668Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9368841Z 2025-08-26T20:43:34.9368957Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9369346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9369691Z return mod(**inputs) 2025-08-26T20:43:34.9370074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9370482Z outputs = self.model( 2025-08-26T20:43:34.9370842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9371226Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9371583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9371952Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9372343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9372767Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9373192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9373662Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9373868Z 2025-08-26T20:43:34.9373979Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9374367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9374714Z return mod(**inputs) 2025-08-26T20:43:34.9375090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9375495Z outputs = self.model( 2025-08-26T20:43:34.9375876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9376290Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9376688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9377088Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9377518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9377955Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9378389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9378802Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9378957Z 2025-08-26T20:43:34.9379068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9379455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9379812Z return mod(**inputs) 2025-08-26T20:43:34.9380193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9380591Z outputs = self.model( 2025-08-26T20:43:34.9380972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9381417Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9381795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9382344Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9382762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9383248Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9383436Z 2025-08-26T20:43:34.9383579Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9384045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9384387Z return mod(**inputs) 2025-08-26T20:43:34.9384770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9385172Z outputs = self.model( 2025-08-26T20:43:34.9385561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9385966Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9386341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9386727Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9387115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9387549Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9387952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9388319Z return self.act(input) 2025-08-26T20:43:34.9388450Z 2025-08-26T20:43:34.9388555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9388923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9389254Z return mod(**inputs) 2025-08-26T20:43:34.9389612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9389995Z outputs = self.model( 2025-08-26T20:43:34.9390358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9390743Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9391087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9391466Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9391917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9392353Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9392522Z 2025-08-26T20:43:34.9392641Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9393018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9393377Z return mod(**inputs) 2025-08-26T20:43:34.9393736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9394149Z outputs = self.model( 2025-08-26T20:43:34.9394533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9394958Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9395334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9395729Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9396144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9396776Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9397226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9397747Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9397924Z 2025-08-26T20:43:34.9398041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9398427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9398832Z return mod(**inputs) 2025-08-26T20:43:34.9399228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9399702Z outputs = self.model( 2025-08-26T20:43:34.9400100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9400524Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9400907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9401285Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9401676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9402167Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9402593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9403015Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9403168Z 2025-08-26T20:43:34.9403284Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9403675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9404029Z return mod(**inputs) 2025-08-26T20:43:34.9404409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9404818Z outputs = self.model( 2025-08-26T20:43:34.9405201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9405620Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9405989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9406388Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9406806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9407293Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9407751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9408201Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9408387Z 2025-08-26T20:43:34.9408500Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9408887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9409240Z return mod(**inputs) 2025-08-26T20:43:34.9409625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9410079Z outputs = self.model( 2025-08-26T20:43:34.9410443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9410845Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9411224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9411609Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9412022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9413376Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9413780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9414228Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9414431Z 2025-08-26T20:43:34.9414544Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9414961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9415313Z return mod(**inputs) 2025-08-26T20:43:34.9415698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9416105Z outputs = self.model( 2025-08-26T20:43:34.9416479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9416870Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9417251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9417645Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9418031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9418442Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9418856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9419260Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9419403Z 2025-08-26T20:43:34.9419515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9419873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9420203Z return mod(**inputs) 2025-08-26T20:43:34.9420566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9420950Z outputs = self.model( 2025-08-26T20:43:34.9421305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9421692Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9422050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9422421Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9422835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9423267Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9423680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9424090Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9424243Z 2025-08-26T20:43:34.9424353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9424717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9425044Z return mod(**inputs) 2025-08-26T20:43:34.9425406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9425787Z outputs = self.model( 2025-08-26T20:43:34.9426152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9426530Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9426884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9427253Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9427662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9428074Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9428477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9428923Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9429135Z 2025-08-26T20:43:34.9429241Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9429606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9429934Z return mod(**inputs) 2025-08-26T20:43:34.9430283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9430665Z outputs = self.model( 2025-08-26T20:43:34.9431024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9431416Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9431760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9432126Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9432519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9432934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9433339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9433725Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9433872Z 2025-08-26T20:43:34.9433978Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9434342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9434673Z return mod(**inputs) 2025-08-26T20:43:34.9435050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9435451Z outputs = self.model( 2025-08-26T20:43:34.9435843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9436309Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9436696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9437077Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9437502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9437971Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9438157Z 2025-08-26T20:43:34.9438278Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9438657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9439013Z return mod(**inputs) 2025-08-26T20:43:34.9439475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9439929Z outputs = self.model( 2025-08-26T20:43:34.9440332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9440755Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9441106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9441480Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9441872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9442336Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9442713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9443057Z return self.act(input) 2025-08-26T20:43:34.9443176Z 2025-08-26T20:43:34.9443281Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9443660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9443990Z return mod(**inputs) 2025-08-26T20:43:34.9444349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9444734Z outputs = self.model( 2025-08-26T20:43:34.9445099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9445488Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9445849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9446246Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9446655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9447090Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9447239Z 2025-08-26T20:43:34.9447357Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9447739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9448069Z return mod(**inputs) 2025-08-26T20:43:34.9448453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9448858Z outputs = self.model( 2025-08-26T20:43:34.9449237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9449640Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9450016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9450404Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9450817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9451247Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9451399Z 2025-08-26T20:43:34.9451531Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9451919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9452281Z return mod(**inputs) 2025-08-26T20:43:34.9452660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9453055Z outputs = self.model( 2025-08-26T20:43:34.9453437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9453859Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9454234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9454616Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9455025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9455460Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9455896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9456348Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9456566Z 2025-08-26T20:43:34.9456684Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9457055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9457394Z return mod(**inputs) 2025-08-26T20:43:34.9457776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9458200Z outputs = self.model( 2025-08-26T20:43:34.9458577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9458993Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9459374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9459749Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9460135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9460549Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9460958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9461356Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9461495Z 2025-08-26T20:43:34.9461608Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9461971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9462310Z return mod(**inputs) 2025-08-26T20:43:34.9462673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9463062Z outputs = self.model( 2025-08-26T20:43:34.9463433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9463812Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9464166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9464532Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9464916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9465324Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9465734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9466187Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9466353Z 2025-08-26T20:43:34.9466466Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9466853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9467181Z return mod(**inputs) 2025-08-26T20:43:34.9467542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9467932Z outputs = self.model( 2025-08-26T20:43:34.9468315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9468724Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9469096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9469498Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9469910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9470347Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9470774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9471274Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9471485Z 2025-08-26T20:43:34.9471596Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9471973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9472302Z return mod(**inputs) 2025-08-26T20:43:34.9472654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9473066Z outputs = self.model( 2025-08-26T20:43:34.9473448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9473852Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9474218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9474604Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9475016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9475450Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9475881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9476299Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9476460Z 2025-08-26T20:43:34.9476571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9476958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9477322Z return mod(**inputs) 2025-08-26T20:43:34.9477716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9478125Z outputs = self.model( 2025-08-26T20:43:34.9478515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9478920Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9479295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9479766Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9480203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9480654Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9481137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9481603Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9481771Z 2025-08-26T20:43:34.9481883Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9482275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9482636Z return mod(**inputs) 2025-08-26T20:43:34.9483021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9483433Z outputs = self.model( 2025-08-26T20:43:34.9483818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9484238Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9484616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9485011Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9485459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9485897Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9486387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9486868Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9487061Z 2025-08-26T20:43:34.9487179Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9487560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9487936Z return mod(**inputs) 2025-08-26T20:43:34.9488322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9488729Z outputs = self.model( 2025-08-26T20:43:34.9489116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9489520Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9489891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9490258Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9490646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9491052Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9491460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9491858Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9491999Z 2025-08-26T20:43:34.9492110Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9492479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9492804Z return mod(**inputs) 2025-08-26T20:43:34.9493169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9493553Z outputs = self.model( 2025-08-26T20:43:34.9493913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9494302Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9494649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9495021Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9495445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9495908Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9496093Z 2025-08-26T20:43:34.9496462Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9496869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9497210Z return mod(**inputs) 2025-08-26T20:43:34.9497599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9498008Z outputs = self.model( 2025-08-26T20:43:34.9498372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9498786Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9499168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9499564Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9499969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9500441Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9500835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9501216Z return self.act(input) 2025-08-26T20:43:34.9501328Z 2025-08-26T20:43:34.9501440Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9501799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9502131Z return mod(**inputs) 2025-08-26T20:43:34.9502494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9502921Z outputs = self.model( 2025-08-26T20:43:34.9503310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9503714Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9504091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9504480Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9504906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9505295Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9505443Z 2025-08-26T20:43:34.9505548Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9505913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9506249Z return mod(**inputs) 2025-08-26T20:43:34.9506610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9507003Z outputs = self.model( 2025-08-26T20:43:34.9507389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9507799Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9508169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9508555Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9508968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9509407Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9509839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9510289Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9510502Z 2025-08-26T20:43:34.9510619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9511035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9511391Z return mod(**inputs) 2025-08-26T20:43:34.9511773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9512183Z outputs = self.model( 2025-08-26T20:43:34.9512558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9512964Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9513337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9513728Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9514130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9514567Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9515000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9515419Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9515591Z 2025-08-26T20:43:34.9515714Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9516105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9516488Z return mod(**inputs) 2025-08-26T20:43:34.9516881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9517318Z outputs = self.model( 2025-08-26T20:43:34.9517711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9518125Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9518510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9518908Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9519332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9519845Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9520298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9520772Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9520946Z 2025-08-26T20:43:34.9521067Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9521458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9521808Z return mod(**inputs) 2025-08-26T20:43:34.9522194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9522605Z outputs = self.model( 2025-08-26T20:43:34.9522989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9523402Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9523766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9524152Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9524562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9524999Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9525464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9525913Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9526108Z 2025-08-26T20:43:34.9526242Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9526610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9526949Z return mod(**inputs) 2025-08-26T20:43:34.9527304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9527687Z outputs = self.model( 2025-08-26T20:43:34.9528052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9528440Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9528781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9529145Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9529525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9529932Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9530339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9530757Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9530910Z 2025-08-26T20:43:34.9531015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9531379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9531713Z return mod(**inputs) 2025-08-26T20:43:34.9532088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9532462Z outputs = self.model( 2025-08-26T20:43:34.9532825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9533206Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9533557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9533922Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9534297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9534701Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9535096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9535501Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9535655Z 2025-08-26T20:43:34.9535759Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9536124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9536452Z return mod(**inputs) 2025-08-26T20:43:34.9536818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9537219Z outputs = self.model( 2025-08-26T20:43:34.9537593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9537984Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9538337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9538702Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9539090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9539508Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9539902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9540351Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9540529Z 2025-08-26T20:43:34.9540638Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9540990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9541310Z return mod(**inputs) 2025-08-26T20:43:34.9541659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9542033Z outputs = self.model( 2025-08-26T20:43:34.9542387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9542757Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9543103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9543460Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9543844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9544266Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9544680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9545078Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9545220Z 2025-08-26T20:43:34.9545335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9545702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9546051Z return mod(**inputs) 2025-08-26T20:43:34.9546418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9546807Z outputs = self.model( 2025-08-26T20:43:34.9547170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9547563Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9547913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9548280Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9548671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9549148Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9549338Z 2025-08-26T20:43:34.9549455Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9549825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9550155Z return mod(**inputs) 2025-08-26T20:43:34.9550518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9550911Z outputs = self.model( 2025-08-26T20:43:34.9551253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9551634Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9551977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9552334Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9552717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9553155Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9553575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9553946Z return self.act(input) 2025-08-26T20:43:34.9554065Z 2025-08-26T20:43:34.9554202Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9554580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9554910Z return mod(**inputs) 2025-08-26T20:43:34.9555272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9555659Z outputs = self.model( 2025-08-26T20:43:34.9556023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9556432Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9556809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9557216Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9557653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9558073Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9558228Z 2025-08-26T20:43:34.9558340Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9558750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9559101Z return mod(**inputs) 2025-08-26T20:43:34.9559551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9559968Z outputs = self.model( 2025-08-26T20:43:34.9560372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9560831Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9561220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9561619Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9562043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9562468Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9562614Z 2025-08-26T20:43:34.9562735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9563123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9563467Z return mod(**inputs) 2025-08-26T20:43:34.9563873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9564288Z outputs = self.model( 2025-08-26T20:43:34.9564678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9565098Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9565464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9565854Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9566273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9566722Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9567158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9567624Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9567806Z 2025-08-26T20:43:34.9567920Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9568336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9568702Z return mod(**inputs) 2025-08-26T20:43:34.9569105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9569517Z outputs = self.model( 2025-08-26T20:43:34.9569900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9570305Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9570653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9571015Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9571400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9571810Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9572217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9572599Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9572741Z 2025-08-26T20:43:34.9572849Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9573215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9573563Z return mod(**inputs) 2025-08-26T20:43:34.9573922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9574299Z outputs = self.model( 2025-08-26T20:43:34.9574664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9575062Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9575411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9575771Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9576157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9576566Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9576971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9577390Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9577552Z 2025-08-26T20:43:34.9577657Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9578023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9578366Z return mod(**inputs) 2025-08-26T20:43:34.9578748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9579151Z outputs = self.model( 2025-08-26T20:43:34.9579532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9579912Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9580273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9580662Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9581062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9581500Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9581907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9582360Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9582546Z 2025-08-26T20:43:34.9582676Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9583040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9583399Z return mod(**inputs) 2025-08-26T20:43:34.9583784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9584192Z outputs = self.model( 2025-08-26T20:43:34.9584573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9584974Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9585350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9585718Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9586109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9586529Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9586962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9587384Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9587534Z 2025-08-26T20:43:34.9587652Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9588057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9588401Z return mod(**inputs) 2025-08-26T20:43:34.9588780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9589184Z outputs = self.model( 2025-08-26T20:43:34.9589569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9590000Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9590370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9590761Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9591169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9591608Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9592031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9592464Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9592631Z 2025-08-26T20:43:34.9592742Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9593127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9593479Z return mod(**inputs) 2025-08-26T20:43:34.9593862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9594280Z outputs = self.model( 2025-08-26T20:43:34.9594670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9595088Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9595474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9595865Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9596455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9596931Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9597373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9597919Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9598126Z 2025-08-26T20:43:34.9598242Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9598669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9599036Z return mod(**inputs) 2025-08-26T20:43:34.9599487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9599904Z outputs = self.model( 2025-08-26T20:43:34.9600303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9600734Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9601107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9601511Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9601915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9602350Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9602791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9603257Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9603402Z 2025-08-26T20:43:34.9603513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9603905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9604254Z return mod(**inputs) 2025-08-26T20:43:34.9604632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9605087Z outputs = self.model( 2025-08-26T20:43:34.9605484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9605913Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9606296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9606702Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9607114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9607648Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9607845Z 2025-08-26T20:43:34.9607959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9608354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9608731Z return mod(**inputs) 2025-08-26T20:43:34.9609120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9609547Z outputs = self.model( 2025-08-26T20:43:34.9609955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9610378Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9610759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9611163Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9611584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9612062Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9612486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9612857Z return self.act(input) 2025-08-26T20:43:34.9612985Z 2025-08-26T20:43:34.9613119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9613516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9613875Z return mod(**inputs) 2025-08-26T20:43:34.9614281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9614691Z outputs = self.model( 2025-08-26T20:43:34.9615080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9615510Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9615890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9616295Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9616710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9617122Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9617259Z 2025-08-26T20:43:34.9617373Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9617753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9618096Z return mod(**inputs) 2025-08-26T20:43:34.9618509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9618934Z outputs = self.model( 2025-08-26T20:43:34.9619326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9619758Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9620125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9620552Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9620966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9621377Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9621780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9622202Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9622376Z 2025-08-26T20:43:34.9622482Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9622845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9623171Z return mod(**inputs) 2025-08-26T20:43:34.9623520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9623907Z outputs = self.model( 2025-08-26T20:43:34.9624269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9624651Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9625004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9625362Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9625753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9626161Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9626581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9626991Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9627146Z 2025-08-26T20:43:34.9627257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9627662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9628014Z return mod(**inputs) 2025-08-26T20:43:34.9628423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9628799Z outputs = self.model( 2025-08-26T20:43:34.9629161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9629543Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9629892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9630275Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9630676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9631109Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9631539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9631990Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9632170Z 2025-08-26T20:43:34.9632275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9632640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9633008Z return mod(**inputs) 2025-08-26T20:43:34.9633389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9633800Z outputs = self.model( 2025-08-26T20:43:34.9634176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9634634Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9635006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9635393Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9635799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9636241Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9636669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9637149Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9637351Z 2025-08-26T20:43:34.9637471Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9637856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9638219Z return mod(**inputs) 2025-08-26T20:43:34.9638611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9639040Z outputs = self.model( 2025-08-26T20:43:34.9639507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9639933Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9640324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9640729Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9641154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9641614Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9642042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9642472Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9642632Z 2025-08-26T20:43:34.9642771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9643163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9643536Z return mod(**inputs) 2025-08-26T20:43:34.9643932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9644352Z outputs = self.model( 2025-08-26T20:43:34.9644792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9645230Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9645612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9646018Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9646425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9646853Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9647269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9647693Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9647854Z 2025-08-26T20:43:34.9647975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9648337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9648670Z return mod(**inputs) 2025-08-26T20:43:34.9649024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9649407Z outputs = self.model( 2025-08-26T20:43:34.9649791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9650176Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9650525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9650887Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9651276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9651688Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9652093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9652530Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9652718Z 2025-08-26T20:43:34.9652823Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9653186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9653515Z return mod(**inputs) 2025-08-26T20:43:34.9653875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9654249Z outputs = self.model( 2025-08-26T20:43:34.9654611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9654995Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9655348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9655716Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9656098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9656504Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9656915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9657327Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9657468Z 2025-08-26T20:43:34.9657576Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9657960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9658297Z return mod(**inputs) 2025-08-26T20:43:34.9658666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9659054Z outputs = self.model( 2025-08-26T20:43:34.9659424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9659810Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9660156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9660520Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9660898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9661329Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9661511Z 2025-08-26T20:43:34.9661615Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9661987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9662309Z return mod(**inputs) 2025-08-26T20:43:34.9662651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9663028Z outputs = self.model( 2025-08-26T20:43:34.9663381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9663771Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9664114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9664463Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9664839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9665255Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9665634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9665966Z return self.act(input) 2025-08-26T20:43:34.9666084Z 2025-08-26T20:43:34.9666185Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9666534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9666855Z return mod(**inputs) 2025-08-26T20:43:34.9667204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9667580Z outputs = self.model( 2025-08-26T20:43:34.9667942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9668325Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9668678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9669042Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9669425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9669810Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9669957Z 2025-08-26T20:43:34.9670062Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9670428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9670749Z return mod(**inputs) 2025-08-26T20:43:34.9671128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9671511Z outputs = self.model( 2025-08-26T20:43:34.9671889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9672282Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9672628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9672999Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9673383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9673776Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9673918Z 2025-08-26T20:43:34.9674029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9674389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9674718Z return mod(**inputs) 2025-08-26T20:43:34.9675107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9675508Z outputs = self.model( 2025-08-26T20:43:34.9675884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9676331Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9676702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9677097Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9677506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9677958Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9678392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9678848Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9679025Z 2025-08-26T20:43:34.9679147Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9679609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9679982Z return mod(**inputs) 2025-08-26T20:43:34.9680381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9680811Z outputs = self.model( 2025-08-26T20:43:34.9681196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9681277Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9681517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9681613Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9681884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9681995Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9682242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9682325Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9682336Z 2025-08-26T20:43:34.9682442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9682641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9682720Z return mod(**inputs) 2025-08-26T20:43:34.9682989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9683065Z outputs = self.model( 2025-08-26T20:43:34.9683329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9683405Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9683635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9683715Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9683967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9684066Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9684315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9684438Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9684442Z 2025-08-26T20:43:34.9684547Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9684753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9684822Z return mod(**inputs) 2025-08-26T20:43:34.9685075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9685160Z outputs = self.model( 2025-08-26T20:43:34.9685408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9685489Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9685711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9685812Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9686082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9686183Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9686440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9686576Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9686582Z 2025-08-26T20:43:34.9686694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9686898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9686973Z return mod(**inputs) 2025-08-26T20:43:34.9687257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9687331Z outputs = self.model( 2025-08-26T20:43:34.9687602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9687682Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9687932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9688011Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9688258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9688367Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9688616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9688712Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9688716Z 2025-08-26T20:43:34.9688819Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9689021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9689114Z return mod(**inputs) 2025-08-26T20:43:34.9689365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9689456Z outputs = self.model( 2025-08-26T20:43:34.9689708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9689790Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9690013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9690091Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9690345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9690446Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9690703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9690799Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9690803Z 2025-08-26T20:43:34.9690907Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9691114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9691200Z return mod(**inputs) 2025-08-26T20:43:34.9691455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9691525Z outputs = self.model( 2025-08-26T20:43:34.9691771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9691851Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9692091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9692181Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9692427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9692534Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9692782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9692913Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9692917Z 2025-08-26T20:43:34.9693029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9693230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9693304Z return mod(**inputs) 2025-08-26T20:43:34.9693553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9693624Z outputs = self.model( 2025-08-26T20:43:34.9693881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9693953Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9694188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9694269Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9694524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9694623Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9694879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9694966Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9694971Z 2025-08-26T20:43:34.9695072Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9695292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9695360Z return mod(**inputs) 2025-08-26T20:43:34.9695617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9695692Z outputs = self.model( 2025-08-26T20:43:34.9695934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9696012Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9696342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9696426Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9696675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9696798Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9696803Z 2025-08-26T20:43:34.9696912Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9697108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9697182Z return mod(**inputs) 2025-08-26T20:43:34.9697428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9697567Z outputs = self.model( 2025-08-26T20:43:34.9697829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9697902Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9698134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9698240Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9698489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9698617Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9698831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9698907Z return self.act(input) 2025-08-26T20:43:34.9698912Z 2025-08-26T20:43:34.9699015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9699228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9699299Z return mod(**inputs) 2025-08-26T20:43:34.9699561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9699644Z outputs = self.model( 2025-08-26T20:43:34.9699907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9699996Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9700230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9700313Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9700587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9700673Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9700676Z 2025-08-26T20:43:34.9700786Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9700984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9701051Z return mod(**inputs) 2025-08-26T20:43:34.9701305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9701376Z outputs = self.model( 2025-08-26T20:43:34.9701658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9701738Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9702019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9702104Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9702375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9702483Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9702733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9702854Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9702859Z 2025-08-26T20:43:34.9702965Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9703169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9703245Z return mod(**inputs) 2025-08-26T20:43:34.9703496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9703570Z outputs = self.model( 2025-08-26T20:43:34.9703833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9703906Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9704140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9704224Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9704490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9704615Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9704882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9704969Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9704973Z 2025-08-26T20:43:34.9705084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9705303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9705373Z return mod(**inputs) 2025-08-26T20:43:34.9705640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9705713Z outputs = self.model( 2025-08-26T20:43:34.9705971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9706066Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9706289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9706376Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9706624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9706730Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9706978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9707088Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9707092Z 2025-08-26T20:43:34.9707205Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9707403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9707478Z return mod(**inputs) 2025-08-26T20:43:34.9707743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9707813Z outputs = self.model( 2025-08-26T20:43:34.9708085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9708162Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9708392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9708473Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9708719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9708824Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9709070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9709216Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9709221Z 2025-08-26T20:43:34.9709326Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9709534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9709601Z return mod(**inputs) 2025-08-26T20:43:34.9709850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9709941Z outputs = self.model( 2025-08-26T20:43:34.9710191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9710272Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9710495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9710593Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9710848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9710945Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9711204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9711288Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9711293Z 2025-08-26T20:43:34.9711398Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9711590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9711655Z return mod(**inputs) 2025-08-26T20:43:34.9711905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9711974Z outputs = self.model( 2025-08-26T20:43:34.9712227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9712298Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9712522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9712608Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9712852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9712958Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9713201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9713296Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9713306Z 2025-08-26T20:43:34.9713408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9713610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9713705Z return mod(**inputs) 2025-08-26T20:43:34.9713969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9714076Z outputs = self.model( 2025-08-26T20:43:34.9714340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9714418Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9714661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9714745Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9715011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9715115Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9715379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9715523Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9715527Z 2025-08-26T20:43:34.9715639Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9715857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9715948Z return mod(**inputs) 2025-08-26T20:43:34.9716224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9716297Z outputs = self.model( 2025-08-26T20:43:34.9716565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9716651Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9716903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9716997Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9717257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9717364Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9717640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9717732Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9717735Z 2025-08-26T20:43:34.9717857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9718097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9718170Z return mod(**inputs) 2025-08-26T20:43:34.9718448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9718523Z outputs = self.model( 2025-08-26T20:43:34.9718803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9718881Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9719131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9719219Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9719551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9719697Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9719701Z 2025-08-26T20:43:34.9719811Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9720035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9720109Z return mod(**inputs) 2025-08-26T20:43:34.9720400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9720481Z outputs = self.model( 2025-08-26T20:43:34.9720767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9720860Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9721108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9721205Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9721471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9721591Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9721819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9721890Z return self.act(input) 2025-08-26T20:43:34.9721895Z 2025-08-26T20:43:34.9722008Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9722209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9722275Z return mod(**inputs) 2025-08-26T20:43:34.9722532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9722620Z outputs = self.model( 2025-08-26T20:43:34.9722879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9722952Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9723177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9723284Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9723535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9723625Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9723629Z 2025-08-26T20:43:34.9723735Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9723943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9724010Z return mod(**inputs) 2025-08-26T20:43:34.9724262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9724340Z outputs = self.model( 2025-08-26T20:43:34.9724591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9724671Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9724898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9724979Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9725241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9725323Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9725327Z 2025-08-26T20:43:34.9725438Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9725643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9725710Z return mod(**inputs) 2025-08-26T20:43:34.9725971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9726039Z outputs = self.model( 2025-08-26T20:43:34.9726299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9726376Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9726632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9726714Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9726978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9727088Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9727337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9727456Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9727459Z 2025-08-26T20:43:34.9727561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9727760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9727836Z return mod(**inputs) 2025-08-26T20:43:34.9728080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9728156Z outputs = self.model( 2025-08-26T20:43:34.9728403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9728483Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9728721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9728799Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9729056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9729154Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9729427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9729510Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9729514Z 2025-08-26T20:43:34.9729616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9729822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9729887Z return mod(**inputs) 2025-08-26T20:43:34.9730143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9730213Z outputs = self.model( 2025-08-26T20:43:34.9730503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9730581Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9730800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9730885Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9731130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9731232Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9731475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9731583Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9731588Z 2025-08-26T20:43:34.9731696Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9731891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9731965Z return mod(**inputs) 2025-08-26T20:43:34.9732206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9732275Z outputs = self.model( 2025-08-26T20:43:34.9732539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9732612Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9732853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9732932Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9733177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9733284Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9733530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9733674Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9733680Z 2025-08-26T20:43:34.9733785Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9733992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9734060Z return mod(**inputs) 2025-08-26T20:43:34.9734312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9734388Z outputs = self.model( 2025-08-26T20:43:34.9734639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9734735Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9734964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9735049Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9735297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9735429Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9735675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9735762Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9735765Z 2025-08-26T20:43:34.9735874Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9736067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9736134Z return mod(**inputs) 2025-08-26T20:43:34.9736385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9736453Z outputs = self.model( 2025-08-26T20:43:34.9736706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9736780Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9736998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9737086Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9737332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9737436Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9737685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9737789Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9737793Z 2025-08-26T20:43:34.9737894Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9738097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9738175Z return mod(**inputs) 2025-08-26T20:43:34.9738439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9738539Z outputs = self.model( 2025-08-26T20:43:34.9738806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9738894Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9739126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9739208Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9739466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9739568Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9739832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9739975Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9739979Z 2025-08-26T20:43:34.9740098Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9740303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9740367Z return mod(**inputs) 2025-08-26T20:43:34.9740615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9740696Z outputs = self.model( 2025-08-26T20:43:34.9740949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9741025Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9741240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9741322Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9741568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9741660Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9741899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9741976Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9741980Z 2025-08-26T20:43:34.9742085Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9742271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9742338Z return mod(**inputs) 2025-08-26T20:43:34.9742572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9742637Z outputs = self.model( 2025-08-26T20:43:34.9742883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9742956Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9743176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9743252Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9743490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9743615Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9743618Z 2025-08-26T20:43:34.9743717Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9743922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9743988Z return mod(**inputs) 2025-08-26T20:43:34.9744232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9744308Z outputs = self.model( 2025-08-26T20:43:34.9744571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9744653Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9744884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9744969Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9745209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9745323Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9745541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9745609Z return self.act(input) 2025-08-26T20:43:34.9745615Z 2025-08-26T20:43:34.9745721Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9745915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9745981Z return mod(**inputs) 2025-08-26T20:43:34.9746229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9746295Z outputs = self.model( 2025-08-26T20:43:34.9746543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9746630Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9746849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9746936Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9747182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9747292Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9747295Z 2025-08-26T20:43:34.9747398Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9747595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9747659Z return mod(**inputs) 2025-08-26T20:43:34.9747900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9747977Z outputs = self.model( 2025-08-26T20:43:34.9748222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9748302Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9748521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9748600Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9748868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9748974Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9749243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9749372Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9749375Z 2025-08-26T20:43:34.9749485Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9749683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9749749Z return mod(**inputs) 2025-08-26T20:43:34.9750003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9750071Z outputs = self.model( 2025-08-26T20:43:34.9750324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9750395Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9750634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9750738Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9750987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9751094Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9751343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9751425Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9751436Z 2025-08-26T20:43:34.9751540Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9751738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9751813Z return mod(**inputs) 2025-08-26T20:43:34.9752062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9752138Z outputs = self.model( 2025-08-26T20:43:34.9752383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9752455Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9752703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9752783Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9753041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9753140Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9753405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9753524Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9753527Z 2025-08-26T20:43:34.9753629Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9753836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9753904Z return mod(**inputs) 2025-08-26T20:43:34.9754155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9754228Z outputs = self.model( 2025-08-26T20:43:34.9754489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9754573Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9754807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9754899Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9755161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9755263Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9755535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9755680Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9755684Z 2025-08-26T20:43:34.9755799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9756014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9756084Z return mod(**inputs) 2025-08-26T20:43:34.9756358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9756434Z outputs = self.model( 2025-08-26T20:43:34.9756724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9756804Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9757069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9757157Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9757430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9757548Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9757819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9757923Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9757928Z 2025-08-26T20:43:34.9758040Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9758260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9758339Z return mod(**inputs) 2025-08-26T20:43:34.9758607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9758687Z outputs = self.model( 2025-08-26T20:43:34.9758958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9759074Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9759322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9759489Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9759776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9759907Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9760186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9760292Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9760298Z 2025-08-26T20:43:34.9760411Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9760641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9760717Z return mod(**inputs) 2025-08-26T20:43:34.9760993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9761070Z outputs = self.model( 2025-08-26T20:43:34.9761335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9761418Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9761642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9761731Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9761982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9762088Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9762335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9762466Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9762470Z 2025-08-26T20:43:34.9762581Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9762781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9762857Z return mod(**inputs) 2025-08-26T20:43:34.9763104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9763187Z outputs = self.model( 2025-08-26T20:43:34.9763461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9763539Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9763782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9763868Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9764131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9764241Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9764507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9764605Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9764609Z 2025-08-26T20:43:34.9764723Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9764936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9765005Z return mod(**inputs) 2025-08-26T20:43:34.9765252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9765345Z outputs = self.model( 2025-08-26T20:43:34.9765594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9765673Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9765899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9765978Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9766252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9766373Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9766376Z 2025-08-26T20:43:34.9766486Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9766692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9766770Z return mod(**inputs) 2025-08-26T20:43:34.9767035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9767107Z outputs = self.model( 2025-08-26T20:43:34.9767376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9767455Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9767702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9767786Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9768050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9768186Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9768416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9768498Z return self.act(input) 2025-08-26T20:43:34.9768501Z 2025-08-26T20:43:34.9769010Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9769242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9769308Z return mod(**inputs) 2025-08-26T20:43:34.9769557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9769635Z outputs = self.model( 2025-08-26T20:43:34.9769899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9769981Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9770218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9770296Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9770553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9770635Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9770638Z 2025-08-26T20:43:34.9770745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9770946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9771013Z return mod(**inputs) 2025-08-26T20:43:34.9771268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9771339Z outputs = self.model( 2025-08-26T20:43:34.9771595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9771668Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9771894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9771995Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9772238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9772326Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9772330Z 2025-08-26T20:43:34.9772432Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9772656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9772721Z return mod(**inputs) 2025-08-26T20:43:34.9772971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9773048Z outputs = self.model( 2025-08-26T20:43:34.9773300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9773381Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9773612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9773694Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9773967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9774071Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9774346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9774466Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9774470Z 2025-08-26T20:43:34.9774586Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9774803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9774874Z return mod(**inputs) 2025-08-26T20:43:34.9775152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9775225Z outputs = self.model( 2025-08-26T20:43:34.9775493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9775569Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9775812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9775904Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9776186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9776322Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9776573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9776663Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9776667Z 2025-08-26T20:43:34.9776770Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9776970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9777045Z return mod(**inputs) 2025-08-26T20:43:34.9777288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9777364Z outputs = self.model( 2025-08-26T20:43:34.9777613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9777686Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9777917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9777995Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9778267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9778365Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9778612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9778728Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9778748Z 2025-08-26T20:43:34.9778851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9779059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9779125Z return mod(**inputs) 2025-08-26T20:43:34.9779379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9779448Z outputs = self.model( 2025-08-26T20:43:34.9779695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9779777Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9780000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9780087Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9780331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9780431Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9780690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9780829Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9780833Z 2025-08-26T20:43:34.9780942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9781144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9781218Z return mod(**inputs) 2025-08-26T20:43:34.9781463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9781531Z outputs = self.model( 2025-08-26T20:43:34.9781789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9781870Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9782125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9782211Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9782490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9782603Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9782867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9782969Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9782973Z 2025-08-26T20:43:34.9783081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9783299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9783372Z return mod(**inputs) 2025-08-26T20:43:34.9783635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9783725Z outputs = self.model( 2025-08-26T20:43:34.9783974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9784054Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9784277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9784377Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9784650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9784756Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9785027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9785145Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9785149Z 2025-08-26T20:43:34.9785257Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9785475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9785546Z return mod(**inputs) 2025-08-26T20:43:34.9785813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9785888Z outputs = self.model( 2025-08-26T20:43:34.9786158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9786234Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9786471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9786564Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9786827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9786938Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9787200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9787335Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9787340Z 2025-08-26T20:43:34.9787454Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9787668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9787743Z return mod(**inputs) 2025-08-26T20:43:34.9788028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9788109Z outputs = self.model( 2025-08-26T20:43:34.9788395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9788487Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9788731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9788830Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9789121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9789226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9789510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9789607Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9789611Z 2025-08-26T20:43:34.9789719Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9789938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9790008Z return mod(**inputs) 2025-08-26T20:43:34.9790292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9790373Z outputs = self.model( 2025-08-26T20:43:34.9790635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9790736Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9790990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9791083Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9791367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9791511Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9791515Z 2025-08-26T20:43:34.9791631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9791846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9791924Z return mod(**inputs) 2025-08-26T20:43:34.9792214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9792286Z outputs = self.model( 2025-08-26T20:43:34.9792587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9792663Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9792921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9793005Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9793269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9793410Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9793640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9793725Z return self.act(input) 2025-08-26T20:43:34.9793728Z 2025-08-26T20:43:34.9793837Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9794056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9794127Z return mod(**inputs) 2025-08-26T20:43:34.9794412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9794492Z outputs = self.model( 2025-08-26T20:43:34.9794779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9794866Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9795130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9795217Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9795537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9795625Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9795631Z 2025-08-26T20:43:34.9795746Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9795956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9796033Z return mod(**inputs) 2025-08-26T20:43:34.9796437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9796517Z outputs = self.model( 2025-08-26T20:43:34.9796810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9796890Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9797135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9797221Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9797524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9797713Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9798027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9798151Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9798155Z 2025-08-26T20:43:34.9798265Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9798507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9798588Z return mod(**inputs) 2025-08-26T20:43:34.9798852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9798936Z outputs = self.model( 2025-08-26T20:43:34.9799198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9799284Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9799565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9799653Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9799923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9800031Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9800301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9800384Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9800388Z 2025-08-26T20:43:34.9800498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9800717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9800789Z return mod(**inputs) 2025-08-26T20:43:34.9801056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9801129Z outputs = self.model( 2025-08-26T20:43:34.9801388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9801470Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9801707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9801832Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9802097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9802236Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9802499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9802619Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9802622Z 2025-08-26T20:43:34.9802741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9802954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9803031Z return mod(**inputs) 2025-08-26T20:43:34.9803291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9803368Z outputs = self.model( 2025-08-26T20:43:34.9803636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9803715Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9803957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9804041Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9804327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9804439Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9804687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9804833Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9804851Z 2025-08-26T20:43:34.9804957Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9805164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9805231Z return mod(**inputs) 2025-08-26T20:43:34.9805480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9805557Z outputs = self.model( 2025-08-26T20:43:34.9805804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9805885Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9806108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9806187Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9806443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9806545Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9806801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9806890Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9806893Z 2025-08-26T20:43:34.9807004Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9807208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9807274Z return mod(**inputs) 2025-08-26T20:43:34.9807541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9807614Z outputs = self.model( 2025-08-26T20:43:34.9807881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9807958Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9808210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9808300Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9808593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9808705Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9808966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9809073Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9809076Z 2025-08-26T20:43:34.9809183Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9809405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9809480Z return mod(**inputs) 2025-08-26T20:43:34.9809726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9809802Z outputs = self.model( 2025-08-26T20:43:34.9810049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9810122Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9810348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9810447Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9810701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9810799Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9811044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9811196Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9811201Z 2025-08-26T20:43:34.9811304Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9811514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9811580Z return mod(**inputs) 2025-08-26T20:43:34.9811836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9811906Z outputs = self.model( 2025-08-26T20:43:34.9812152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9812232Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9812456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9812545Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9812792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9812890Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9813146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9813227Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9813232Z 2025-08-26T20:43:34.9813343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9813541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9813613Z return mod(**inputs) 2025-08-26T20:43:34.9813859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9813929Z outputs = self.model( 2025-08-26T20:43:34.9814185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9814273Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9814523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9814604Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9814852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9814981Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9814985Z 2025-08-26T20:43:34.9815085Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9815290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9815355Z return mod(**inputs) 2025-08-26T20:43:34.9815614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9815685Z outputs = self.model( 2025-08-26T20:43:34.9815933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9816013Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9816238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9816340Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9816586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9816703Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9816926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9817012Z return self.act(input) 2025-08-26T20:43:34.9817015Z 2025-08-26T20:43:34.9817129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9817332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9817397Z return mod(**inputs) 2025-08-26T20:43:34.9817655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9817722Z outputs = self.model( 2025-08-26T20:43:34.9817977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9818051Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9818279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9818360Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9818607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9818697Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9818702Z 2025-08-26T20:43:34.9818806Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9819016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9819082Z return mod(**inputs) 2025-08-26T20:43:34.9819329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9819406Z outputs = self.model( 2025-08-26T20:43:34.9819651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9819731Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9819949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9820031Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9820310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9820393Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9820396Z 2025-08-26T20:43:34.9820522Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9820724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9820800Z return mod(**inputs) 2025-08-26T20:43:34.9821048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9821116Z outputs = self.model( 2025-08-26T20:43:34.9821372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9821445Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9821679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9821761Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9822009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9822117Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9822365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9822498Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9822502Z 2025-08-26T20:43:34.9822604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9822810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9822875Z return mod(**inputs) 2025-08-26T20:43:34.9823139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9823217Z outputs = self.model( 2025-08-26T20:43:34.9823461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9823539Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9823758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9823837Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9824095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9824193Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9824437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9824517Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9824520Z 2025-08-26T20:43:34.9824621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9824829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9824894Z return mod(**inputs) 2025-08-26T20:43:34.9825153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9825221Z outputs = self.model( 2025-08-26T20:43:34.9825476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9825549Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9825767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9825852Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9826112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9826242Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9826508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9826639Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9826643Z 2025-08-26T20:43:34.9826761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9826977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9827058Z return mod(**inputs) 2025-08-26T20:43:34.9827324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9827398Z outputs = self.model( 2025-08-26T20:43:34.9827665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9827744Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9827992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9828071Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9828325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9828423Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9828687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9828832Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9828835Z 2025-08-26T20:43:34.9828937Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9829144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9829227Z return mod(**inputs) 2025-08-26T20:43:34.9829477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9829553Z outputs = self.model( 2025-08-26T20:43:34.9829800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9829881Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9830117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9830207Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9830475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9830573Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9830828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9830917Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9830921Z 2025-08-26T20:43:34.9831028Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9831228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9831294Z return mod(**inputs) 2025-08-26T20:43:34.9831548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9831618Z outputs = self.model( 2025-08-26T20:43:34.9831873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9831945Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9832163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9832250Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9832519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9832628Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9832893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9832997Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9833002Z 2025-08-26T20:43:34.9833104Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9833304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9833377Z return mod(**inputs) 2025-08-26T20:43:34.9833644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9833725Z outputs = self.model( 2025-08-26T20:43:34.9834028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9834104Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9834349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9834433Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9834707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9834826Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9835103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9835237Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9835257Z 2025-08-26T20:43:34.9835368Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9835588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9835659Z return mod(**inputs) 2025-08-26T20:43:34.9835937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9836010Z outputs = self.model( 2025-08-26T20:43:34.9836279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9836366Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9836602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9836693Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9836962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9837068Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9837346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9837432Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9837436Z 2025-08-26T20:43:34.9837553Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9837763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9837844Z return mod(**inputs) 2025-08-26T20:43:34.9838119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9838194Z outputs = self.model( 2025-08-26T20:43:34.9838490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9838570Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9838820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9838923Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9839212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9839411Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9839418Z 2025-08-26T20:43:34.9839542Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9839779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9839849Z return mod(**inputs) 2025-08-26T20:43:34.9840122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9840194Z outputs = self.model( 2025-08-26T20:43:34.9840461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9840547Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9840786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9840878Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9841143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9841287Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9841510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9841581Z return self.act(input) 2025-08-26T20:43:34.9841584Z 2025-08-26T20:43:34.9841695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9841895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9841989Z return mod(**inputs) 2025-08-26T20:43:34.9842247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9842316Z outputs = self.model( 2025-08-26T20:43:34.9842570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9842644Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9842874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9842951Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9843196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9843285Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9843290Z 2025-08-26T20:43:34.9843391Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9843596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9843661Z return mod(**inputs) 2025-08-26T20:43:34.9843909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9843985Z outputs = self.model( 2025-08-26T20:43:34.9844229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9844310Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9844532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9844610Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9844861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9844961Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9845230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9845344Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9845347Z 2025-08-26T20:43:34.9845470Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9845671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9845739Z return mod(**inputs) 2025-08-26T20:43:34.9845994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9846062Z outputs = self.model( 2025-08-26T20:43:34.9846315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9846389Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9846610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9846698Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9846947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9847053Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9847302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9847407Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9847411Z 2025-08-26T20:43:34.9847515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9847715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9847789Z return mod(**inputs) 2025-08-26T20:43:34.9848053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9848130Z outputs = self.model( 2025-08-26T20:43:34.9848377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9848452Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9848679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9848761Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9849014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9849114Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9849361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9849479Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9849483Z 2025-08-26T20:43:34.9849588Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9849795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9849863Z return mod(**inputs) 2025-08-26T20:43:34.9850122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9850194Z outputs = self.model( 2025-08-26T20:43:34.9850454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9850540Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9850779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9850874Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9851137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9851267Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9851558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9851704Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9851708Z 2025-08-26T20:43:34.9851828Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9852038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9852115Z return mod(**inputs) 2025-08-26T20:43:34.9852374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9852446Z outputs = self.model( 2025-08-26T20:43:34.9852717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9852800Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9853045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9853129Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9853392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9853523Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9853784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9853882Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9853886Z 2025-08-26T20:43:34.9853995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9854229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9854299Z return mod(**inputs) 2025-08-26T20:43:34.9854568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9854650Z outputs = self.model( 2025-08-26T20:43:34.9854916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9854999Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9855244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9855326Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9855602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9855706Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9855983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9856088Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9856091Z 2025-08-26T20:43:34.9856201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9856425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9856495Z return mod(**inputs) 2025-08-26T20:43:34.9856772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9856844Z outputs = self.model( 2025-08-26T20:43:34.9857118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9857194Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9857432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9857527Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9857812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9857924Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9858221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9858361Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9858365Z 2025-08-26T20:43:34.9858482Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9858693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9858771Z return mod(**inputs) 2025-08-26T20:43:34.9859029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9859104Z outputs = self.model( 2025-08-26T20:43:34.9859371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9859448Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9859691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9859773Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9860057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9860169Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9860413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9860501Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9860522Z 2025-08-26T20:43:34.9860627Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9860836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9860904Z return mod(**inputs) 2025-08-26T20:43:34.9861165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9861244Z outputs = self.model( 2025-08-26T20:43:34.9861505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9861590Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9861824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9861915Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9862172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9862299Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9862303Z 2025-08-26T20:43:34.9862420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9862632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9862712Z return mod(**inputs) 2025-08-26T20:43:34.9862972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9863046Z outputs = self.model( 2025-08-26T20:43:34.9863315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9863393Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9863636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9863723Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9864000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9864136Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9864382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9864466Z return self.act(input) 2025-08-26T20:43:34.9864469Z 2025-08-26T20:43:34.9864585Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9864805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9864872Z return mod(**inputs) 2025-08-26T20:43:34.9865122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9865200Z outputs = self.model( 2025-08-26T20:43:34.9865452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9865533Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9865758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9865839Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9866092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9866209Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9866212Z 2025-08-26T20:43:34.9866324Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9866526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9866592Z return mod(**inputs) 2025-08-26T20:43:34.9866851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9866942Z outputs = self.model( 2025-08-26T20:43:34.9867205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9883199Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9883706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9883808Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9884118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9884211Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9884218Z 2025-08-26T20:43:34.9884347Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9884575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9884666Z return mod(**inputs) 2025-08-26T20:43:34.9884943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9885029Z outputs = self.model( 2025-08-26T20:43:34.9885305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9885391Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9885641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9885732Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9885993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9886114Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9886375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9886509Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9886515Z 2025-08-26T20:43:34.9886720Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9886946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9887066Z return mod(**inputs) 2025-08-26T20:43:34.9887335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9887422Z outputs = self.model( 2025-08-26T20:43:34.9887686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9887772Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9888011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9888095Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9888356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9888459Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9888719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9888802Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9888806Z 2025-08-26T20:43:34.9888940Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9889155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9889221Z return mod(**inputs) 2025-08-26T20:43:34.9889476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9889546Z outputs = self.model( 2025-08-26T20:43:34.9889821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9889905Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9890130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9890218Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9890479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9890591Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9890853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9890973Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9890977Z 2025-08-26T20:43:34.9891096Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9891315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9891394Z return mod(**inputs) 2025-08-26T20:43:34.9891657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9891730Z outputs = self.model( 2025-08-26T20:43:34.9891998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9892079Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9892319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9892403Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9892670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9892774Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9893035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9893206Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9893211Z 2025-08-26T20:43:34.9893321Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9893573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9893646Z return mod(**inputs) 2025-08-26T20:43:34.9893909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9893990Z outputs = self.model( 2025-08-26T20:43:34.9894261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9894341Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9894571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9894653Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9894915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9895022Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9895293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9895404Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9895408Z 2025-08-26T20:43:34.9895525Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9895740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9895810Z return mod(**inputs) 2025-08-26T20:43:34.9896081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9896292Z outputs = self.model( 2025-08-26T20:43:34.9896580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9896657Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9896896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9896992Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9897256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9897367Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9897629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9897741Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9897747Z 2025-08-26T20:43:34.9897858Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9898072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9898152Z return mod(**inputs) 2025-08-26T20:43:34.9898417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9898500Z outputs = self.model( 2025-08-26T20:43:34.9898766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9898846Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9899092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9899177Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9899445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9899554Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9899902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9900086Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9900090Z 2025-08-26T20:43:34.9900213Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9900441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9900511Z return mod(**inputs) 2025-08-26T20:43:34.9900778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9900851Z outputs = self.model( 2025-08-26T20:43:34.9901112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9901200Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9901437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9901531Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9901792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9901897Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9902195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9902284Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9902288Z 2025-08-26T20:43:34.9902407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9902620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9902728Z return mod(**inputs) 2025-08-26T20:43:34.9903001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9903076Z outputs = self.model( 2025-08-26T20:43:34.9903356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9903445Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9903687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9903774Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9904040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9904185Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9904189Z 2025-08-26T20:43:34.9904301Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9904531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9904605Z return mod(**inputs) 2025-08-26T20:43:34.9904881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9904957Z outputs = self.model( 2025-08-26T20:43:34.9905227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9905315Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9905557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9905649Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9905916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9906050Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9906309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9906387Z return self.act(input) 2025-08-26T20:43:34.9906391Z 2025-08-26T20:43:34.9906508Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9906744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9906818Z return mod(**inputs) 2025-08-26T20:43:34.9907098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9907173Z outputs = self.model( 2025-08-26T20:43:34.9907450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9907530Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9907778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9907863Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9908133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9908233Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9908237Z 2025-08-26T20:43:34.9908351Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9908595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9908666Z return mod(**inputs) 2025-08-26T20:43:34.9908936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9909021Z outputs = self.model( 2025-08-26T20:43:34.9909287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9909391Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9909639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9909727Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9910006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9910114Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9910391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9910515Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9910519Z 2025-08-26T20:43:34.9910637Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9910854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9910928Z return mod(**inputs) 2025-08-26T20:43:34.9911208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9911281Z outputs = self.model( 2025-08-26T20:43:34.9911567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9911646Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9911893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9911990Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9912268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9912381Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9912650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9912741Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9912753Z 2025-08-26T20:43:34.9912880Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9913109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9913205Z return mod(**inputs) 2025-08-26T20:43:34.9913479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9913560Z outputs = self.model( 2025-08-26T20:43:34.9913829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9913908Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9914149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9914236Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9914517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9914621Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9914898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9915024Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9915046Z 2025-08-26T20:43:34.9915160Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9915385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9915456Z return mod(**inputs) 2025-08-26T20:43:34.9915732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9915825Z outputs = self.model( 2025-08-26T20:43:34.9916102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9916192Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9916431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9916524Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9916811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9916918Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9917205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9917355Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9917359Z 2025-08-26T20:43:34.9917478Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9917695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9917776Z return mod(**inputs) 2025-08-26T20:43:34.9918059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9918135Z outputs = self.model( 2025-08-26T20:43:34.9918423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9918504Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9918753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9918839Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9919114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9919232Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9919631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9919748Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9919752Z 2025-08-26T20:43:34.9919892Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9920112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9920194Z return mod(**inputs) 2025-08-26T20:43:34.9920474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9920556Z outputs = self.model( 2025-08-26T20:43:34.9920825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9920911Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9921146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9921232Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9921500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9921599Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9921853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9921968Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9921971Z 2025-08-26T20:43:34.9922074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9922284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9922349Z return mod(**inputs) 2025-08-26T20:43:34.9922601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9922687Z outputs = self.model( 2025-08-26T20:43:34.9922936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9923015Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9923239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9923327Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9923575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9923680Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9923940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9924076Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9924082Z 2025-08-26T20:43:34.9924197Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9924422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9924497Z return mod(**inputs) 2025-08-26T20:43:34.9924746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9924815Z outputs = self.model( 2025-08-26T20:43:34.9925073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9925147Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9925375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9925454Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9925709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9925824Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9926073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9926179Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9926183Z 2025-08-26T20:43:34.9926288Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9926494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9926562Z return mod(**inputs) 2025-08-26T20:43:34.9926808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9926884Z outputs = self.model( 2025-08-26T20:43:34.9927131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9927215Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9927439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9927517Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9927784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9927911Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9927931Z 2025-08-26T20:43:34.9928049Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9928268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9928341Z return mod(**inputs) 2025-08-26T20:43:34.9928586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9928674Z outputs = self.model( 2025-08-26T20:43:34.9928939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9929016Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9929260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9929345Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9929604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9929737Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9929966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9930048Z return self.act(input) 2025-08-26T20:43:34.9930052Z 2025-08-26T20:43:34.9930161Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9930380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9930451Z return mod(**inputs) 2025-08-26T20:43:34.9930711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9930791Z outputs = self.model( 2025-08-26T20:43:34.9931052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9931139Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9931372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9931456Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9931722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9931811Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9931815Z 2025-08-26T20:43:34.9931932Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9932158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9932230Z return mod(**inputs) 2025-08-26T20:43:34.9932532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9932606Z outputs = self.model( 2025-08-26T20:43:34.9932880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9932969Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9933204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9933295Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9933557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9933645Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9933651Z 2025-08-26T20:43:34.9933769Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9933984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9934062Z return mod(**inputs) 2025-08-26T20:43:34.9934323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9934414Z outputs = self.model( 2025-08-26T20:43:34.9934686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9934764Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9935008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9935112Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9935383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9935488Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9935753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9935879Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9935884Z 2025-08-26T20:43:34.9935993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9936211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9936282Z return mod(**inputs) 2025-08-26T20:43:34.9936543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9936627Z outputs = self.model( 2025-08-26T20:43:34.9936891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9936974Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9937210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9937294Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9937562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9937669Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9937940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9938025Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9938029Z 2025-08-26T20:43:34.9938145Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9938356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9938443Z return mod(**inputs) 2025-08-26T20:43:34.9938712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9938800Z outputs = self.model( 2025-08-26T20:43:34.9939069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9939146Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9939380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9939479Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9939724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9939831Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9940079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9940194Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9940198Z 2025-08-26T20:43:34.9940312Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9940505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9940593Z return mod(**inputs) 2025-08-26T20:43:34.9940835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9940911Z outputs = self.model( 2025-08-26T20:43:34.9941159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9941231Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9941488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9941569Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9941823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9941924Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9942174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9942319Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9942323Z 2025-08-26T20:43:34.9942427Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9942633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9942701Z return mod(**inputs) 2025-08-26T20:43:34.9942956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9943027Z outputs = self.model( 2025-08-26T20:43:34.9943275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9943358Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9943593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9943688Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9943951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9944062Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9944320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9944410Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9944415Z 2025-08-26T20:43:34.9944527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9944749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9944829Z return mod(**inputs) 2025-08-26T20:43:34.9945111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9945187Z outputs = self.model( 2025-08-26T20:43:34.9945462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9945539Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9945782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9945864Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9946129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9946242Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9946503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9946615Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9946619Z 2025-08-26T20:43:34.9946733Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9946953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9947019Z return mod(**inputs) 2025-08-26T20:43:34.9947257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9947334Z outputs = self.model( 2025-08-26T20:43:34.9947573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9947667Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9947886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9947966Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9948219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9948316Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9948568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9948704Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9948708Z 2025-08-26T20:43:34.9948816Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9949033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9949104Z return mod(**inputs) 2025-08-26T20:43:34.9949375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9949448Z outputs = self.model( 2025-08-26T20:43:34.9949717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9949793Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9950029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9950120Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9950378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9950489Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9950756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9950859Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9950862Z 2025-08-26T20:43:34.9950981Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9951211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9951291Z return mod(**inputs) 2025-08-26T20:43:34.9951559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9951636Z outputs = self.model( 2025-08-26T20:43:34.9951911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9951989Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9952233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9952319Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9952590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9952717Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9952720Z 2025-08-26T20:43:34.9952831Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9953050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9953136Z return mod(**inputs) 2025-08-26T20:43:34.9953411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9953484Z outputs = self.model( 2025-08-26T20:43:34.9953748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9953852Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9954088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9954179Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9954443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9954574Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9954802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9954875Z return self.act(input) 2025-08-26T20:43:34.9954879Z 2025-08-26T20:43:34.9954993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9955205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9955282Z return mod(**inputs) 2025-08-26T20:43:34.9955546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9955617Z outputs = self.model( 2025-08-26T20:43:34.9955886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9955964Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9956207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9956293Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9956552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9956647Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9956651Z 2025-08-26T20:43:34.9956758Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9956976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9957047Z return mod(**inputs) 2025-08-26T20:43:34.9957337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9957414Z outputs = self.model( 2025-08-26T20:43:34.9957702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9957792Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9958032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9958125Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9958396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9958504Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9958784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9958909Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9958913Z 2025-08-26T20:43:34.9959032Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9959248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9959321Z return mod(**inputs) 2025-08-26T20:43:34.9959679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9959783Z outputs = self.model( 2025-08-26T20:43:34.9960068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9960149Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9960407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9960524Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9960787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9960902Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9961164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9961259Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9961264Z 2025-08-26T20:43:34.9961373Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9961584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9961662Z return mod(**inputs) 2025-08-26T20:43:34.9961921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9962003Z outputs = self.model( 2025-08-26T20:43:34.9962266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9962351Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9962586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9962671Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9962939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9963044Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9963310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9963425Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9963429Z 2025-08-26T20:43:34.9963538Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9963800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9963873Z return mod(**inputs) 2025-08-26T20:43:34.9964156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9964230Z outputs = self.model( 2025-08-26T20:43:34.9964487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9964571Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9964805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9964896Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9965156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9965268Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9965530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9965675Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9965679Z 2025-08-26T20:43:34.9965797Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9966008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9966116Z return mod(**inputs) 2025-08-26T20:43:34.9966381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9966455Z outputs = self.model( 2025-08-26T20:43:34.9966725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9966821Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9967069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9967154Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9967426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9967530Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9967788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9967889Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9967893Z 2025-08-26T20:43:34.9968001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9968219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9968287Z return mod(**inputs) 2025-08-26T20:43:34.9968551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9968633Z outputs = self.model( 2025-08-26T20:43:34.9968894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9968978Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9969215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9969299Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9969565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9969669Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9969934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9970034Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9970038Z 2025-08-26T20:43:34.9970164Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9970377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9970446Z return mod(**inputs) 2025-08-26T20:43:34.9970730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9970802Z outputs = self.model( 2025-08-26T20:43:34.9971056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9971128Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9971361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9971452Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9971714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9971826Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9972094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9972236Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9972240Z 2025-08-26T20:43:34.9972362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9972567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9972644Z return mod(**inputs) 2025-08-26T20:43:34.9972902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9972981Z outputs = self.model( 2025-08-26T20:43:34.9973264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9973340Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9973586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9973673Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9973940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9974045Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9974306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9974400Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9974404Z 2025-08-26T20:43:34.9974513Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9974731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9974802Z return mod(**inputs) 2025-08-26T20:43:34.9975071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9975144Z outputs = self.model( 2025-08-26T20:43:34.9975406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9975493Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9975731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9975818Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9976063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9976183Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9976189Z 2025-08-26T20:43:34.9976299Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9976516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9976591Z return mod(**inputs) 2025-08-26T20:43:34.9976853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9976929Z outputs = self.model( 2025-08-26T20:43:34.9977176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9977248Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9977476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9977554Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9977809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:34.9977929Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:34.9978145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:34.9978224Z return self.act(input) 2025-08-26T20:43:34.9978227Z 2025-08-26T20:43:34.9978331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9978537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9978620Z return mod(**inputs) 2025-08-26T20:43:34.9978866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9978941Z outputs = self.model( 2025-08-26T20:43:34.9979189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9979284Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9979526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9979618Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9979887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:34.9979975Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:34.9979979Z 2025-08-26T20:43:34.9980096Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9980311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9980388Z return mod(**inputs) 2025-08-26T20:43:34.9980657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9980727Z outputs = self.model( 2025-08-26T20:43:34.9981007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9981084Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9981330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9981415Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9981680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:34.9981774Z hidden_states = residual + hidden_states 2025-08-26T20:43:34.9981778Z 2025-08-26T20:43:34.9981887Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9982109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9982179Z return mod(**inputs) 2025-08-26T20:43:34.9982461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9982530Z outputs = self.model( 2025-08-26T20:43:34.9982799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9982881Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9983126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9983218Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9983482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9983587Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9983855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9983971Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9983977Z 2025-08-26T20:43:34.9984093Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9984307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9984384Z return mod(**inputs) 2025-08-26T20:43:34.9984648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9984721Z outputs = self.model( 2025-08-26T20:43:34.9984987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9985081Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9985328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9985411Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9985677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9985803Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9986069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:34.9986160Z key_states = self.k_proj(current_states) 2025-08-26T20:43:34.9986165Z 2025-08-26T20:43:34.9986281Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9986483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9986557Z return mod(**inputs) 2025-08-26T20:43:34.9986803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9986881Z outputs = self.model( 2025-08-26T20:43:34.9987128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9987209Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9987433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9987512Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9987768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9987867Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9988136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:34.9988255Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:34.9988259Z 2025-08-26T20:43:34.9988370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9988591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9988664Z return mod(**inputs) 2025-08-26T20:43:34.9988934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9989024Z outputs = self.model( 2025-08-26T20:43:34.9989284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9989394Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9989630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9989725Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9989988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9990098Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9990359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:34.9990504Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:34.9990508Z 2025-08-26T20:43:34.9990632Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9990844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9990922Z return mod(**inputs) 2025-08-26T20:43:34.9991184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9991276Z outputs = self.model( 2025-08-26T20:43:34.9991547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9991624Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9991870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9991969Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9992240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9992345Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9992610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:34.9992711Z value_states = self.v_proj(current_states) 2025-08-26T20:43:34.9992714Z 2025-08-26T20:43:34.9992824Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9993042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9993116Z return mod(**inputs) 2025-08-26T20:43:34.9993381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9993461Z outputs = self.model( 2025-08-26T20:43:34.9993724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9993811Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9994044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9994127Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9994426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9994533Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9994819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:34.9994920Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:34.9994924Z 2025-08-26T20:43:34.9995039Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9995252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9995322Z return mod(**inputs) 2025-08-26T20:43:34.9995635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9995710Z outputs = self.model( 2025-08-26T20:43:34.9996011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9996091Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9996601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9996697Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9996980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9997093Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9997358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:34.9997506Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:34.9997510Z 2025-08-26T20:43:34.9997619Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:34.9997831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:34.9997911Z return mod(**inputs) 2025-08-26T20:43:34.9998239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:34.9998319Z outputs = self.model( 2025-08-26T20:43:34.9998635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:34.9998714Z layer_outputs = decoder_layer( 2025-08-26T20:43:34.9998996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:34.9999082Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:34.9999419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:34.9999537Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:34.9999826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:34.9999919Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:34.9999924Z 2025-08-26T20:43:35.0000036Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0000260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0000332Z return mod(**inputs) 2025-08-26T20:43:35.0000616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0000703Z outputs = self.model( 2025-08-26T20:43:35.0000989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0001074Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0001323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0001414Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0001677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0001802Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0001814Z 2025-08-26T20:43:35.0001924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0002134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0002214Z return mod(**inputs) 2025-08-26T20:43:35.0002580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0002661Z outputs = self.model( 2025-08-26T20:43:35.0002973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0003052Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0003309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0003394Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0003682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0003808Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0004034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:35.0004118Z return self.act(input) 2025-08-26T20:43:35.0004122Z 2025-08-26T20:43:35.0004231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0004447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0004517Z return mod(**inputs) 2025-08-26T20:43:35.0004776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0004875Z outputs = self.model( 2025-08-26T20:43:35.0005136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0005221Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0005458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0005568Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0005834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:35.0005920Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:35.0005925Z 2025-08-26T20:43:35.0006044Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0006258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0006335Z return mod(**inputs) 2025-08-26T20:43:35.0006598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0006671Z outputs = self.model( 2025-08-26T20:43:35.0006940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0007016Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0007262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0007345Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0007607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0007724Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0007986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0008114Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0008118Z 2025-08-26T20:43:35.0008227Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0008446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0008516Z return mod(**inputs) 2025-08-26T20:43:35.0008778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0008860Z outputs = self.model( 2025-08-26T20:43:35.0009139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0009225Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0009478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0009563Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0009832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0009935Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0010204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:35.0010291Z key_states = self.k_proj(current_states) 2025-08-26T20:43:35.0010297Z 2025-08-26T20:43:35.0010412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0010628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0010699Z return mod(**inputs) 2025-08-26T20:43:35.0010970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0011043Z outputs = self.model( 2025-08-26T20:43:35.0011314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0011407Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0011644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0011734Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0012001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0012129Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0012393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0012511Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0012522Z 2025-08-26T20:43:35.0012631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0012844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0012927Z return mod(**inputs) 2025-08-26T20:43:35.0013187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0013265Z outputs = self.model( 2025-08-26T20:43:35.0013527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0013615Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0013843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0013921Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0014168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0014263Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0014523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:35.0014676Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:35.0014680Z 2025-08-26T20:43:35.0014789Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0015015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0015079Z return mod(**inputs) 2025-08-26T20:43:35.0015347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0015415Z outputs = self.model( 2025-08-26T20:43:35.0015668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0015750Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0015964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0016050Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0016291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0016389Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0016642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:35.0016730Z value_states = self.v_proj(current_states) 2025-08-26T20:43:35.0016734Z 2025-08-26T20:43:35.0016842Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0017050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0017120Z return mod(**inputs) 2025-08-26T20:43:35.0017360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0017441Z outputs = self.model( 2025-08-26T20:43:35.0017695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0017766Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0017992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0018083Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0018324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0018426Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0018668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:35.0018768Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:35.0018772Z 2025-08-26T20:43:35.0018871Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0019066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0019138Z return mod(**inputs) 2025-08-26T20:43:35.0019377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0019453Z outputs = self.model( 2025-08-26T20:43:35.0019695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0019775Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0019994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0020073Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0020320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0020418Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0020670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:35.0020794Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:35.0020797Z 2025-08-26T20:43:35.0020897Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0021102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0021186Z return mod(**inputs) 2025-08-26T20:43:35.0021443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0021535Z outputs = self.model( 2025-08-26T20:43:35.0021791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0021864Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0022087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0022174Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0022422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0022530Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0022776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:35.0022859Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:35.0022863Z 2025-08-26T20:43:35.0022983Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0023177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0023267Z return mod(**inputs) 2025-08-26T20:43:35.0023508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0023577Z outputs = self.model( 2025-08-26T20:43:35.0023825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0023895Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0024139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0024218Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0024466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0024582Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0024585Z 2025-08-26T20:43:35.0024686Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0024888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0024953Z return mod(**inputs) 2025-08-26T20:43:35.0025199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0025264Z outputs = self.model( 2025-08-26T20:43:35.0025507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0025587Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0025802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0025887Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0026132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0026246Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0026465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:35.0026534Z return self.act(input) 2025-08-26T20:43:35.0026537Z 2025-08-26T20:43:35.0026645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0026837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0026912Z return mod(**inputs) 2025-08-26T20:43:35.0027170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0027240Z outputs = self.model( 2025-08-26T20:43:35.0027503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0027575Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0027800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0027878Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0028118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:35.0028203Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:35.0028207Z 2025-08-26T20:43:35.0028306Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0028503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0028568Z return mod(**inputs) 2025-08-26T20:43:35.0028816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0028883Z outputs = self.model( 2025-08-26T20:43:35.0029124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0029220Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0029435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0029520Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0029758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:35.0029851Z hidden_states = residual + hidden_states 2025-08-26T20:43:35.0029854Z 2025-08-26T20:43:35.0029960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0030154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0030225Z return mod(**inputs) 2025-08-26T20:43:35.0030467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0030533Z outputs = self.model( 2025-08-26T20:43:35.0030780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0030860Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0031075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0031150Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0031393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0031488Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0031721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0031834Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0031837Z 2025-08-26T20:43:35.0031935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0032133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0032196Z return mod(**inputs) 2025-08-26T20:43:35.0032434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0032508Z outputs = self.model( 2025-08-26T20:43:35.0032741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0032817Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0033043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0033119Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0033383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0033481Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0033727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:35.0033807Z key_states = self.k_proj(current_states) 2025-08-26T20:43:35.0033810Z 2025-08-26T20:43:35.0033915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0034112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0034178Z return mod(**inputs) 2025-08-26T20:43:35.0034428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0034496Z outputs = self.model( 2025-08-26T20:43:35.0034745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0034815Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0035030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0035132Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0035379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0035487Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0035730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0035874Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0035879Z 2025-08-26T20:43:35.0035982Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0036185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0036260Z return mod(**inputs) 2025-08-26T20:43:35.0036505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0036583Z outputs = self.model( 2025-08-26T20:43:35.0036830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0036905Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0037152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0037239Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0037568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0037666Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0037914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:35.0038061Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:35.0038066Z 2025-08-26T20:43:35.0038169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0038376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0038442Z return mod(**inputs) 2025-08-26T20:43:35.0038698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0038769Z outputs = self.model( 2025-08-26T20:43:35.0039033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0039119Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0039446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0039553Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0039833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0039946Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0040236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:35.0040332Z value_states = self.v_proj(current_states) 2025-08-26T20:43:35.0040337Z 2025-08-26T20:43:35.0040459Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0040681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0040754Z return mod(**inputs) 2025-08-26T20:43:35.0041000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0041069Z outputs = self.model( 2025-08-26T20:43:35.0041328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0041415Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0041636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0041713Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0041957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0042079Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0042320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:35.0042423Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:35.0042427Z 2025-08-26T20:43:35.0042527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0042732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0042800Z return mod(**inputs) 2025-08-26T20:43:35.0043045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0043119Z outputs = self.model( 2025-08-26T20:43:35.0043358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0043434Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0043651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0043728Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0043973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0044070Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0044322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:35.0044458Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:35.0044462Z 2025-08-26T20:43:35.0044570Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0044792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0044867Z return mod(**inputs) 2025-08-26T20:43:35.0045121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0045205Z outputs = self.model( 2025-08-26T20:43:35.0045462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0045549Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0045774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0045862Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0046161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0046261Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0046503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:35.0046584Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:35.0046587Z 2025-08-26T20:43:35.0046694Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0046889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0046962Z return mod(**inputs) 2025-08-26T20:43:35.0047203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0047271Z outputs = self.model( 2025-08-26T20:43:35.0047535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0047605Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0047826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0047904Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0048167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0048285Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0048289Z 2025-08-26T20:43:35.0048388Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0048596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0048662Z return mod(**inputs) 2025-08-26T20:43:35.0048918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0048986Z outputs = self.model( 2025-08-26T20:43:35.0049230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0049311Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0049532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0049621Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0049868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0049994Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0050211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:35.0050283Z return self.act(input) 2025-08-26T20:43:35.0050287Z 2025-08-26T20:43:35.0050396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0050600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0050675Z return mod(**inputs) 2025-08-26T20:43:35.0050923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0050995Z outputs = self.model( 2025-08-26T20:43:35.0051281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0051360Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0051624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0051708Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0051966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:35.0052062Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:35.0052066Z 2025-08-26T20:43:35.0052179Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0052395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0052465Z return mod(**inputs) 2025-08-26T20:43:35.0052736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0052810Z outputs = self.model( 2025-08-26T20:43:35.0053073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0053159Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0053392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0053503Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0053763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0053867Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0054137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0054272Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0054276Z 2025-08-26T20:43:35.0054393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0054607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0054680Z return mod(**inputs) 2025-08-26T20:43:35.0054955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0055032Z outputs = self.model( 2025-08-26T20:43:35.0055310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0055388Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0055629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0055714Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0055975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0056087Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0056348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:35.0056443Z key_states = self.k_proj(current_states) 2025-08-26T20:43:35.0056446Z 2025-08-26T20:43:35.0056555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0056765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0056842Z return mod(**inputs) 2025-08-26T20:43:35.0057103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0057183Z outputs = self.model( 2025-08-26T20:43:35.0057441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0057526Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0057778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0057865Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0058155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0058262Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0058539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0058654Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0058658Z 2025-08-26T20:43:35.0058767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0058985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0059057Z return mod(**inputs) 2025-08-26T20:43:35.0059337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0059410Z outputs = self.model( 2025-08-26T20:43:35.0059686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0059770Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0060032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0060124Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0060395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0060504Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0060804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:35.0060949Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:35.0060953Z 2025-08-26T20:43:35.0061068Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0061280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0061356Z return mod(**inputs) 2025-08-26T20:43:35.0061614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0061686Z outputs = self.model( 2025-08-26T20:43:35.0061976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0062054Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0062294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0062380Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0062686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0062790Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0063060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:35.0063163Z value_states = self.v_proj(current_states) 2025-08-26T20:43:35.0063167Z 2025-08-26T20:43:35.0063275Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0063500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0063571Z return mod(**inputs) 2025-08-26T20:43:35.0063845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0063930Z outputs = self.model( 2025-08-26T20:43:35.0064219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0064306Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0064595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0064680Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0064957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0065062Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0065338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:35.0065439Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:35.0065445Z 2025-08-26T20:43:35.0065561Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0065781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0065853Z return mod(**inputs) 2025-08-26T20:43:35.0066141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0066216Z outputs = self.model( 2025-08-26T20:43:35.0066536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0066633Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0066877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0066972Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0067248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0067381Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0067660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:35.0067807Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:35.0067811Z 2025-08-26T20:43:35.0067922Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0068140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0068223Z return mod(**inputs) 2025-08-26T20:43:35.0068492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0068573Z outputs = self.model( 2025-08-26T20:43:35.0068842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0068922Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0069173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0069260Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0069538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0069644Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0069911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:35.0070010Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:35.0070014Z 2025-08-26T20:43:35.0070127Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0070351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0070422Z return mod(**inputs) 2025-08-26T20:43:35.0070701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0071638Z outputs = self.model( 2025-08-26T20:43:35.0071925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0072034Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0072277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0072372Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0072640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0072771Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0072775Z 2025-08-26T20:43:35.0072895Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0073115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0073194Z return mod(**inputs) 2025-08-26T20:43:35.0073464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0073545Z outputs = self.model( 2025-08-26T20:43:35.0073816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0073911Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0074165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0074250Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0074526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0074654Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0074908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:35.0074994Z return self.act(input) 2025-08-26T20:43:35.0074998Z 2025-08-26T20:43:35.0075109Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0075334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0075404Z return mod(**inputs) 2025-08-26T20:43:35.0075673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0075755Z outputs = self.model( 2025-08-26T20:43:35.0076019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0076105Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0076345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0076439Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0076707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:35.0076796Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:35.0076800Z 2025-08-26T20:43:35.0076920Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0077135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0077216Z return mod(**inputs) 2025-08-26T20:43:35.0077502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0077576Z outputs = self.model( 2025-08-26T20:43:35.0077849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0077929Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0078174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0078275Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0078561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-26T20:43:35.0078659Z hidden_states = residual + hidden_states 2025-08-26T20:43:35.0078663Z 2025-08-26T20:43:35.0078776Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0079000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0079072Z return mod(**inputs) 2025-08-26T20:43:35.0079437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0079526Z outputs = self.model( 2025-08-26T20:43:35.0079804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0079894Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0080144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0080244Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0080522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0080652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0080928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0081049Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0081053Z 2025-08-26T20:43:35.0081174Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0081410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0081492Z return mod(**inputs) 2025-08-26T20:43:35.0081764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0081840Z outputs = self.model( 2025-08-26T20:43:35.0082117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0082197Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0082447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0082535Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0082801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0082916Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0083187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-26T20:43:35.0083281Z key_states = self.k_proj(current_states) 2025-08-26T20:43:35.0083285Z 2025-08-26T20:43:35.0083391Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0083604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0083681Z return mod(**inputs) 2025-08-26T20:43:35.0083943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0084023Z outputs = self.model( 2025-08-26T20:43:35.0084285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0084366Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0084600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0084686Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0084968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0085078Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0085383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-26T20:43:35.0085507Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-26T20:43:35.0085511Z 2025-08-26T20:43:35.0085621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0085848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0085922Z return mod(**inputs) 2025-08-26T20:43:35.0086197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0086274Z outputs = self.model( 2025-08-26T20:43:35.0086547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0086633Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0086885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0086976Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0087254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0087364Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0087625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-26T20:43:35.0087769Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-26T20:43:35.0087791Z 2025-08-26T20:43:35.0087910Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0088121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0088201Z return mod(**inputs) 2025-08-26T20:43:35.0088461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0088535Z outputs = self.model( 2025-08-26T20:43:35.0088805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0088882Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0089124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0089207Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0089476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0089582Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0089843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-26T20:43:35.0089943Z value_states = self.v_proj(current_states) 2025-08-26T20:43:35.0089949Z 2025-08-26T20:43:35.0090057Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0090274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0090344Z return mod(**inputs) 2025-08-26T20:43:35.0090606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0090686Z outputs = self.model( 2025-08-26T20:43:35.0090945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0091030Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0091280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0091362Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0091642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0091747Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0092016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-26T20:43:35.0092117Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-26T20:43:35.0092120Z 2025-08-26T20:43:35.0092233Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0092440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0092512Z return mod(**inputs) 2025-08-26T20:43:35.0092782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0092856Z outputs = self.model( 2025-08-26T20:43:35.0093122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0093199Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0093432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0093542Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0093802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0093915Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0094177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-26T20:43:35.0094339Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-26T20:43:35.0094343Z 2025-08-26T20:43:35.0094453Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0094664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0094742Z return mod(**inputs) 2025-08-26T20:43:35.0095005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0095087Z outputs = self.model( 2025-08-26T20:43:35.0095345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0095421Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0095663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0095749Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0096017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-26T20:43:35.0096121Z hidden_states, self_attn_weights = self.self_attn( 2025-08-26T20:43:35.0096561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-26T20:43:35.0096654Z attn_output = self.out_proj(attn_output) 2025-08-26T20:43:35.0096659Z 2025-08-26T20:43:35.0096771Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0096993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0097065Z return mod(**inputs) 2025-08-26T20:43:35.0097335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0097409Z outputs = self.model( 2025-08-26T20:43:35.0097675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0097808Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0098047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0098160Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0098423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0098553Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0098565Z 2025-08-26T20:43:35.0098675Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0098886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0098965Z return mod(**inputs) 2025-08-26T20:43:35.0099223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0099304Z outputs = self.model( 2025-08-26T20:43:35.0099572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0099649Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0099898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0099983Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0100282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-26T20:43:35.0100409Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-26T20:43:35.0100637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:43:35.0100720Z return self.act(input) 2025-08-26T20:43:35.0100748Z 2025-08-26T20:43:35.0100857Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0101077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0101146Z return mod(**inputs) 2025-08-26T20:43:35.0101408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-26T20:43:35.0101488Z outputs = self.model( 2025-08-26T20:43:35.0101752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-26T20:43:35.0101837Z layer_outputs = decoder_layer( 2025-08-26T20:43:35.0102081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:43:35.0102169Z return super().__call__(*args, **kwargs) 2025-08-26T20:43:35.0102414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-26T20:43:35.0102499Z hidden_states = self.fc2(hidden_states) 2025-08-26T20:43:35.0102503Z 2025-08-26T20:43:35.0102612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0102809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0102882Z return mod(**inputs) 2025-08-26T20:43:35.0103129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 681, in forward 2025-08-26T20:43:35.0103210Z logits = self.lm_head(outputs[0]) 2025-08-26T20:43:35.0103214Z 2025-08-26T20:43:35.0103325Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:43:35.0103525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:43:35.0103600Z return mod(**inputs) 2025-08-26T20:43:35.0103846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-08-26T20:43:35.0103930Z loss = self.loss_function( 2025-08-26T20:43:35.0104190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-26T20:43:35.0104385Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-26T20:43:35.0104648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-26T20:43:35.0104848Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-26T20:43:35.0104851Z 2025-08-26T20:43:47.3928864Z Compilation time (from dynamo_timed): 26.351385579 2025-08-26T20:43:47.4046623Z pass 2025-08-26T20:43:47.4047300Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:43:47.4048927Z TIMING: _recursive_pre_grad_passes:0.01454 _recursive_joint_graph_passes:0.84112 _recursive_post_grad_passes:0.30097 async_compile.wait:0.86928 code_gen:11.59031 inductor_compile:15.07892 backend_compile:21.41099 gc:0.00105 entire_frame_compile:26.35139 total_wall_time:26.35139 2025-08-26T20:43:53.3818151Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:29106 | FakeTensor.__torch_dispatch__:9977 | ProxyTorchDispatchMode.__torch_dispatch__:10816 2025-08-26T20:43:53.3818746Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-08-26T20:43:53.3820115Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:43:53.3821243Z from pkg_resources import resource_filename 2025-08-26T20:43:53.9777061Z 2025-08-26T20:43:57.4269456Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:43:57.4274534Z loading model: 0it [00:03, ?it/s] 2025-08-26T20:43:57.4298455Z cpu eval XLNetLMHeadModel 2025-08-26T20:44:00.1079895Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:44:01.0820610Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:44:02.0487289Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:44:24.5576934Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5577740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5587105Z return mod(**inputs) 2025-08-26T20:44:24.5588074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5588607Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5589099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-08-26T20:44:24.5589560Z word_emb_k = self.word_embedding(input_ids) 2025-08-26T20:44:24.5589732Z 2025-08-26T20:44:24.5589864Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5590278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5590649Z return mod(**inputs) 2025-08-26T20:44:24.5591069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5591498Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5591897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-26T20:44:24.5592354Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-26T20:44:24.5592864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-26T20:44:24.5593777Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-26T20:44:24.5594426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-26T20:44:24.5595043Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-26T20:44:24.5595280Z 2025-08-26T20:44:24.5595399Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5595810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5596342Z return mod(**inputs) 2025-08-26T20:44:24.5596760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5597206Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5597651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-26T20:44:24.5598187Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-26T20:44:24.5598755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-26T20:44:24.5599313Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-26T20:44:24.5600243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-26T20:44:24.5600820Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-26T20:44:24.5601053Z 2025-08-26T20:44:24.5601168Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5601646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5602015Z return mod(**inputs) 2025-08-26T20:44:24.5602417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5602862Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5603289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5603730Z outputs = layer_module( 2025-08-26T20:44:24.5604150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5604564Z outputs = self.rel_attn( 2025-08-26T20:44:24.5604976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5605420Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5605872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5606357Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5606558Z 2025-08-26T20:44:24.5606673Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5607064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5607414Z return mod(**inputs) 2025-08-26T20:44:24.5607806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5608229Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5608652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5609067Z outputs = layer_module( 2025-08-26T20:44:24.5609483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5609959Z outputs = self.rel_attn( 2025-08-26T20:44:24.5610390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5610890Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5611347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5611843Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5612025Z 2025-08-26T20:44:24.5612146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5612526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5612880Z return mod(**inputs) 2025-08-26T20:44:24.5613266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5613708Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5614123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5614560Z outputs = layer_module( 2025-08-26T20:44:24.5614960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5615411Z outputs = self.rel_attn( 2025-08-26T20:44:24.5615820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5616264Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5616732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5617235Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5617416Z 2025-08-26T20:44:24.5617537Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5617937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5618279Z return mod(**inputs) 2025-08-26T20:44:24.5618706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5619153Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5619580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5619997Z outputs = layer_module( 2025-08-26T20:44:24.5620402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5620887Z outputs = self.rel_attn( 2025-08-26T20:44:24.5621294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5621744Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5622181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5622664Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5622847Z 2025-08-26T20:44:24.5622959Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5623350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5623698Z return mod(**inputs) 2025-08-26T20:44:24.5624084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5624525Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5624963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5625374Z outputs = layer_module( 2025-08-26T20:44:24.5625794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5626223Z outputs = self.rel_attn( 2025-08-26T20:44:24.5626656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5627113Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5627580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5628074Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5628265Z 2025-08-26T20:44:24.5628381Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5628783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5629143Z return mod(**inputs) 2025-08-26T20:44:24.5629556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5629989Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5630759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5631206Z outputs = layer_module( 2025-08-26T20:44:24.5631615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5632042Z outputs = self.rel_attn( 2025-08-26T20:44:24.5632443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5632890Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5633370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5633865Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5634047Z 2025-08-26T20:44:24.5634164Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5634571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5634928Z return mod(**inputs) 2025-08-26T20:44:24.5635338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5635794Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5636224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5636658Z outputs = layer_module( 2025-08-26T20:44:24.5637082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5637502Z outputs = self.rel_attn( 2025-08-26T20:44:24.5637914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5638356Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5638830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5639317Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5639593Z 2025-08-26T20:44:24.5639731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5640137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5640491Z return mod(**inputs) 2025-08-26T20:44:24.5640896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5641347Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5641804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5642217Z outputs = layer_module( 2025-08-26T20:44:24.5642648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5643070Z outputs = self.rel_attn( 2025-08-26T20:44:24.5643476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5643921Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5644396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5644876Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5645057Z 2025-08-26T20:44:24.5645169Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5645556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5645902Z return mod(**inputs) 2025-08-26T20:44:24.5646284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5646709Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5647181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5647590Z outputs = layer_module( 2025-08-26T20:44:24.5647978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5648398Z outputs = self.rel_attn( 2025-08-26T20:44:24.5648824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5649266Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5649709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5650173Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5650359Z 2025-08-26T20:44:24.5650474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5650861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5651205Z return mod(**inputs) 2025-08-26T20:44:24.5651587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5652022Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5652453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5652874Z outputs = layer_module( 2025-08-26T20:44:24.5653281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5653694Z outputs = self.rel_attn( 2025-08-26T20:44:24.5654106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5654559Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5655010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5656306Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5656527Z 2025-08-26T20:44:24.5656663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5657205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5657641Z return mod(**inputs) 2025-08-26T20:44:24.5658222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5658713Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5659215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5659847Z outputs = layer_module( 2025-08-26T20:44:24.5660468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5661084Z outputs = self.rel_attn( 2025-08-26T20:44:24.5661650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5662440Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5662958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5663654Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5663941Z 2025-08-26T20:44:24.5664074Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5664614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5665093Z return mod(**inputs) 2025-08-26T20:44:24.5665669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5666186Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5666734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5667229Z outputs = layer_module( 2025-08-26T20:44:24.5667807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5668417Z outputs = self.rel_attn( 2025-08-26T20:44:24.5668940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5669549Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5670018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5670614Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5670805Z 2025-08-26T20:44:24.5670989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5671544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5672012Z return mod(**inputs) 2025-08-26T20:44:24.5672564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5673024Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5673467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5673895Z outputs = layer_module( 2025-08-26T20:44:24.5674405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5675053Z outputs = self.rel_attn( 2025-08-26T20:44:24.5675616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5676261Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5676940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5677668Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5677858Z 2025-08-26T20:44:24.5677975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5678511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5679073Z return mod(**inputs) 2025-08-26T20:44:24.5679647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5680353Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5681043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5681632Z outputs = layer_module( 2025-08-26T20:44:24.5682046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5682466Z outputs = self.rel_attn( 2025-08-26T20:44:24.5682994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5683546Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5684003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5684490Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5684762Z 2025-08-26T20:44:24.5684901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5685413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5685789Z return mod(**inputs) 2025-08-26T20:44:24.5686197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5686638Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5687116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5687535Z outputs = layer_module( 2025-08-26T20:44:24.5687939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5688360Z outputs = self.rel_attn( 2025-08-26T20:44:24.5688763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5689195Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5689642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5690118Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5690294Z 2025-08-26T20:44:24.5690414Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5690797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5691147Z return mod(**inputs) 2025-08-26T20:44:24.5691538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5691964Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5692385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5692795Z outputs = layer_module( 2025-08-26T20:44:24.5693188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5693601Z outputs = self.rel_attn( 2025-08-26T20:44:24.5694003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5694438Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5694901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5695405Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5695590Z 2025-08-26T20:44:24.5695713Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5696123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5696627Z return mod(**inputs) 2025-08-26T20:44:24.5697040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5697480Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5697904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5698320Z outputs = layer_module( 2025-08-26T20:44:24.5698719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5699127Z outputs = self.rel_attn( 2025-08-26T20:44:24.5699526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5699958Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5700399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5700976Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5701156Z 2025-08-26T20:44:24.5701269Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5701663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5702010Z return mod(**inputs) 2025-08-26T20:44:24.5702372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5702817Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5703225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5703629Z outputs = layer_module( 2025-08-26T20:44:24.5704015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5704425Z outputs = self.rel_attn( 2025-08-26T20:44:24.5704818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5705251Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5705702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5706176Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5706358Z 2025-08-26T20:44:24.5706472Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5706855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5707205Z return mod(**inputs) 2025-08-26T20:44:24.5707599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5708020Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5708448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5708856Z outputs = layer_module( 2025-08-26T20:44:24.5709244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5709658Z outputs = self.rel_attn( 2025-08-26T20:44:24.5710053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5710511Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5710967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5711499Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5711679Z 2025-08-26T20:44:24.5711792Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5712175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5712525Z return mod(**inputs) 2025-08-26T20:44:24.5712942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5713384Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5713805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5714228Z outputs = layer_module( 2025-08-26T20:44:24.5714623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5715036Z outputs = self.rel_attn( 2025-08-26T20:44:24.5715421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5715873Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5716320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5716795Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5716970Z 2025-08-26T20:44:24.5717089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5717520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5717878Z return mod(**inputs) 2025-08-26T20:44:24.5718279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5718713Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5719143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5719922Z outputs = layer_module( 2025-08-26T20:44:24.5720323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5720733Z outputs = self.rel_attn( 2025-08-26T20:44:24.5721128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5721550Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5722006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5722482Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5722659Z 2025-08-26T20:44:24.5722777Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5723176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5723506Z return mod(**inputs) 2025-08-26T20:44:24.5723894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5724324Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5724750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5725164Z outputs = layer_module( 2025-08-26T20:44:24.5725535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5725947Z outputs = self.rel_attn( 2025-08-26T20:44:24.5727201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5727675Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5728121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5728593Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5728767Z 2025-08-26T20:44:24.5728876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5729242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5729576Z return mod(**inputs) 2025-08-26T20:44:24.5729941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5730351Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5730752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5731137Z outputs = layer_module( 2025-08-26T20:44:24.5731508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5731907Z outputs = self.rel_attn( 2025-08-26T20:44:24.5732279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5732684Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5733109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5733579Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5733746Z 2025-08-26T20:44:24.5733852Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5734218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5734627Z return mod(**inputs) 2025-08-26T20:44:24.5735010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5735414Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5735819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5736218Z outputs = layer_module( 2025-08-26T20:44:24.5736644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5737060Z outputs = self.rel_attn( 2025-08-26T20:44:24.5737445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5737880Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5738330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5738804Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5738969Z 2025-08-26T20:44:24.5739082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5739445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5739781Z return mod(**inputs) 2025-08-26T20:44:24.5740170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5740595Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5741011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5741415Z outputs = layer_module( 2025-08-26T20:44:24.5741824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5742261Z outputs = self.rel_attn( 2025-08-26T20:44:24.5742655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.5743094Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.5743266Z 2025-08-26T20:44:24.5743379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5743762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5744107Z return mod(**inputs) 2025-08-26T20:44:24.5744489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5744919Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5745346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5745763Z outputs = layer_module( 2025-08-26T20:44:24.5746153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5746557Z outputs = self.rel_attn( 2025-08-26T20:44:24.5746974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.5747424Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.5747593Z 2025-08-26T20:44:24.5747713Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5748100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5748467Z return mod(**inputs) 2025-08-26T20:44:24.5748865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5749273Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5749669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5750047Z outputs = layer_module( 2025-08-26T20:44:24.5750418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5750806Z outputs = self.rel_attn( 2025-08-26T20:44:24.5751181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5751577Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5751982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.5752456Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.5752656Z 2025-08-26T20:44:24.5752762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5753130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5753464Z return mod(**inputs) 2025-08-26T20:44:24.5753826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5754239Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5754639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-26T20:44:24.5755090Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-26T20:44:24.5755595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-26T20:44:24.5756145Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-26T20:44:24.5756650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-26T20:44:24.5757214Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-26T20:44:24.5757441Z 2025-08-26T20:44:24.5757564Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5757955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5758301Z return mod(**inputs) 2025-08-26T20:44:24.5758695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5759130Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5759647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5760071Z outputs = layer_module( 2025-08-26T20:44:24.5760479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5760911Z outputs = self.rel_attn( 2025-08-26T20:44:24.5761315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.5761817Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.5762021Z 2025-08-26T20:44:24.5762128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5762503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5762831Z return mod(**inputs) 2025-08-26T20:44:24.5763197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5763627Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5764025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5764412Z outputs = layer_module( 2025-08-26T20:44:24.5764786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5765173Z outputs = self.rel_attn( 2025-08-26T20:44:24.5765538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5765954Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5766404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.5766880Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.5767071Z 2025-08-26T20:44:24.5767187Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5767553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5767879Z return mod(**inputs) 2025-08-26T20:44:24.5768238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5768638Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5769033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5769414Z outputs = layer_module( 2025-08-26T20:44:24.5769792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5770165Z outputs = self.rel_attn( 2025-08-26T20:44:24.5770527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.5770981Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.5771148Z 2025-08-26T20:44:24.5771255Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5771638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5771972Z return mod(**inputs) 2025-08-26T20:44:24.5772341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5772747Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5773152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5773545Z outputs = layer_module( 2025-08-26T20:44:24.5773917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5774301Z outputs = self.rel_attn( 2025-08-26T20:44:24.5774668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5775059Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5775465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.5775938Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.5776121Z 2025-08-26T20:44:24.5776228Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5776595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5776928Z return mod(**inputs) 2025-08-26T20:44:24.5777285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5777710Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5778107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5778486Z outputs = layer_module( 2025-08-26T20:44:24.5778844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5779220Z outputs = self.rel_attn( 2025-08-26T20:44:24.5779576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5779987Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5780417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5780876Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5781055Z 2025-08-26T20:44:24.5781167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5781516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5781851Z return mod(**inputs) 2025-08-26T20:44:24.5782223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5782634Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5783049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5783433Z outputs = layer_module( 2025-08-26T20:44:24.5783809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5784197Z outputs = self.rel_attn( 2025-08-26T20:44:24.5784607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5785046Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5785527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5786031Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5786221Z 2025-08-26T20:44:24.5786335Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5786737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5787061Z return mod(**inputs) 2025-08-26T20:44:24.5787429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5787834Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5788231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5788619Z outputs = layer_module( 2025-08-26T20:44:24.5788985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5789524Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5790062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5790492Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5790892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5791279Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5791665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.5792085Z output = self.layer_1(output) 2025-08-26T20:44:24.5792211Z 2025-08-26T20:44:24.5792323Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5792685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5793016Z return mod(**inputs) 2025-08-26T20:44:24.5793392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5793821Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5794246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5794698Z outputs = layer_module( 2025-08-26T20:44:24.5795091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5795658Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5796384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5796822Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5797236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5797657Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5798070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.5798511Z output = self.activation_function(output) 2025-08-26T20:44:24.5798913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.5799289Z return self.act(input) 2025-08-26T20:44:24.5799474Z 2025-08-26T20:44:24.5799594Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5800003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5800434Z return mod(**inputs) 2025-08-26T20:44:24.5800856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5801342Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5801771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5802191Z outputs = layer_module( 2025-08-26T20:44:24.5802590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5803162Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5803736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5804166Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5804586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5805000Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5805396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.5805846Z output = self.layer_2(output) 2025-08-26T20:44:24.5805991Z 2025-08-26T20:44:24.5806106Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5806499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5806845Z return mod(**inputs) 2025-08-26T20:44:24.5807239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5807703Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5808138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5808553Z outputs = layer_module( 2025-08-26T20:44:24.5808949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5809370Z outputs = self.rel_attn( 2025-08-26T20:44:24.5809788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.5810217Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.5810376Z 2025-08-26T20:44:24.5810498Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5810887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5811244Z return mod(**inputs) 2025-08-26T20:44:24.5811643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5812082Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5812509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5812928Z outputs = layer_module( 2025-08-26T20:44:24.5813333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5813753Z outputs = self.rel_attn( 2025-08-26T20:44:24.5814157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.5814599Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.5814767Z 2025-08-26T20:44:24.5814875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5815251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5815611Z return mod(**inputs) 2025-08-26T20:44:24.5815971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5816394Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5816797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5817187Z outputs = layer_module( 2025-08-26T20:44:24.5817562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5817966Z outputs = self.rel_attn( 2025-08-26T20:44:24.5818366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5818787Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5819223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.5819709Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.5819901Z 2025-08-26T20:44:24.5820009Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5820377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5820731Z return mod(**inputs) 2025-08-26T20:44:24.5821104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5821516Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5821916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5822350Z outputs = layer_module( 2025-08-26T20:44:24.5822722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5823112Z outputs = self.rel_attn( 2025-08-26T20:44:24.5823478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.5823934Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.5824133Z 2025-08-26T20:44:24.5824240Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5824607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5824935Z return mod(**inputs) 2025-08-26T20:44:24.5825296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5825700Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5826108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5826491Z outputs = layer_module( 2025-08-26T20:44:24.5826851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5827240Z outputs = self.rel_attn( 2025-08-26T20:44:24.5827610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5828004Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5828423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.5828908Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.5829117Z 2025-08-26T20:44:24.5829228Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5829599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5829919Z return mod(**inputs) 2025-08-26T20:44:24.5830307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5830720Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5831113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5831496Z outputs = layer_module( 2025-08-26T20:44:24.5831854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5832219Z outputs = self.rel_attn( 2025-08-26T20:44:24.5832578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.5832983Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.5833138Z 2025-08-26T20:44:24.5833246Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5833598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5833911Z return mod(**inputs) 2025-08-26T20:44:24.5834268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5834661Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5835071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5835449Z outputs = layer_module( 2025-08-26T20:44:24.5835813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5836204Z outputs = self.rel_attn( 2025-08-26T20:44:24.5836597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5836992Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5837397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.5837890Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.5838086Z 2025-08-26T20:44:24.5838201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5838589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5838941Z return mod(**inputs) 2025-08-26T20:44:24.5839437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5839911Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5840374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5840804Z outputs = layer_module( 2025-08-26T20:44:24.5841170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5841545Z outputs = self.rel_attn( 2025-08-26T20:44:24.5841917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5842318Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5842757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5843189Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5843361Z 2025-08-26T20:44:24.5843467Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5843820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5844140Z return mod(**inputs) 2025-08-26T20:44:24.5844531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5844919Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5845337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5845737Z outputs = layer_module( 2025-08-26T20:44:24.5846112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5846499Z outputs = self.rel_attn( 2025-08-26T20:44:24.5846865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5847274Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5847697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5848150Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5848316Z 2025-08-26T20:44:24.5848431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5848791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5849121Z return mod(**inputs) 2025-08-26T20:44:24.5849499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5849893Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5850272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5850649Z outputs = layer_module( 2025-08-26T20:44:24.5851007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5851544Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5852076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5852477Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5852859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5853244Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5853616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.5853992Z output = self.layer_1(output) 2025-08-26T20:44:24.5854115Z 2025-08-26T20:44:24.5854218Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5854579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5854898Z return mod(**inputs) 2025-08-26T20:44:24.5855257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5855653Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5856045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5856426Z outputs = layer_module( 2025-08-26T20:44:24.5856790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5857303Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5857821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5858223Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5858657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5859081Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5859520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.5859941Z output = self.activation_function(output) 2025-08-26T20:44:24.5860330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.5860691Z return self.act(input) 2025-08-26T20:44:24.5860803Z 2025-08-26T20:44:24.5860916Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5861274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5861607Z return mod(**inputs) 2025-08-26T20:44:24.5861991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5862418Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5862840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5863243Z outputs = layer_module( 2025-08-26T20:44:24.5863634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5864229Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5864788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5865215Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5865640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5866062Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5866468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.5866889Z output = self.layer_2(output) 2025-08-26T20:44:24.5867021Z 2025-08-26T20:44:24.5867140Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5867523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5867868Z return mod(**inputs) 2025-08-26T20:44:24.5868258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5868684Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5869100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5869515Z outputs = layer_module( 2025-08-26T20:44:24.5869911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5870322Z outputs = self.rel_attn( 2025-08-26T20:44:24.5870703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.5871135Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.5871310Z 2025-08-26T20:44:24.5871420Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5871795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5872123Z return mod(**inputs) 2025-08-26T20:44:24.5872483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5872891Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5873321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5873712Z outputs = layer_module( 2025-08-26T20:44:24.5874112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5874526Z outputs = self.rel_attn( 2025-08-26T20:44:24.5874923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.5875378Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.5875543Z 2025-08-26T20:44:24.5875663Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5876050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5876396Z return mod(**inputs) 2025-08-26T20:44:24.5876799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5877242Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5877681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5878107Z outputs = layer_module( 2025-08-26T20:44:24.5878511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5878969Z outputs = self.rel_attn( 2025-08-26T20:44:24.5879452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5879910Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5880350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.5880895Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.5881117Z 2025-08-26T20:44:24.5881236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5881635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5881978Z return mod(**inputs) 2025-08-26T20:44:24.5882345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5882743Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5883130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5883510Z outputs = layer_module( 2025-08-26T20:44:24.5883871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5884262Z outputs = self.rel_attn( 2025-08-26T20:44:24.5884641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.5885119Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.5885320Z 2025-08-26T20:44:24.5885440Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5885819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5886171Z return mod(**inputs) 2025-08-26T20:44:24.5886570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5886975Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5887394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5887796Z outputs = layer_module( 2025-08-26T20:44:24.5888188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5888638Z outputs = self.rel_attn( 2025-08-26T20:44:24.5889040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5889477Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5889909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.5890405Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.5890599Z 2025-08-26T20:44:24.5890719Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5891110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5891467Z return mod(**inputs) 2025-08-26T20:44:24.5891862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5892299Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5892725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5893147Z outputs = layer_module( 2025-08-26T20:44:24.5893538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5893975Z outputs = self.rel_attn( 2025-08-26T20:44:24.5894377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.5894836Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.5895002Z 2025-08-26T20:44:24.5895119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5895518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5895878Z return mod(**inputs) 2025-08-26T20:44:24.5896501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5896949Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5897385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5897825Z outputs = layer_module( 2025-08-26T20:44:24.5898227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5898645Z outputs = self.rel_attn( 2025-08-26T20:44:24.5899056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5899473Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5899882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.5900345Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.5900528Z 2025-08-26T20:44:24.5900648Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5901028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5901378Z return mod(**inputs) 2025-08-26T20:44:24.5901769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5902207Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5902628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5903043Z outputs = layer_module( 2025-08-26T20:44:24.5903440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5903833Z outputs = self.rel_attn( 2025-08-26T20:44:24.5904266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5904705Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5905127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5905585Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5905766Z 2025-08-26T20:44:24.5905876Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5906251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5906600Z return mod(**inputs) 2025-08-26T20:44:24.5907002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5907444Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5907876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5908302Z outputs = layer_module( 2025-08-26T20:44:24.5908676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5909098Z outputs = self.rel_attn( 2025-08-26T20:44:24.5909473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5909918Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5910371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5910880Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5911069Z 2025-08-26T20:44:24.5911180Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5911575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5911925Z return mod(**inputs) 2025-08-26T20:44:24.5912308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5912750Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5913185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5913572Z outputs = layer_module( 2025-08-26T20:44:24.5913952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5914520Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5915068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5915494Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5915917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5916331Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5916726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.5917146Z output = self.layer_1(output) 2025-08-26T20:44:24.5917288Z 2025-08-26T20:44:24.5917401Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5917791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5918139Z return mod(**inputs) 2025-08-26T20:44:24.5918521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5918962Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5919457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5919918Z outputs = layer_module( 2025-08-26T20:44:24.5920319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5920980Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5921541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5921971Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5922387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5922796Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5923202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.5923636Z output = self.activation_function(output) 2025-08-26T20:44:24.5924026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.5924419Z return self.act(input) 2025-08-26T20:44:24.5924539Z 2025-08-26T20:44:24.5924650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5925035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5925383Z return mod(**inputs) 2025-08-26T20:44:24.5925771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5926208Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5926631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5927042Z outputs = layer_module( 2025-08-26T20:44:24.5927437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5927994Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5928523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5928915Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5929298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5929688Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5930076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.5930465Z output = self.layer_2(output) 2025-08-26T20:44:24.5930599Z 2025-08-26T20:44:24.5930705Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5931078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5931423Z return mod(**inputs) 2025-08-26T20:44:24.5931776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5932176Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5932564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5932941Z outputs = layer_module( 2025-08-26T20:44:24.5933300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5933680Z outputs = self.rel_attn( 2025-08-26T20:44:24.5934064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.5934471Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.5934647Z 2025-08-26T20:44:24.5934761Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5935119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5935432Z return mod(**inputs) 2025-08-26T20:44:24.5935797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5936209Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5936596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5936979Z outputs = layer_module( 2025-08-26T20:44:24.5937359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5937778Z outputs = self.rel_attn( 2025-08-26T20:44:24.5938175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.5938617Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.5938812Z 2025-08-26T20:44:24.5938917Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5939279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5939609Z return mod(**inputs) 2025-08-26T20:44:24.5939975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5940397Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5940798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5941191Z outputs = layer_module( 2025-08-26T20:44:24.5941568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5941967Z outputs = self.rel_attn( 2025-08-26T20:44:24.5942333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5942725Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5943125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.5943588Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.5943783Z 2025-08-26T20:44:24.5943901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5944263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5944603Z return mod(**inputs) 2025-08-26T20:44:24.5944985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5945400Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5945826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5946243Z outputs = layer_module( 2025-08-26T20:44:24.5946640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5947048Z outputs = self.rel_attn( 2025-08-26T20:44:24.5947436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.5947921Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.5948135Z 2025-08-26T20:44:24.5948268Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5948657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5949023Z return mod(**inputs) 2025-08-26T20:44:24.5949412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5949836Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5950259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5950672Z outputs = layer_module( 2025-08-26T20:44:24.5951066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5951477Z outputs = self.rel_attn( 2025-08-26T20:44:24.5951899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5952289Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5952711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.5953214Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.5953432Z 2025-08-26T20:44:24.5953549Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5953932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5954292Z return mod(**inputs) 2025-08-26T20:44:24.5954683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5955112Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5955564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5955990Z outputs = layer_module( 2025-08-26T20:44:24.5956390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5956802Z outputs = self.rel_attn( 2025-08-26T20:44:24.5957206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.5957651Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.5957824Z 2025-08-26T20:44:24.5957936Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5958334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5958701Z return mod(**inputs) 2025-08-26T20:44:24.5959105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5959646Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5960097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5960539Z outputs = layer_module( 2025-08-26T20:44:24.5960950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5961368Z outputs = self.rel_attn( 2025-08-26T20:44:24.5961805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.5962237Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.5962673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.5963168Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.5963362Z 2025-08-26T20:44:24.5963476Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5963896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5964247Z return mod(**inputs) 2025-08-26T20:44:24.5964659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5965078Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5965501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5965909Z outputs = layer_module( 2025-08-26T20:44:24.5966298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5966707Z outputs = self.rel_attn( 2025-08-26T20:44:24.5967096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5967529Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5967977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5968459Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5968640Z 2025-08-26T20:44:24.5968759Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5969162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5969515Z return mod(**inputs) 2025-08-26T20:44:24.5969905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5970342Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5970779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5971195Z outputs = layer_module( 2025-08-26T20:44:24.5971593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5972005Z outputs = self.rel_attn( 2025-08-26T20:44:24.5972401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.5972830Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.5973287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.5973764Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.5973944Z 2025-08-26T20:44:24.5974063Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5974451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5974789Z return mod(**inputs) 2025-08-26T20:44:24.5975179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5975605Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5976032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5976438Z outputs = layer_module( 2025-08-26T20:44:24.5976832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5977399Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5977958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5978386Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5978822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5979218Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5979633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.5980050Z output = self.layer_1(output) 2025-08-26T20:44:24.5980184Z 2025-08-26T20:44:24.5980301Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5980678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5981026Z return mod(**inputs) 2025-08-26T20:44:24.5981410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5981835Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5982257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5982659Z outputs = layer_module( 2025-08-26T20:44:24.5983048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5983586Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5984137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5984537Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5984927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5985317Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5985734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.5986164Z output = self.activation_function(output) 2025-08-26T20:44:24.5986543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.5986909Z return self.act(input) 2025-08-26T20:44:24.5987039Z 2025-08-26T20:44:24.5987152Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5987542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5987889Z return mod(**inputs) 2025-08-26T20:44:24.5988269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5988693Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5989116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5989525Z outputs = layer_module( 2025-08-26T20:44:24.5989909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.5990470Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.5991024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.5991446Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.5991861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.5992266Z output_x = self.ff(output_x) 2025-08-26T20:44:24.5992671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.5993088Z output = self.layer_2(output) 2025-08-26T20:44:24.5993219Z 2025-08-26T20:44:24.5993339Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5993757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5994102Z return mod(**inputs) 2025-08-26T20:44:24.5994504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5994934Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.5995362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.5995773Z outputs = layer_module( 2025-08-26T20:44:24.5996334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.5996771Z outputs = self.rel_attn( 2025-08-26T20:44:24.5997180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.5997633Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.5997802Z 2025-08-26T20:44:24.5997915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.5998309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.5998666Z return mod(**inputs) 2025-08-26T20:44:24.5999071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.5999623Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6000063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6000496Z outputs = layer_module( 2025-08-26T20:44:24.6000894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6001344Z outputs = self.rel_attn( 2025-08-26T20:44:24.6001734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6002184Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6002359Z 2025-08-26T20:44:24.6002474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6002859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6003207Z return mod(**inputs) 2025-08-26T20:44:24.6003589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6004038Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6004465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6004882Z outputs = layer_module( 2025-08-26T20:44:24.6005280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6005691Z outputs = self.rel_attn( 2025-08-26T20:44:24.6006088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6006506Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6006937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6007431Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6007650Z 2025-08-26T20:44:24.6007762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6008146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6008498Z return mod(**inputs) 2025-08-26T20:44:24.6008915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6009338Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6009792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6010206Z outputs = layer_module( 2025-08-26T20:44:24.6010601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6011015Z outputs = self.rel_attn( 2025-08-26T20:44:24.6011403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6011886Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6012095Z 2025-08-26T20:44:24.6012209Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6012593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6012938Z return mod(**inputs) 2025-08-26T20:44:24.6013223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6013315Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6013594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6013686Z outputs = layer_module( 2025-08-26T20:44:24.6013963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6014040Z outputs = self.rel_attn( 2025-08-26T20:44:24.6014310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6014416Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6014707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6014856Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6014860Z 2025-08-26T20:44:24.6014971Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6015184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6015262Z return mod(**inputs) 2025-08-26T20:44:24.6015537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6015631Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6015902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6015982Z outputs = layer_module( 2025-08-26T20:44:24.6016255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6016330Z outputs = self.rel_attn( 2025-08-26T20:44:24.6016606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6016716Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6016720Z 2025-08-26T20:44:24.6016838Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6017047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6017118Z return mod(**inputs) 2025-08-26T20:44:24.6017396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6017483Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6017762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6017894Z outputs = layer_module( 2025-08-26T20:44:24.6018172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6018260Z outputs = self.rel_attn( 2025-08-26T20:44:24.6018532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6018621Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6018917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6019060Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6019063Z 2025-08-26T20:44:24.6019178Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6019394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6019475Z return mod(**inputs) 2025-08-26T20:44:24.6019750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6019851Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6020129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6020222Z outputs = layer_module( 2025-08-26T20:44:24.6020497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6020570Z outputs = self.rel_attn( 2025-08-26T20:44:24.6020846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6020962Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6021267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6021391Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6021395Z 2025-08-26T20:44:24.6021507Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6021726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6021801Z return mod(**inputs) 2025-08-26T20:44:24.6022078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6022165Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6022433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6022517Z outputs = layer_module( 2025-08-26T20:44:24.6022786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6022867Z outputs = self.rel_attn( 2025-08-26T20:44:24.6023135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6023238Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6023529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6023651Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6023655Z 2025-08-26T20:44:24.6023773Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6023983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6024061Z return mod(**inputs) 2025-08-26T20:44:24.6024333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6024440Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6024717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6024807Z outputs = layer_module( 2025-08-26T20:44:24.6025087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6025315Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6025602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6025688Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6025969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6026052Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6026303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6026383Z output = self.layer_1(output) 2025-08-26T20:44:24.6026387Z 2025-08-26T20:44:24.6026489Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6026688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6026781Z return mod(**inputs) 2025-08-26T20:44:24.6027037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6027128Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6027395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6027498Z outputs = layer_module( 2025-08-26T20:44:24.6027775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6028001Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6028273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6028354Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6028618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6028692Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6028948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6029047Z output = self.activation_function(output) 2025-08-26T20:44:24.6029265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6029343Z return self.act(input) 2025-08-26T20:44:24.6029347Z 2025-08-26T20:44:24.6029449Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6029657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6029727Z return mod(**inputs) 2025-08-26T20:44:24.6029987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6030078Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6030332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6030417Z outputs = layer_module( 2025-08-26T20:44:24.6030662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6030882Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6031146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6031236Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6031492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6031566Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6031832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6031917Z output = self.layer_2(output) 2025-08-26T20:44:24.6031921Z 2025-08-26T20:44:24.6032030Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6032249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6032322Z return mod(**inputs) 2025-08-26T20:44:24.6032596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6032683Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6032952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6033053Z outputs = layer_module( 2025-08-26T20:44:24.6033322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6033405Z outputs = self.rel_attn( 2025-08-26T20:44:24.6033676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6033781Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6033801Z 2025-08-26T20:44:24.6033918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6034131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6034212Z return mod(**inputs) 2025-08-26T20:44:24.6034482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6034578Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6034849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6034923Z outputs = layer_module( 2025-08-26T20:44:24.6035198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6035272Z outputs = self.rel_attn( 2025-08-26T20:44:24.6035550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6035661Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6035666Z 2025-08-26T20:44:24.6035774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6035995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6036066Z return mod(**inputs) 2025-08-26T20:44:24.6036343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6036431Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6036699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6036778Z outputs = layer_module( 2025-08-26T20:44:24.6037048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6037130Z outputs = self.rel_attn( 2025-08-26T20:44:24.6037414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6037505Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6037814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6037959Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6037964Z 2025-08-26T20:44:24.6038080Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6038287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6038365Z return mod(**inputs) 2025-08-26T20:44:24.6038635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6038726Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6039005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6039078Z outputs = layer_module( 2025-08-26T20:44:24.6039439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6039522Z outputs = self.rel_attn( 2025-08-26T20:44:24.6039824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6039972Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6039976Z 2025-08-26T20:44:24.6040089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6040317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6040411Z return mod(**inputs) 2025-08-26T20:44:24.6040710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6040795Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6041055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6041135Z outputs = layer_module( 2025-08-26T20:44:24.6041389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6041468Z outputs = self.rel_attn( 2025-08-26T20:44:24.6041724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6041797Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6042086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6042220Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6042224Z 2025-08-26T20:44:24.6042336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6042533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6042607Z return mod(**inputs) 2025-08-26T20:44:24.6042866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6042951Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6043213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6043281Z outputs = layer_module( 2025-08-26T20:44:24.6043542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6043613Z outputs = self.rel_attn( 2025-08-26T20:44:24.6043892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6044006Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6044009Z 2025-08-26T20:44:24.6044129Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6044338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6044405Z return mod(**inputs) 2025-08-26T20:44:24.6044669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6044754Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6045011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6045088Z outputs = layer_module( 2025-08-26T20:44:24.6045347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6045424Z outputs = self.rel_attn( 2025-08-26T20:44:24.6045682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6045754Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6046036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6046181Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6046185Z 2025-08-26T20:44:24.6046297Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6046496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6046569Z return mod(**inputs) 2025-08-26T20:44:24.6046845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6046930Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6047193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6047263Z outputs = layer_module( 2025-08-26T20:44:24.6047524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6047594Z outputs = self.rel_attn( 2025-08-26T20:44:24.6047845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6047943Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6048217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6048340Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6048343Z 2025-08-26T20:44:24.6048447Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6048653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6048722Z return mod(**inputs) 2025-08-26T20:44:24.6048983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6049077Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6049333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6049407Z outputs = layer_module( 2025-08-26T20:44:24.6049661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6049731Z outputs = self.rel_attn( 2025-08-26T20:44:24.6050012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6050105Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6050402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6050516Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6050522Z 2025-08-26T20:44:24.6050631Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6050833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6050899Z return mod(**inputs) 2025-08-26T20:44:24.6051163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6051246Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6051509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6051579Z outputs = layer_module( 2025-08-26T20:44:24.6051833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6052051Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6052343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6052429Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6052684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6052758Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6053021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6053112Z output = self.layer_1(output) 2025-08-26T20:44:24.6053117Z 2025-08-26T20:44:24.6053229Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6053427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6053500Z return mod(**inputs) 2025-08-26T20:44:24.6053756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6053842Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6054103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6054171Z outputs = layer_module( 2025-08-26T20:44:24.6054429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6054638Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6054902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6054989Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6055242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6055324Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6055581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6055676Z output = self.activation_function(output) 2025-08-26T20:44:24.6055895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6055969Z return self.act(input) 2025-08-26T20:44:24.6055972Z 2025-08-26T20:44:24.6056081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6056296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6056371Z return mod(**inputs) 2025-08-26T20:44:24.6056645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6056730Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6056992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6057062Z outputs = layer_module( 2025-08-26T20:44:24.6057319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6057525Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6057797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6057873Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6058134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6058220Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6058487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6058594Z output = self.layer_2(output) 2025-08-26T20:44:24.6058597Z 2025-08-26T20:44:24.6058708Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6058918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6058997Z return mod(**inputs) 2025-08-26T20:44:24.6059286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6059384Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6059655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6059731Z outputs = layer_module( 2025-08-26T20:44:24.6060008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6060084Z outputs = self.rel_attn( 2025-08-26T20:44:24.6060356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6060456Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6060460Z 2025-08-26T20:44:24.6060571Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6060771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6060838Z return mod(**inputs) 2025-08-26T20:44:24.6061105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6061189Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6061449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6061517Z outputs = layer_module( 2025-08-26T20:44:24.6061773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6061849Z outputs = self.rel_attn( 2025-08-26T20:44:24.6062115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6062230Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6062235Z 2025-08-26T20:44:24.6062343Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6062582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6062650Z return mod(**inputs) 2025-08-26T20:44:24.6062920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6063012Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6063272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6063351Z outputs = layer_module( 2025-08-26T20:44:24.6063610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6063681Z outputs = self.rel_attn( 2025-08-26T20:44:24.6063952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6064029Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6064305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6064438Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6064442Z 2025-08-26T20:44:24.6064551Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6064764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6064830Z return mod(**inputs) 2025-08-26T20:44:24.6065092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6065177Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6065438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6065523Z outputs = layer_module( 2025-08-26T20:44:24.6065780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6065858Z outputs = self.rel_attn( 2025-08-26T20:44:24.6066113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6066254Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6066259Z 2025-08-26T20:44:24.6066362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6066567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6066633Z return mod(**inputs) 2025-08-26T20:44:24.6066890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6066982Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6067239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6067316Z outputs = layer_module( 2025-08-26T20:44:24.6067571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6067641Z outputs = self.rel_attn( 2025-08-26T20:44:24.6067902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6067979Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6068262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6068396Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6068401Z 2025-08-26T20:44:24.6068503Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6068730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6068798Z return mod(**inputs) 2025-08-26T20:44:24.6069096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6069181Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6069453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6069528Z outputs = layer_module( 2025-08-26T20:44:24.6069793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6069876Z outputs = self.rel_attn( 2025-08-26T20:44:24.6070144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6070256Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6070259Z 2025-08-26T20:44:24.6070373Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6070576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6070653Z return mod(**inputs) 2025-08-26T20:44:24.6070908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6071018Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6071280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6071359Z outputs = layer_module( 2025-08-26T20:44:24.6071630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6071722Z outputs = self.rel_attn( 2025-08-26T20:44:24.6072002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6072080Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6072376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6072509Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6072515Z 2025-08-26T20:44:24.6072624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6072842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6072911Z return mod(**inputs) 2025-08-26T20:44:24.6073187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6073278Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6073547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6073627Z outputs = layer_module( 2025-08-26T20:44:24.6073898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6073978Z outputs = self.rel_attn( 2025-08-26T20:44:24.6074248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6074354Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6074647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6074768Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6074772Z 2025-08-26T20:44:24.6074890Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6075099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6075199Z return mod(**inputs) 2025-08-26T20:44:24.6075472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6075587Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6075877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6075954Z outputs = layer_module( 2025-08-26T20:44:24.6076255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6076328Z outputs = self.rel_attn( 2025-08-26T20:44:24.6076616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6076714Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6077031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6077157Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6077161Z 2025-08-26T20:44:24.6077271Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6077487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6077576Z return mod(**inputs) 2025-08-26T20:44:24.6077867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6077962Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6078257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6078355Z outputs = layer_module( 2025-08-26T20:44:24.6078639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6078865Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6079155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6079238Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6079608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6079693Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6079991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6080073Z output = self.layer_1(output) 2025-08-26T20:44:24.6080081Z 2025-08-26T20:44:24.6080195Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6080424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6080497Z return mod(**inputs) 2025-08-26T20:44:24.6080793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6080882Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6081183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6081257Z outputs = layer_module( 2025-08-26T20:44:24.6081538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6081766Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6082046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6082158Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6082456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6082551Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6082834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6082932Z output = self.activation_function(output) 2025-08-26T20:44:24.6083170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6083245Z return self.act(input) 2025-08-26T20:44:24.6083249Z 2025-08-26T20:44:24.6083364Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6083577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6083651Z return mod(**inputs) 2025-08-26T20:44:24.6083952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6084040Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6084343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6084415Z outputs = layer_module( 2025-08-26T20:44:24.6084732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6084960Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6085256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6085362Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6085662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6085739Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6086046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6086124Z output = self.layer_2(output) 2025-08-26T20:44:24.6086128Z 2025-08-26T20:44:24.6086249Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6086462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6086538Z return mod(**inputs) 2025-08-26T20:44:24.6086810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6086896Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6087186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6087256Z outputs = layer_module( 2025-08-26T20:44:24.6087518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6087589Z outputs = self.rel_attn( 2025-08-26T20:44:24.6087849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6087959Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6087963Z 2025-08-26T20:44:24.6088067Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6088273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6088340Z return mod(**inputs) 2025-08-26T20:44:24.6088604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6088689Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6088962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6089042Z outputs = layer_module( 2025-08-26T20:44:24.6089312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6089391Z outputs = self.rel_attn( 2025-08-26T20:44:24.6089646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6089755Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6089759Z 2025-08-26T20:44:24.6089875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6090086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6090168Z return mod(**inputs) 2025-08-26T20:44:24.6090441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6090523Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6090786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6090855Z outputs = layer_module( 2025-08-26T20:44:24.6091142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6091215Z outputs = self.rel_attn( 2025-08-26T20:44:24.6091487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6091565Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6091870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6092012Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6092016Z 2025-08-26T20:44:24.6092120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6092326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6092393Z return mod(**inputs) 2025-08-26T20:44:24.6092648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6092741Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6092996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6093069Z outputs = layer_module( 2025-08-26T20:44:24.6093321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6093398Z outputs = self.rel_attn( 2025-08-26T20:44:24.6093651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6093786Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6093790Z 2025-08-26T20:44:24.6093900Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6094098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6094171Z return mod(**inputs) 2025-08-26T20:44:24.6094425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6094507Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6094770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6094845Z outputs = layer_module( 2025-08-26T20:44:24.6095137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6095212Z outputs = self.rel_attn( 2025-08-26T20:44:24.6095509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6095590Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6095887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6096034Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6096038Z 2025-08-26T20:44:24.6096146Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6096542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6096622Z return mod(**inputs) 2025-08-26T20:44:24.6096895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6096994Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6097285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6097368Z outputs = layer_module( 2025-08-26T20:44:24.6097687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6097761Z outputs = self.rel_attn( 2025-08-26T20:44:24.6098040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6098149Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6098153Z 2025-08-26T20:44:24.6098296Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6098506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6098584Z return mod(**inputs) 2025-08-26T20:44:24.6098870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6098960Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6099255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6099330Z outputs = layer_module( 2025-08-26T20:44:24.6099612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6099686Z outputs = self.rel_attn( 2025-08-26T20:44:24.6099968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6100056Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6100348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6100491Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6100495Z 2025-08-26T20:44:24.6100604Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6100821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6100893Z return mod(**inputs) 2025-08-26T20:44:24.6101173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6101268Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6101551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6101632Z outputs = layer_module( 2025-08-26T20:44:24.6101954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6102025Z outputs = self.rel_attn( 2025-08-26T20:44:24.6102318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6102410Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6102695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6102812Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6102815Z 2025-08-26T20:44:24.6102924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6103124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6103192Z return mod(**inputs) 2025-08-26T20:44:24.6103456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6103540Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6103807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6103875Z outputs = layer_module( 2025-08-26T20:44:24.6104132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6104231Z outputs = self.rel_attn( 2025-08-26T20:44:24.6104489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6104589Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6104863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6104996Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6105007Z 2025-08-26T20:44:24.6105112Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6105311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6105388Z return mod(**inputs) 2025-08-26T20:44:24.6105644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6105735Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6105989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6106058Z outputs = layer_module( 2025-08-26T20:44:24.6106346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6106569Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6106855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6106937Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6107223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6107310Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6107579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6107666Z output = self.layer_1(output) 2025-08-26T20:44:24.6107670Z 2025-08-26T20:44:24.6107780Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6107998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6108068Z return mod(**inputs) 2025-08-26T20:44:24.6108355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6108451Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6108738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6108819Z outputs = layer_module( 2025-08-26T20:44:24.6109096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6109317Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6109603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6109687Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6109968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6110044Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6110320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6110414Z output = self.activation_function(output) 2025-08-26T20:44:24.6110639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6110739Z return self.act(input) 2025-08-26T20:44:24.6110742Z 2025-08-26T20:44:24.6110850Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6111067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6111138Z return mod(**inputs) 2025-08-26T20:44:24.6111408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6111521Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6111791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6111870Z outputs = layer_module( 2025-08-26T20:44:24.6112138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6112367Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6112642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6112724Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6113001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6113079Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6113354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6113434Z output = self.layer_2(output) 2025-08-26T20:44:24.6113438Z 2025-08-26T20:44:24.6113549Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6113769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6113841Z return mod(**inputs) 2025-08-26T20:44:24.6114118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6114209Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6114483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6114558Z outputs = layer_module( 2025-08-26T20:44:24.6114825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6114925Z outputs = self.rel_attn( 2025-08-26T20:44:24.6115212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6115327Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6115331Z 2025-08-26T20:44:24.6115440Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6115653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6115733Z return mod(**inputs) 2025-08-26T20:44:24.6116007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6116102Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6116376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6116450Z outputs = layer_module( 2025-08-26T20:44:24.6116726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6116803Z outputs = self.rel_attn( 2025-08-26T20:44:24.6117080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6117208Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6117212Z 2025-08-26T20:44:24.6117329Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6117542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6117612Z return mod(**inputs) 2025-08-26T20:44:24.6117894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6118034Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6118330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6118403Z outputs = layer_module( 2025-08-26T20:44:24.6118684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6118767Z outputs = self.rel_attn( 2025-08-26T20:44:24.6119056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6119145Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6119523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6119685Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6119692Z 2025-08-26T20:44:24.6119802Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6120019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6120099Z return mod(**inputs) 2025-08-26T20:44:24.6120380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6120486Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6120775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6120848Z outputs = layer_module( 2025-08-26T20:44:24.6121126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6121200Z outputs = self.rel_attn( 2025-08-26T20:44:24.6121489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6121652Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6121657Z 2025-08-26T20:44:24.6121774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6122008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6122081Z return mod(**inputs) 2025-08-26T20:44:24.6122359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6122449Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6122740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6122813Z outputs = layer_module( 2025-08-26T20:44:24.6123090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6123172Z outputs = self.rel_attn( 2025-08-26T20:44:24.6123454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6123538Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6123825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6123984Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6123995Z 2025-08-26T20:44:24.6124104Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6124311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6124391Z return mod(**inputs) 2025-08-26T20:44:24.6124661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6124782Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6125062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6125135Z outputs = layer_module( 2025-08-26T20:44:24.6125409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6125483Z outputs = self.rel_attn( 2025-08-26T20:44:24.6125775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6125882Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6125886Z 2025-08-26T20:44:24.6125993Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6126208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6126281Z return mod(**inputs) 2025-08-26T20:44:24.6126559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6126649Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6126939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6127012Z outputs = layer_module( 2025-08-26T20:44:24.6127290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6127369Z outputs = self.rel_attn( 2025-08-26T20:44:24.6127650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6127735Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6128023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6128157Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6128178Z 2025-08-26T20:44:24.6128297Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6128524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6137209Z return mod(**inputs) 2025-08-26T20:44:24.6137780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6137911Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6138218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6138303Z outputs = layer_module( 2025-08-26T20:44:24.6138593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6138680Z outputs = self.rel_attn( 2025-08-26T20:44:24.6138957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6139072Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6139408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6139550Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6139655Z 2025-08-26T20:44:24.6139794Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6140011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6140093Z return mod(**inputs) 2025-08-26T20:44:24.6140360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6140489Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6140766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6140844Z outputs = layer_module( 2025-08-26T20:44:24.6141128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6141207Z outputs = self.rel_attn( 2025-08-26T20:44:24.6141485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6141595Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6141893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6142024Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6142031Z 2025-08-26T20:44:24.6142145Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6142371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6142444Z return mod(**inputs) 2025-08-26T20:44:24.6142722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6142818Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6143072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6143151Z outputs = layer_module( 2025-08-26T20:44:24.6143402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6143651Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6143923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6144014Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6144274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6144382Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6144716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6144796Z output = self.layer_1(output) 2025-08-26T20:44:24.6144801Z 2025-08-26T20:44:24.6144915Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6145122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6145196Z return mod(**inputs) 2025-08-26T20:44:24.6145451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6145539Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6145806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6145877Z outputs = layer_module( 2025-08-26T20:44:24.6146146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6146371Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6147177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6147263Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6147522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6147627Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6147879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6147978Z output = self.activation_function(output) 2025-08-26T20:44:24.6148210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6148291Z return self.act(input) 2025-08-26T20:44:24.6148295Z 2025-08-26T20:44:24.6148421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6148639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6148718Z return mod(**inputs) 2025-08-26T20:44:24.6148991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6149079Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6149362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6149437Z outputs = layer_module( 2025-08-26T20:44:24.6149715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6149938Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6150220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6150307Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6150578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6150664Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6150931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6151020Z output = self.layer_2(output) 2025-08-26T20:44:24.6151024Z 2025-08-26T20:44:24.6151137Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6151348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6151444Z return mod(**inputs) 2025-08-26T20:44:24.6151755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6151853Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6152125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6152200Z outputs = layer_module( 2025-08-26T20:44:24.6152472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6152550Z outputs = self.rel_attn( 2025-08-26T20:44:24.6152824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6152932Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6152936Z 2025-08-26T20:44:24.6153058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6153271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6153359Z return mod(**inputs) 2025-08-26T20:44:24.6153638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6153725Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6154004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6154076Z outputs = layer_module( 2025-08-26T20:44:24.6154367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6154449Z outputs = self.rel_attn( 2025-08-26T20:44:24.6154723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6154845Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6154849Z 2025-08-26T20:44:24.6154960Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6155187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6155258Z return mod(**inputs) 2025-08-26T20:44:24.6155533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6155629Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6155903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6155981Z outputs = layer_module( 2025-08-26T20:44:24.6156250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6156324Z outputs = self.rel_attn( 2025-08-26T20:44:24.6156605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6156691Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6156987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6157136Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6157140Z 2025-08-26T20:44:24.6157259Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6157473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6157543Z return mod(**inputs) 2025-08-26T20:44:24.6157822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6157908Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6158239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6158317Z outputs = layer_module( 2025-08-26T20:44:24.6158586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6158672Z outputs = self.rel_attn( 2025-08-26T20:44:24.6158940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6159093Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6159099Z 2025-08-26T20:44:24.6159210Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6159736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6159820Z return mod(**inputs) 2025-08-26T20:44:24.6160106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6160208Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6160518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6160611Z outputs = layer_module( 2025-08-26T20:44:24.6160892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6160967Z outputs = self.rel_attn( 2025-08-26T20:44:24.6161270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6161351Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6161649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6161793Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6161799Z 2025-08-26T20:44:24.6161909Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6162137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6162210Z return mod(**inputs) 2025-08-26T20:44:24.6162489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6162582Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6162876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6162951Z outputs = layer_module( 2025-08-26T20:44:24.6163251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6163332Z outputs = self.rel_attn( 2025-08-26T20:44:24.6163614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6163733Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6163737Z 2025-08-26T20:44:24.6163846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6164057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6164136Z return mod(**inputs) 2025-08-26T20:44:24.6164405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6164501Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6164781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6164851Z outputs = layer_module( 2025-08-26T20:44:24.6165169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6165246Z outputs = self.rel_attn( 2025-08-26T20:44:24.6165542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6165618Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6165916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6166053Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6166059Z 2025-08-26T20:44:24.6166175Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6166384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6166450Z return mod(**inputs) 2025-08-26T20:44:24.6166716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6166802Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6167054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6167147Z outputs = layer_module( 2025-08-26T20:44:24.6167400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6167476Z outputs = self.rel_attn( 2025-08-26T20:44:24.6167730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6167844Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6168119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6168240Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6168244Z 2025-08-26T20:44:24.6168355Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6168554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6168628Z return mod(**inputs) 2025-08-26T20:44:24.6168883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6168966Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6169231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6169300Z outputs = layer_module( 2025-08-26T20:44:24.6169556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6169625Z outputs = self.rel_attn( 2025-08-26T20:44:24.6169882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6169979Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6170265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6170389Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6170393Z 2025-08-26T20:44:24.6170496Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6170713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6170785Z return mod(**inputs) 2025-08-26T20:44:24.6171054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6171148Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6171439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6171538Z outputs = layer_module( 2025-08-26T20:44:24.6171809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6172034Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6172318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6172401Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6172687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6172761Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6173020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6173096Z output = self.layer_1(output) 2025-08-26T20:44:24.6173101Z 2025-08-26T20:44:24.6173210Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6173435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6173502Z return mod(**inputs) 2025-08-26T20:44:24.6173769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6173858Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6174128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6174227Z outputs = layer_module( 2025-08-26T20:44:24.6174503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6174739Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6175034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6175129Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6175412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6175491Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6175787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6175883Z output = self.activation_function(output) 2025-08-26T20:44:24.6176122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6176198Z return self.act(input) 2025-08-26T20:44:24.6176202Z 2025-08-26T20:44:24.6176318Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6176531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6176599Z return mod(**inputs) 2025-08-26T20:44:24.6176869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6176950Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6177217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6177289Z outputs = layer_module( 2025-08-26T20:44:24.6177549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6177778Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6178093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6178185Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6178457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6178534Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6178811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6178893Z output = self.layer_2(output) 2025-08-26T20:44:24.6178898Z 2025-08-26T20:44:24.6179021Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6179241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6179321Z return mod(**inputs) 2025-08-26T20:44:24.6179628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6179716Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6179998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6180091Z outputs = layer_module( 2025-08-26T20:44:24.6180367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6180443Z outputs = self.rel_attn( 2025-08-26T20:44:24.6180714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6180860Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6180864Z 2025-08-26T20:44:24.6180973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6181193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6181265Z return mod(**inputs) 2025-08-26T20:44:24.6181547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6181646Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6181931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6182011Z outputs = layer_module( 2025-08-26T20:44:24.6182292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6182376Z outputs = self.rel_attn( 2025-08-26T20:44:24.6182656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6182766Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6182770Z 2025-08-26T20:44:24.6182888Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6183103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6183183Z return mod(**inputs) 2025-08-26T20:44:24.6183464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6183552Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6183831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6183903Z outputs = layer_module( 2025-08-26T20:44:24.6184193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6184266Z outputs = self.rel_attn( 2025-08-26T20:44:24.6184561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6184657Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6184963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6185118Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6185122Z 2025-08-26T20:44:24.6185231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6185449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6185518Z return mod(**inputs) 2025-08-26T20:44:24.6185789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6185885Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6186165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6186246Z outputs = layer_module( 2025-08-26T20:44:24.6186534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6186631Z outputs = self.rel_attn( 2025-08-26T20:44:24.6186920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6187062Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6187066Z 2025-08-26T20:44:24.6187183Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6187416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6187493Z return mod(**inputs) 2025-08-26T20:44:24.6187778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6187866Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6188160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6188235Z outputs = layer_module( 2025-08-26T20:44:24.6188520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6188596Z outputs = self.rel_attn( 2025-08-26T20:44:24.6188880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6188965Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6189258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6189408Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6189412Z 2025-08-26T20:44:24.6189519Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6189753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6189824Z return mod(**inputs) 2025-08-26T20:44:24.6190119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6190215Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6190497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6190576Z outputs = layer_module( 2025-08-26T20:44:24.6190866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6190940Z outputs = self.rel_attn( 2025-08-26T20:44:24.6191262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6191392Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6191396Z 2025-08-26T20:44:24.6191534Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6191756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6191839Z return mod(**inputs) 2025-08-26T20:44:24.6192116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6192207Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6192501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6192576Z outputs = layer_module( 2025-08-26T20:44:24.6192848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6192925Z outputs = self.rel_attn( 2025-08-26T20:44:24.6193199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6193285Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6193590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6193732Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6193736Z 2025-08-26T20:44:24.6193846Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6194066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6194154Z return mod(**inputs) 2025-08-26T20:44:24.6194972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6195069Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6195343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6195427Z outputs = layer_module( 2025-08-26T20:44:24.6195699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6195774Z outputs = self.rel_attn( 2025-08-26T20:44:24.6196053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6196151Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6196747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6196872Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6196876Z 2025-08-26T20:44:24.6196995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6197210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6197284Z return mod(**inputs) 2025-08-26T20:44:24.6197566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6197659Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6197946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6198020Z outputs = layer_module( 2025-08-26T20:44:24.6198296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6198385Z outputs = self.rel_attn( 2025-08-26T20:44:24.6198662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6198768Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6199210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6199338Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6199350Z 2025-08-26T20:44:24.6199523Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6199748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6199830Z return mod(**inputs) 2025-08-26T20:44:24.6200108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6200209Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6200486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6200561Z outputs = layer_module( 2025-08-26T20:44:24.6200850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6201078Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6201409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6201496Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6201782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6201902Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6202181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6202270Z output = self.layer_1(output) 2025-08-26T20:44:24.6202274Z 2025-08-26T20:44:24.6202386Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6202616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6202686Z return mod(**inputs) 2025-08-26T20:44:24.6202972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6203071Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6203354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6203437Z outputs = layer_module( 2025-08-26T20:44:24.6203719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6203949Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6204247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6204333Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6204621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6204703Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6204993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6205090Z output = self.activation_function(output) 2025-08-26T20:44:24.6205326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6205415Z return self.act(input) 2025-08-26T20:44:24.6205418Z 2025-08-26T20:44:24.6205530Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6205757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6205849Z return mod(**inputs) 2025-08-26T20:44:24.6206150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6206251Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6206532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6206614Z outputs = layer_module( 2025-08-26T20:44:24.6206898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6207133Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6207425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6207509Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6207800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6207881Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6208195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6208278Z output = self.layer_2(output) 2025-08-26T20:44:24.6208282Z 2025-08-26T20:44:24.6208393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6208618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6208708Z return mod(**inputs) 2025-08-26T20:44:24.6209014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6209104Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6209391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6209468Z outputs = layer_module( 2025-08-26T20:44:24.6209745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6209831Z outputs = self.rel_attn( 2025-08-26T20:44:24.6210107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6210225Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6210229Z 2025-08-26T20:44:24.6210341Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6210559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6210640Z return mod(**inputs) 2025-08-26T20:44:24.6210916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6211015Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6211300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6211376Z outputs = layer_module( 2025-08-26T20:44:24.6211661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6211737Z outputs = self.rel_attn( 2025-08-26T20:44:24.6212016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6212130Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6212134Z 2025-08-26T20:44:24.6212252Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6212469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6212541Z return mod(**inputs) 2025-08-26T20:44:24.6212860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6212955Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6213254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6213327Z outputs = layer_module( 2025-08-26T20:44:24.6213595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6213679Z outputs = self.rel_attn( 2025-08-26T20:44:24.6213946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6214034Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6214325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6214476Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6214480Z 2025-08-26T20:44:24.6214610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6214823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6214902Z return mod(**inputs) 2025-08-26T20:44:24.6215174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6215268Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6215555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6215626Z outputs = layer_module( 2025-08-26T20:44:24.6215903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6215978Z outputs = self.rel_attn( 2025-08-26T20:44:24.6216254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6216397Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6216401Z 2025-08-26T20:44:24.6216518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6216727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6216796Z return mod(**inputs) 2025-08-26T20:44:24.6217074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6217163Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6217439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6217511Z outputs = layer_module( 2025-08-26T20:44:24.6217781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6217862Z outputs = self.rel_attn( 2025-08-26T20:44:24.6218130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6218214Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6218498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6218639Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6218650Z 2025-08-26T20:44:24.6218762Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6218971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6219049Z return mod(**inputs) 2025-08-26T20:44:24.6219356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6219455Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6219725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6219796Z outputs = layer_module( 2025-08-26T20:44:24.6220070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6220143Z outputs = self.rel_attn( 2025-08-26T20:44:24.6220421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6220529Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6220533Z 2025-08-26T20:44:24.6220641Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6220862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6220932Z return mod(**inputs) 2025-08-26T20:44:24.6221227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6221313Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6221588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6221659Z outputs = layer_module( 2025-08-26T20:44:24.6221928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6222032Z outputs = self.rel_attn( 2025-08-26T20:44:24.6222303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6222390Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6222680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6222814Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6222818Z 2025-08-26T20:44:24.6222935Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6223146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6223224Z return mod(**inputs) 2025-08-26T20:44:24.6223494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6223584Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6223861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6223934Z outputs = layer_module( 2025-08-26T20:44:24.6224210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6224283Z outputs = self.rel_attn( 2025-08-26T20:44:24.6224561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6224658Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6224948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6225076Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6225081Z 2025-08-26T20:44:24.6225190Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6225408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6225477Z return mod(**inputs) 2025-08-26T20:44:24.6225772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6225869Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6226142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6226225Z outputs = layer_module( 2025-08-26T20:44:24.6226492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6226573Z outputs = self.rel_attn( 2025-08-26T20:44:24.6226844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6226942Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6227243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6227363Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6227368Z 2025-08-26T20:44:24.6227485Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6227724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6227795Z return mod(**inputs) 2025-08-26T20:44:24.6228075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6228217Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6228592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6228711Z outputs = layer_module( 2025-08-26T20:44:24.6229198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6229444Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6229768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6246531Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6246982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6247075Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6247370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6247479Z output = self.layer_1(output) 2025-08-26T20:44:24.6247486Z 2025-08-26T20:44:24.6247612Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6247849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6247926Z return mod(**inputs) 2025-08-26T20:44:24.6248229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6248333Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6248613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6248698Z outputs = layer_module( 2025-08-26T20:44:24.6248967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6249203Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6249491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6249577Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6249991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6250075Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6250358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6250458Z output = self.activation_function(output) 2025-08-26T20:44:24.6250700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6250777Z return self.act(input) 2025-08-26T20:44:24.6250781Z 2025-08-26T20:44:24.6250901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6251131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6251206Z return mod(**inputs) 2025-08-26T20:44:24.6251489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6251586Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6251867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6251985Z outputs = layer_module( 2025-08-26T20:44:24.6252263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6252498Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6252783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6252903Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6253178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6253258Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6253543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6253623Z output = self.layer_2(output) 2025-08-26T20:44:24.6253628Z 2025-08-26T20:44:24.6253750Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6253974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6254047Z return mod(**inputs) 2025-08-26T20:44:24.6254334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6254430Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6254720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6254796Z outputs = layer_module( 2025-08-26T20:44:24.6255075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6255164Z outputs = self.rel_attn( 2025-08-26T20:44:24.6255448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6255568Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6255572Z 2025-08-26T20:44:24.6255685Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6255906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6255979Z return mod(**inputs) 2025-08-26T20:44:24.6256250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6256345Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6256630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6256731Z outputs = layer_module( 2025-08-26T20:44:24.6257004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6257082Z outputs = self.rel_attn( 2025-08-26T20:44:24.6257360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6257474Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6257478Z 2025-08-26T20:44:24.6257597Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6257814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6257892Z return mod(**inputs) 2025-08-26T20:44:24.6258163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6258255Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6258538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6258629Z outputs = layer_module( 2025-08-26T20:44:24.6258914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6258987Z outputs = self.rel_attn( 2025-08-26T20:44:24.6259267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6259376Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6259672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6259831Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6259836Z 2025-08-26T20:44:24.6259949Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6260172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6260245Z return mod(**inputs) 2025-08-26T20:44:24.6260515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6260615Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6260886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6260969Z outputs = layer_module( 2025-08-26T20:44:24.6261240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6261314Z outputs = self.rel_attn( 2025-08-26T20:44:24.6261590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6261740Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6261744Z 2025-08-26T20:44:24.6261863Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6262077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6262149Z return mod(**inputs) 2025-08-26T20:44:24.6262430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6262519Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6262802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6262875Z outputs = layer_module( 2025-08-26T20:44:24.6263152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6263241Z outputs = self.rel_attn( 2025-08-26T20:44:24.6263530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6263620Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6263915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6264064Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6264068Z 2025-08-26T20:44:24.6264178Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6264397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6264475Z return mod(**inputs) 2025-08-26T20:44:24.6264754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6264847Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6265108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6265219Z outputs = layer_module( 2025-08-26T20:44:24.6265476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6265544Z outputs = self.rel_attn( 2025-08-26T20:44:24.6265804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6265924Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6265928Z 2025-08-26T20:44:24.6266038Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6266238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6266305Z return mod(**inputs) 2025-08-26T20:44:24.6266573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6266657Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6266925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6266993Z outputs = layer_module( 2025-08-26T20:44:24.6267259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6267340Z outputs = self.rel_attn( 2025-08-26T20:44:24.6267614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6267701Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6267996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6268140Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6268146Z 2025-08-26T20:44:24.6268256Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6268471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6268551Z return mod(**inputs) 2025-08-26T20:44:24.6268836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6268933Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6269213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6269291Z outputs = layer_module( 2025-08-26T20:44:24.6269577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6269653Z outputs = self.rel_attn( 2025-08-26T20:44:24.6270009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6270115Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6270426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6270562Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6270567Z 2025-08-26T20:44:24.6270678Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6270899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6270972Z return mod(**inputs) 2025-08-26T20:44:24.6271253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6271344Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6271617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6271718Z outputs = layer_module( 2025-08-26T20:44:24.6271988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6272067Z outputs = self.rel_attn( 2025-08-26T20:44:24.6272336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6272440Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6272756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6272876Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6272879Z 2025-08-26T20:44:24.6273001Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6273217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6273296Z return mod(**inputs) 2025-08-26T20:44:24.6273574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6273662Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6273943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6274016Z outputs = layer_module( 2025-08-26T20:44:24.6274294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6274522Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6274811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6274899Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6275171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6275260Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6275530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6275616Z output = self.layer_1(output) 2025-08-26T20:44:24.6275620Z 2025-08-26T20:44:24.6275729Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6275943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6276021Z return mod(**inputs) 2025-08-26T20:44:24.6276293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6276407Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6276700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6276777Z outputs = layer_module( 2025-08-26T20:44:24.6277055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6277276Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6277561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6277647Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6277927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6278004Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6278279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6278383Z output = self.activation_function(output) 2025-08-26T20:44:24.6278640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6278728Z return self.act(input) 2025-08-26T20:44:24.6278732Z 2025-08-26T20:44:24.6278844Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6279062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6279161Z return mod(**inputs) 2025-08-26T20:44:24.6279574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6279685Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6279968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6280058Z outputs = layer_module( 2025-08-26T20:44:24.6280340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6280574Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6280873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6280958Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6281259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6281336Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6281606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6281698Z output = self.layer_2(output) 2025-08-26T20:44:24.6281702Z 2025-08-26T20:44:24.6281815Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6282038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6282109Z return mod(**inputs) 2025-08-26T20:44:24.6282389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6282478Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6282747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6282830Z outputs = layer_module( 2025-08-26T20:44:24.6283098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6283182Z outputs = self.rel_attn( 2025-08-26T20:44:24.6283493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6283606Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6283612Z 2025-08-26T20:44:24.6283730Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6283947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6284026Z return mod(**inputs) 2025-08-26T20:44:24.6284299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6284398Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6284672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6284745Z outputs = layer_module( 2025-08-26T20:44:24.6285026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6285103Z outputs = self.rel_attn( 2025-08-26T20:44:24.6285381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6285511Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6285515Z 2025-08-26T20:44:24.6285625Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6285846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6285935Z return mod(**inputs) 2025-08-26T20:44:24.6286213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6286301Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6286596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6286675Z outputs = layer_module( 2025-08-26T20:44:24.6286955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6287035Z outputs = self.rel_attn( 2025-08-26T20:44:24.6287331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6287415Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6287719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6287865Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6287869Z 2025-08-26T20:44:24.6287986Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6288200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6288277Z return mod(**inputs) 2025-08-26T20:44:24.6288550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6288640Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6288940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6289010Z outputs = layer_module( 2025-08-26T20:44:24.6289304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6289381Z outputs = self.rel_attn( 2025-08-26T20:44:24.6289663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6289810Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6289814Z 2025-08-26T20:44:24.6289954Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6290203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6290278Z return mod(**inputs) 2025-08-26T20:44:24.6290572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6290661Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6290944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6291028Z outputs = layer_module( 2025-08-26T20:44:24.6291302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6291379Z outputs = self.rel_attn( 2025-08-26T20:44:24.6291637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6291709Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6291993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6292144Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6292148Z 2025-08-26T20:44:24.6292263Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6292461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6292537Z return mod(**inputs) 2025-08-26T20:44:24.6292817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6292901Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6293167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6293238Z outputs = layer_module( 2025-08-26T20:44:24.6293504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6293574Z outputs = self.rel_attn( 2025-08-26T20:44:24.6293830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6293939Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6293943Z 2025-08-26T20:44:24.6294046Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6294257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6294323Z return mod(**inputs) 2025-08-26T20:44:24.6294594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6294678Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6294940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6295021Z outputs = layer_module( 2025-08-26T20:44:24.6295279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6295357Z outputs = self.rel_attn( 2025-08-26T20:44:24.6295614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6295687Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6295973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6296101Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6296105Z 2025-08-26T20:44:24.6296389Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6296725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6296802Z return mod(**inputs) 2025-08-26T20:44:24.6297062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6297146Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6297412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6297482Z outputs = layer_module( 2025-08-26T20:44:24.6297748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6297819Z outputs = self.rel_attn( 2025-08-26T20:44:24.6298074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6298176Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6298456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6298615Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6298619Z 2025-08-26T20:44:24.6298726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6298947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6299020Z return mod(**inputs) 2025-08-26T20:44:24.6299309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6299442Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6299725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6299815Z outputs = layer_module( 2025-08-26T20:44:24.6300089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6300173Z outputs = self.rel_attn( 2025-08-26T20:44:24.6300444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6300541Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6300841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6300973Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6300977Z 2025-08-26T20:44:24.6301088Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6301287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6301356Z return mod(**inputs) 2025-08-26T20:44:24.6301622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6301707Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6301974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6302043Z outputs = layer_module( 2025-08-26T20:44:24.6302316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6302543Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6302826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6302921Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6303217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6303321Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6303596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6303675Z output = self.layer_1(output) 2025-08-26T20:44:24.6303679Z 2025-08-26T20:44:24.6303795Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6304010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6304091Z return mod(**inputs) 2025-08-26T20:44:24.6304366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6304463Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6304736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6304811Z outputs = layer_module( 2025-08-26T20:44:24.6305090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6305334Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6305602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6305681Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6305937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6306038Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6306292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6306394Z output = self.activation_function(output) 2025-08-26T20:44:24.6306611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6306685Z return self.act(input) 2025-08-26T20:44:24.6306695Z 2025-08-26T20:44:24.6306799Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6307002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6307083Z return mod(**inputs) 2025-08-26T20:44:24.6307353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6307451Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6307720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6307793Z outputs = layer_module( 2025-08-26T20:44:24.6308072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6308296Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6308580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6308661Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6308933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6309019Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6309296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6309383Z output = self.layer_2(output) 2025-08-26T20:44:24.6309387Z 2025-08-26T20:44:24.6309501Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6309761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6309834Z return mod(**inputs) 2025-08-26T20:44:24.6310117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6310217Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6310496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6310579Z outputs = layer_module( 2025-08-26T20:44:24.6310857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6310936Z outputs = self.rel_attn( 2025-08-26T20:44:24.6311224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6311336Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6311340Z 2025-08-26T20:44:24.6311463Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6311690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6311787Z return mod(**inputs) 2025-08-26T20:44:24.6312064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6312151Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6312437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6312529Z outputs = layer_module( 2025-08-26T20:44:24.6312805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6312879Z outputs = self.rel_attn( 2025-08-26T20:44:24.6313149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6313269Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6313275Z 2025-08-26T20:44:24.6313384Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6313603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6313673Z return mod(**inputs) 2025-08-26T20:44:24.6313946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6314045Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6314319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6314403Z outputs = layer_module( 2025-08-26T20:44:24.6314676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6314760Z outputs = self.rel_attn( 2025-08-26T20:44:24.6315031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6315112Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6315409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6315554Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6315558Z 2025-08-26T20:44:24.6315677Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6315886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6315957Z return mod(**inputs) 2025-08-26T20:44:24.6316266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6316372Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6316652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6316727Z outputs = layer_module( 2025-08-26T20:44:24.6317005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6317079Z outputs = self.rel_attn( 2025-08-26T20:44:24.6317382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6317539Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6317543Z 2025-08-26T20:44:24.6317654Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6317881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6317955Z return mod(**inputs) 2025-08-26T20:44:24.6318240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6318389Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6318665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6318746Z outputs = layer_module( 2025-08-26T20:44:24.6319031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6319131Z outputs = self.rel_attn( 2025-08-26T20:44:24.6319484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6319572Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6319885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6320031Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6320036Z 2025-08-26T20:44:24.6320168Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6320380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6320450Z return mod(**inputs) 2025-08-26T20:44:24.6320734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6320824Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6321107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6321179Z outputs = layer_module( 2025-08-26T20:44:24.6321452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6321539Z outputs = self.rel_attn( 2025-08-26T20:44:24.6321819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6321945Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6321949Z 2025-08-26T20:44:24.6322059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6322285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6322356Z return mod(**inputs) 2025-08-26T20:44:24.6322630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6322729Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6323002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6323105Z outputs = layer_module( 2025-08-26T20:44:24.6323393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6323470Z outputs = self.rel_attn( 2025-08-26T20:44:24.6323746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6323824Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6324115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6324251Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6324255Z 2025-08-26T20:44:24.6324370Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6324581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6324649Z return mod(**inputs) 2025-08-26T20:44:24.6324932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6325039Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6325316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6325388Z outputs = layer_module( 2025-08-26T20:44:24.6325654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6325738Z outputs = self.rel_attn( 2025-08-26T20:44:24.6326037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6326144Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6326447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6326569Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6326573Z 2025-08-26T20:44:24.6326679Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6326878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6326956Z return mod(**inputs) 2025-08-26T20:44:24.6327215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6327313Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6327589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6327662Z outputs = layer_module( 2025-08-26T20:44:24.6327940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6328016Z outputs = self.rel_attn( 2025-08-26T20:44:24.6328306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6328406Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6328707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6328834Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6328838Z 2025-08-26T20:44:24.6328947Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6329172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6329240Z return mod(**inputs) 2025-08-26T20:44:24.6329505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6329608Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6329886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6329966Z outputs = layer_module( 2025-08-26T20:44:24.6330222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6330440Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6330710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6330792Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6331085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6331162Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6331446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6331525Z output = self.layer_1(output) 2025-08-26T20:44:24.6331551Z 2025-08-26T20:44:24.6331668Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6331881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6331951Z return mod(**inputs) 2025-08-26T20:44:24.6332243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6332379Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6332672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6332745Z outputs = layer_module( 2025-08-26T20:44:24.6333016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6333252Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6333539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6333625Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6333885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6333964Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6334225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6334315Z output = self.activation_function(output) 2025-08-26T20:44:24.6334542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6334615Z return self.act(input) 2025-08-26T20:44:24.6334618Z 2025-08-26T20:44:24.6334732Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6334934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6335000Z return mod(**inputs) 2025-08-26T20:44:24.6335270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6335355Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6335622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6335695Z outputs = layer_module( 2025-08-26T20:44:24.6335963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6336193Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6336475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6336565Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6336823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6336905Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6337162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6337238Z output = self.layer_2(output) 2025-08-26T20:44:24.6337242Z 2025-08-26T20:44:24.6337353Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6337555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6337629Z return mod(**inputs) 2025-08-26T20:44:24.6337888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6337981Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6338257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6338325Z outputs = layer_module( 2025-08-26T20:44:24.6338590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6338661Z outputs = self.rel_attn( 2025-08-26T20:44:24.6338951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6339052Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6339056Z 2025-08-26T20:44:24.6339161Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6339370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6339440Z return mod(**inputs) 2025-08-26T20:44:24.6339701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6339786Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6340042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6340118Z outputs = layer_module( 2025-08-26T20:44:24.6340371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6340451Z outputs = self.rel_attn( 2025-08-26T20:44:24.6340709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6340818Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6340824Z 2025-08-26T20:44:24.6340928Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6341129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6341204Z return mod(**inputs) 2025-08-26T20:44:24.6341461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6341554Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6341809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6341879Z outputs = layer_module( 2025-08-26T20:44:24.6342143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6342213Z outputs = self.rel_attn( 2025-08-26T20:44:24.6342493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6342585Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6342869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6343006Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6343010Z 2025-08-26T20:44:24.6343114Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6343324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6343393Z return mod(**inputs) 2025-08-26T20:44:24.6343657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6343742Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6343999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6344076Z outputs = layer_module( 2025-08-26T20:44:24.6344333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6344432Z outputs = self.rel_attn( 2025-08-26T20:44:24.6344694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6344832Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6344842Z 2025-08-26T20:44:24.6344973Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6345167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6345241Z return mod(**inputs) 2025-08-26T20:44:24.6345493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6345585Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6345836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6345906Z outputs = layer_module( 2025-08-26T20:44:24.6346163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6346230Z outputs = self.rel_attn( 2025-08-26T20:44:24.6346484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6346558Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6346826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6346966Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6346969Z 2025-08-26T20:44:24.6347073Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6347275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6347342Z return mod(**inputs) 2025-08-26T20:44:24.6347608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6347694Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6347949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6348030Z outputs = layer_module( 2025-08-26T20:44:24.6348283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6348358Z outputs = self.rel_attn( 2025-08-26T20:44:24.6348631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6348749Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6348755Z 2025-08-26T20:44:24.6348866Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6349065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6349140Z return mod(**inputs) 2025-08-26T20:44:24.6349407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6349504Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6349775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6349848Z outputs = layer_module( 2025-08-26T20:44:24.6350124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6350199Z outputs = self.rel_attn( 2025-08-26T20:44:24.6350479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6350579Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6350869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6351013Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6351016Z 2025-08-26T20:44:24.6351126Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6351371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6351437Z return mod(**inputs) 2025-08-26T20:44:24.6351692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6351786Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6352048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6352126Z outputs = layer_module( 2025-08-26T20:44:24.6352382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6352459Z outputs = self.rel_attn( 2025-08-26T20:44:24.6352724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6352823Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6353124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6353246Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6353250Z 2025-08-26T20:44:24.6353367Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6353584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6353657Z return mod(**inputs) 2025-08-26T20:44:24.6353935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6354023Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6354299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6354373Z outputs = layer_module( 2025-08-26T20:44:24.6354652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6354727Z outputs = self.rel_attn( 2025-08-26T20:44:24.6355005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6355149Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6355453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6355585Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6355589Z 2025-08-26T20:44:24.6355700Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6355918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6355997Z return mod(**inputs) 2025-08-26T20:44:24.6356278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6356376Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6356653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6356737Z outputs = layer_module( 2025-08-26T20:44:24.6357017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6357275Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6357566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6357650Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6357930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6358033Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6358312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6358399Z output = self.layer_1(output) 2025-08-26T20:44:24.6358405Z 2025-08-26T20:44:24.6358516Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6358740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6358811Z return mod(**inputs) 2025-08-26T20:44:24.6359110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6359200Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6359572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6359665Z outputs = layer_module( 2025-08-26T20:44:24.6359946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6360188Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6360482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6360571Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6360861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6360942Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6361236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6361335Z output = self.activation_function(output) 2025-08-26T20:44:24.6361588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6361667Z return self.act(input) 2025-08-26T20:44:24.6361671Z 2025-08-26T20:44:24.6361786Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6362050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6362143Z return mod(**inputs) 2025-08-26T20:44:24.6362435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6362530Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6362809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6362894Z outputs = layer_module( 2025-08-26T20:44:24.6363170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6363409Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6363703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6363796Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6364080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6364189Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6364475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6364556Z output = self.layer_2(output) 2025-08-26T20:44:24.6364560Z 2025-08-26T20:44:24.6364681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6364919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6364993Z return mod(**inputs) 2025-08-26T20:44:24.6365283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6365373Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6365666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6365743Z outputs = layer_module( 2025-08-26T20:44:24.6366022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6366106Z outputs = self.rel_attn( 2025-08-26T20:44:24.6366385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6366503Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6366509Z 2025-08-26T20:44:24.6366621Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6366850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6366923Z return mod(**inputs) 2025-08-26T20:44:24.6367205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6367306Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6367589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6367672Z outputs = layer_module( 2025-08-26T20:44:24.6367951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6368028Z outputs = self.rel_attn( 2025-08-26T20:44:24.6368319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6368433Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6368437Z 2025-08-26T20:44:24.6368556Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6368796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6368893Z return mod(**inputs) 2025-08-26T20:44:24.6369177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6369271Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6369562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6369636Z outputs = layer_module( 2025-08-26T20:44:24.6369918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6369995Z outputs = self.rel_attn( 2025-08-26T20:44:24.6370285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6370373Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6370674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6370828Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6370851Z 2025-08-26T20:44:24.6370975Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6371185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6371264Z return mod(**inputs) 2025-08-26T20:44:24.6371535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6371660Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6371919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6371996Z outputs = layer_module( 2025-08-26T20:44:24.6372259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6372332Z outputs = self.rel_attn( 2025-08-26T20:44:24.6372606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6372744Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6372748Z 2025-08-26T20:44:24.6372860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6373063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6373133Z return mod(**inputs) 2025-08-26T20:44:24.6373405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6373497Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6373791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6373867Z outputs = layer_module( 2025-08-26T20:44:24.6374164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6374239Z outputs = self.rel_attn( 2025-08-26T20:44:24.6374505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6374589Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6374873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6375018Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6375022Z 2025-08-26T20:44:24.6375127Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6375353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6375452Z return mod(**inputs) 2025-08-26T20:44:24.6375708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6375802Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6376061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6376136Z outputs = layer_module( 2025-08-26T20:44:24.6376388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6376461Z outputs = self.rel_attn( 2025-08-26T20:44:24.6376735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6376843Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6376847Z 2025-08-26T20:44:24.6376965Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6377178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6377274Z return mod(**inputs) 2025-08-26T20:44:24.6377530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6377611Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6377868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6377954Z outputs = layer_module( 2025-08-26T20:44:24.6378202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6378277Z outputs = self.rel_attn( 2025-08-26T20:44:24.6378535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6378618Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6378891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6379027Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6379031Z 2025-08-26T20:44:24.6379136Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6379339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6379414Z return mod(**inputs) 2025-08-26T20:44:24.6379680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6379771Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6380027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6380097Z outputs = layer_module( 2025-08-26T20:44:24.6380370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6380440Z outputs = self.rel_attn( 2025-08-26T20:44:24.6380695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6380786Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6381065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6381181Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6381185Z 2025-08-26T20:44:24.6381289Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6381495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6381579Z return mod(**inputs) 2025-08-26T20:44:24.6381858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6381941Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6382197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6382275Z outputs = layer_module( 2025-08-26T20:44:24.6382526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6382605Z outputs = self.rel_attn( 2025-08-26T20:44:24.6382859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6382949Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6383236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6383353Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6383376Z 2025-08-26T20:44:24.6383490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6383689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6383762Z return mod(**inputs) 2025-08-26T20:44:24.6384027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6384124Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6384384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6384451Z outputs = layer_module( 2025-08-26T20:44:24.6384723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6384947Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6385253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6385332Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6385591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6385672Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6385943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6386027Z output = self.layer_1(output) 2025-08-26T20:44:24.6386030Z 2025-08-26T20:44:24.6386138Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6386352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6386433Z return mod(**inputs) 2025-08-26T20:44:24.6386711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6386807Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6387078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6387150Z outputs = layer_module( 2025-08-26T20:44:24.6387428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6387654Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6387943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6388047Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6388355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6388436Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6388703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6388805Z output = self.activation_function(output) 2025-08-26T20:44:24.6389032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6389117Z return self.act(input) 2025-08-26T20:44:24.6389121Z 2025-08-26T20:44:24.6389231Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6389445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6389524Z return mod(**inputs) 2025-08-26T20:44:24.6389797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6389892Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6390181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6390262Z outputs = layer_module( 2025-08-26T20:44:24.6390531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6390752Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6391058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6391138Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6391420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6391501Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6391766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6391853Z output = self.layer_2(output) 2025-08-26T20:44:24.6391857Z 2025-08-26T20:44:24.6391967Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6392187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6392258Z return mod(**inputs) 2025-08-26T20:44:24.6392537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6392626Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6392896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6392980Z outputs = layer_module( 2025-08-26T20:44:24.6393249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6393333Z outputs = self.rel_attn( 2025-08-26T20:44:24.6393604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6393711Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6393715Z 2025-08-26T20:44:24.6393832Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6394047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6394127Z return mod(**inputs) 2025-08-26T20:44:24.6394400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6394488Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6394799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6394877Z outputs = layer_module( 2025-08-26T20:44:24.6395159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6395234Z outputs = self.rel_attn( 2025-08-26T20:44:24.6395513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6395623Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6395629Z 2025-08-26T20:44:24.6395738Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6395963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6396033Z return mod(**inputs) 2025-08-26T20:44:24.6396476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6396572Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6396922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6397006Z outputs = layer_module( 2025-08-26T20:44:24.6397274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6397356Z outputs = self.rel_attn( 2025-08-26T20:44:24.6397626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6397745Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6398046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6398196Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6398200Z 2025-08-26T20:44:24.6398322Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6398536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6398618Z return mod(**inputs) 2025-08-26T20:44:24.6398898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6398987Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6399274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6399349Z outputs = layer_module( 2025-08-26T20:44:24.6399682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6399759Z outputs = self.rel_attn( 2025-08-26T20:44:24.6400030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6400182Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6400187Z 2025-08-26T20:44:24.6400295Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6400516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6400585Z return mod(**inputs) 2025-08-26T20:44:24.6400863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6400955Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6401228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6401316Z outputs = layer_module( 2025-08-26T20:44:24.6401646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6401726Z outputs = self.rel_attn( 2025-08-26T20:44:24.6401979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6402053Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6402335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6402471Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6402476Z 2025-08-26T20:44:24.6402589Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6402787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6402860Z return mod(**inputs) 2025-08-26T20:44:24.6403116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6403202Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6403461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6403548Z outputs = layer_module( 2025-08-26T20:44:24.6403814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6403884Z outputs = self.rel_attn( 2025-08-26T20:44:24.6404142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6404272Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6404276Z 2025-08-26T20:44:24.6404383Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6404590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6404659Z return mod(**inputs) 2025-08-26T20:44:24.6404923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6405011Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6405266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6405342Z outputs = layer_module( 2025-08-26T20:44:24.6405596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6405675Z outputs = self.rel_attn( 2025-08-26T20:44:24.6405930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6406004Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6406287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6406414Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6406418Z 2025-08-26T20:44:24.6406529Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6406732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6406806Z return mod(**inputs) 2025-08-26T20:44:24.6407062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6407148Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6407411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6407481Z outputs = layer_module( 2025-08-26T20:44:24.6407758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6407850Z outputs = self.rel_attn( 2025-08-26T20:44:24.6408109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6408211Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6408489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6408611Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6408615Z 2025-08-26T20:44:24.6408722Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6408921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6408996Z return mod(**inputs) 2025-08-26T20:44:24.6409254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6409349Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6409608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6409707Z outputs = layer_module( 2025-08-26T20:44:24.6409962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6410031Z outputs = self.rel_attn( 2025-08-26T20:44:24.6410295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6410405Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6410692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6410812Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6410816Z 2025-08-26T20:44:24.6410924Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6411137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6411209Z return mod(**inputs) 2025-08-26T20:44:24.6411476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6411565Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6411837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6411911Z outputs = layer_module( 2025-08-26T20:44:24.6412163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6412386Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6412654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6412746Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6413007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6413084Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6413352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6413431Z output = self.layer_1(output) 2025-08-26T20:44:24.6413437Z 2025-08-26T20:44:24.6413555Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6413779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6413861Z return mod(**inputs) 2025-08-26T20:44:24.6414175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6414278Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6414544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6414614Z outputs = layer_module( 2025-08-26T20:44:24.6414879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6415088Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6415352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6415437Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6415696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6415779Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6416036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6416150Z output = self.activation_function(output) 2025-08-26T20:44:24.6416365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6416436Z return self.act(input) 2025-08-26T20:44:24.6416440Z 2025-08-26T20:44:24.6416554Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6416777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6416850Z return mod(**inputs) 2025-08-26T20:44:24.6417106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6417190Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6417454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6417524Z outputs = layer_module( 2025-08-26T20:44:24.6417785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6417994Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6418261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6418341Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6418598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6418678Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6418936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6419018Z output = self.layer_2(output) 2025-08-26T20:44:24.6419023Z 2025-08-26T20:44:24.6419128Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6419327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6419400Z return mod(**inputs) 2025-08-26T20:44:24.6419657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6419747Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6420006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6420078Z outputs = layer_module( 2025-08-26T20:44:24.6420340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6420431Z outputs = self.rel_attn( 2025-08-26T20:44:24.6421073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6421181Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6421185Z 2025-08-26T20:44:24.6421297Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6421501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6421571Z return mod(**inputs) 2025-08-26T20:44:24.6421842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6421931Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6422196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6422267Z outputs = layer_module( 2025-08-26T20:44:24.6422525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6422630Z outputs = self.rel_attn( 2025-08-26T20:44:24.6422895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6423004Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6423007Z 2025-08-26T20:44:24.6423109Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6423307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6423390Z return mod(**inputs) 2025-08-26T20:44:24.6423640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6423731Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6423983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6424056Z outputs = layer_module( 2025-08-26T20:44:24.6424305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6424373Z outputs = self.rel_attn( 2025-08-26T20:44:24.6424630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6424701Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6424976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6425110Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6425113Z 2025-08-26T20:44:24.6425223Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6425420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6425488Z return mod(**inputs) 2025-08-26T20:44:24.6425750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6425832Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6426090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6426158Z outputs = layer_module( 2025-08-26T20:44:24.6426407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6426491Z outputs = self.rel_attn( 2025-08-26T20:44:24.6426743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6426905Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6426908Z 2025-08-26T20:44:24.6427029Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6427233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6427307Z return mod(**inputs) 2025-08-26T20:44:24.6427563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6427664Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6427914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6427990Z outputs = layer_module( 2025-08-26T20:44:24.6428238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6428308Z outputs = self.rel_attn( 2025-08-26T20:44:24.6428564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6428634Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6428931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6429062Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6429065Z 2025-08-26T20:44:24.6429167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6429372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6429458Z return mod(**inputs) 2025-08-26T20:44:24.6429721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6429804Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6430070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6430138Z outputs = layer_module( 2025-08-26T20:44:24.6430394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6430473Z outputs = self.rel_attn( 2025-08-26T20:44:24.6430728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6430839Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6430846Z 2025-08-26T20:44:24.6430948Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6431148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6431224Z return mod(**inputs) 2025-08-26T20:44:24.6431482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6431576Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6431830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6431900Z outputs = layer_module( 2025-08-26T20:44:24.6432162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6432232Z outputs = self.rel_attn( 2025-08-26T20:44:24.6432495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6432569Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6432851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6432977Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6432998Z 2025-08-26T20:44:24.6433119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6433328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6433395Z return mod(**inputs) 2025-08-26T20:44:24.6433663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6433748Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6434004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6434081Z outputs = layer_module( 2025-08-26T20:44:24.6434338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6434414Z outputs = self.rel_attn( 2025-08-26T20:44:24.6434669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6434765Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6435062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6435177Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6435180Z 2025-08-26T20:44:24.6435291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6435490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6435592Z return mod(**inputs) 2025-08-26T20:44:24.6435848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6435931Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6436210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6436283Z outputs = layer_module( 2025-08-26T20:44:24.6436558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6436633Z outputs = self.rel_attn( 2025-08-26T20:44:24.6436901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6437003Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6437295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6437424Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6437428Z 2025-08-26T20:44:24.6437538Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6437757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6437829Z return mod(**inputs) 2025-08-26T20:44:24.6438102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6438201Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6438473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6438553Z outputs = layer_module( 2025-08-26T20:44:24.6438822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6439049Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6439343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6439529Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6439849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6439935Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6440226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6440309Z output = self.layer_1(output) 2025-08-26T20:44:24.6440312Z 2025-08-26T20:44:24.6440427Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6440657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6440726Z return mod(**inputs) 2025-08-26T20:44:24.6440995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6441081Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6441341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6441422Z outputs = layer_module( 2025-08-26T20:44:24.6441698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6441918Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6442186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6442294Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6442555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6442631Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6442905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6442999Z output = self.activation_function(output) 2025-08-26T20:44:24.6443231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6443304Z return self.act(input) 2025-08-26T20:44:24.6443307Z 2025-08-26T20:44:24.6443412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6443624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6443692Z return mod(**inputs) 2025-08-26T20:44:24.6443962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6444047Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6444316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6444386Z outputs = layer_module( 2025-08-26T20:44:24.6444650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6444870Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6445140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6445224Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6445486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6445561Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6445825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6445900Z output = self.layer_2(output) 2025-08-26T20:44:24.6445903Z 2025-08-26T20:44:24.6446033Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6446254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6446331Z return mod(**inputs) 2025-08-26T20:44:24.6446588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6446673Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6446936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6447008Z outputs = layer_module( 2025-08-26T20:44:24.6447273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6447345Z outputs = self.rel_attn( 2025-08-26T20:44:24.6447604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6447714Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6447718Z 2025-08-26T20:44:24.6447841Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6448047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6448115Z return mod(**inputs) 2025-08-26T20:44:24.6448372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6448464Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6448744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6448818Z outputs = layer_module( 2025-08-26T20:44:24.6449065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6449143Z outputs = self.rel_attn( 2025-08-26T20:44:24.6449391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6449493Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6449496Z 2025-08-26T20:44:24.6449605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6449797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6449868Z return mod(**inputs) 2025-08-26T20:44:24.6450117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6450200Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6450464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6450533Z outputs = layer_module( 2025-08-26T20:44:24.6450795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6450867Z outputs = self.rel_attn( 2025-08-26T20:44:24.6451134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6451209Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6451484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6451627Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6451633Z 2025-08-26T20:44:24.6451738Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6451944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6452011Z return mod(**inputs) 2025-08-26T20:44:24.6452299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6452392Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6452651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6452726Z outputs = layer_module( 2025-08-26T20:44:24.6452985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6453055Z outputs = self.rel_attn( 2025-08-26T20:44:24.6453320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6453455Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6453459Z 2025-08-26T20:44:24.6453569Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6453773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6453846Z return mod(**inputs) 2025-08-26T20:44:24.6454133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6454215Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6454469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6454536Z outputs = layer_module( 2025-08-26T20:44:24.6454792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6454881Z outputs = self.rel_attn( 2025-08-26T20:44:24.6455129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6455210Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6455477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6455613Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6455617Z 2025-08-26T20:44:24.6455717Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6455917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6455983Z return mod(**inputs) 2025-08-26T20:44:24.6456231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6456323Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6456573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6456646Z outputs = layer_module( 2025-08-26T20:44:24.6456900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6456968Z outputs = self.rel_attn( 2025-08-26T20:44:24.6457230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6457331Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6457335Z 2025-08-26T20:44:24.6457446Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6457645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6457721Z return mod(**inputs) 2025-08-26T20:44:24.6457975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6458059Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6458341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6458431Z outputs = layer_module( 2025-08-26T20:44:24.6458695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6458765Z outputs = self.rel_attn( 2025-08-26T20:44:24.6459031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6459110Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6459376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6459506Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6459510Z 2025-08-26T20:44:24.6459610Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6459813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6459879Z return mod(**inputs) 2025-08-26T20:44:24.6460125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6460248Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6460497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6460571Z outputs = layer_module( 2025-08-26T20:44:24.6460828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6460920Z outputs = self.rel_attn( 2025-08-26T20:44:24.6461212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6461311Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6461618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6461738Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6461744Z 2025-08-26T20:44:24.6461862Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6462083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6462163Z return mod(**inputs) 2025-08-26T20:44:24.6462430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6462516Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6462782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6462851Z outputs = layer_module( 2025-08-26T20:44:24.6463111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6463191Z outputs = self.rel_attn( 2025-08-26T20:44:24.6463470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6463572Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6463877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6463997Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6464010Z 2025-08-26T20:44:24.6464120Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6464339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6464418Z return mod(**inputs) 2025-08-26T20:44:24.6464720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6464831Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6465113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6465186Z outputs = layer_module( 2025-08-26T20:44:24.6465462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6465684Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6465971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6466054Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6466338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6466425Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6466707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6466814Z output = self.layer_1(output) 2025-08-26T20:44:24.6466818Z 2025-08-26T20:44:24.6466928Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6467156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6467226Z return mod(**inputs) 2025-08-26T20:44:24.6467496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6467610Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6467899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6467978Z outputs = layer_module( 2025-08-26T20:44:24.6468271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6468511Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6468808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6468891Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6469173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6469254Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6469548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6469645Z output = self.activation_function(output) 2025-08-26T20:44:24.6469879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6469964Z return self.act(input) 2025-08-26T20:44:24.6469967Z 2025-08-26T20:44:24.6470079Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6470313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6470385Z return mod(**inputs) 2025-08-26T20:44:24.6470670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6470767Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6471053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6471133Z outputs = layer_module( 2025-08-26T20:44:24.6471404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6471665Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6471968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6472052Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6472336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6472414Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6472718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6472800Z output = self.layer_2(output) 2025-08-26T20:44:24.6472803Z 2025-08-26T20:44:24.6472914Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6473132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6473202Z return mod(**inputs) 2025-08-26T20:44:24.6473495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6473602Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6473886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6473969Z outputs = layer_module( 2025-08-26T20:44:24.6474235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6474338Z outputs = self.rel_attn( 2025-08-26T20:44:24.6474622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6474736Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6474739Z 2025-08-26T20:44:24.6474851Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6475063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6475142Z return mod(**inputs) 2025-08-26T20:44:24.6475424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6475520Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6475804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6475878Z outputs = layer_module( 2025-08-26T20:44:24.6476158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6476231Z outputs = self.rel_attn( 2025-08-26T20:44:24.6476515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6476625Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6476629Z 2025-08-26T20:44:24.6476745Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6476960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6477030Z return mod(**inputs) 2025-08-26T20:44:24.6477310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6477397Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6477692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6477764Z outputs = layer_module( 2025-08-26T20:44:24.6478040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6478147Z outputs = self.rel_attn( 2025-08-26T20:44:24.6478460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6478550Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6478839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6478984Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6478995Z 2025-08-26T20:44:24.6479108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6479330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6479499Z return mod(**inputs) 2025-08-26T20:44:24.6479792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6479890Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6480167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6480262Z outputs = layer_module( 2025-08-26T20:44:24.6480544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6480618Z outputs = self.rel_attn( 2025-08-26T20:44:24.6480898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6481045Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6481068Z 2025-08-26T20:44:24.6481181Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6481409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6481480Z return mod(**inputs) 2025-08-26T20:44:24.6481792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6481885Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6482186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6482258Z outputs = layer_module( 2025-08-26T20:44:24.6482529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6482613Z outputs = self.rel_attn( 2025-08-26T20:44:24.6482889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6482977Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6483274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6483421Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6483426Z 2025-08-26T20:44:24.6483553Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6483764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6483841Z return mod(**inputs) 2025-08-26T20:44:24.6484111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6484205Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6484475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6484549Z outputs = layer_module( 2025-08-26T20:44:24.6484826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6484906Z outputs = self.rel_attn( 2025-08-26T20:44:24.6485218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6485325Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6485328Z 2025-08-26T20:44:24.6485431Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6485640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6485706Z return mod(**inputs) 2025-08-26T20:44:24.6485969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6486053Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6486310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6486386Z outputs = layer_module( 2025-08-26T20:44:24.6486645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6486724Z outputs = self.rel_attn( 2025-08-26T20:44:24.6487016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6487100Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6487395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6487524Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6487543Z 2025-08-26T20:44:24.6487658Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6487860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6487935Z return mod(**inputs) 2025-08-26T20:44:24.6488196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6488283Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6488556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6488628Z outputs = layer_module( 2025-08-26T20:44:24.6488911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6488992Z outputs = self.rel_attn( 2025-08-26T20:44:24.6489285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6489379Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6489664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6489788Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6489792Z 2025-08-26T20:44:24.6489902Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6490126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6490197Z return mod(**inputs) 2025-08-26T20:44:24.6490477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6490582Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6490845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6490923Z outputs = layer_module( 2025-08-26T20:44:24.6491184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6491254Z outputs = self.rel_attn( 2025-08-26T20:44:24.6491559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6491655Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6491959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6492079Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6492083Z 2025-08-26T20:44:24.6492201Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6492418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6492491Z return mod(**inputs) 2025-08-26T20:44:24.6492773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6492862Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6493141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6493214Z outputs = layer_module( 2025-08-26T20:44:24.6493510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6493743Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6494029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6494139Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6494409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6494495Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6494766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6494846Z output = self.layer_1(output) 2025-08-26T20:44:24.6494850Z 2025-08-26T20:44:24.6494983Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6495185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6495260Z return mod(**inputs) 2025-08-26T20:44:24.6495514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6495599Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6495864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6495932Z outputs = layer_module( 2025-08-26T20:44:24.6496376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6496602Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6496878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6496958Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6497217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6497301Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6497559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6497661Z output = self.activation_function(output) 2025-08-26T20:44:24.6497876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6497948Z return self.act(input) 2025-08-26T20:44:24.6497961Z 2025-08-26T20:44:24.6498130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6498359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6498437Z return mod(**inputs) 2025-08-26T20:44:24.6498703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6498798Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6499068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6499144Z outputs = layer_module( 2025-08-26T20:44:24.6499421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6499637Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6499923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6500003Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6500307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6500392Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6500659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6500746Z output = self.layer_2(output) 2025-08-26T20:44:24.6500779Z 2025-08-26T20:44:24.6500893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6501116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6501186Z return mod(**inputs) 2025-08-26T20:44:24.6501460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6501559Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6501834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6501915Z outputs = layer_module( 2025-08-26T20:44:24.6502186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6502262Z outputs = self.rel_attn( 2025-08-26T20:44:24.6502542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6502653Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6502659Z 2025-08-26T20:44:24.6502774Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6502987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6503067Z return mod(**inputs) 2025-08-26T20:44:24.6503340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6503430Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6503712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6503785Z outputs = layer_module( 2025-08-26T20:44:24.6504064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6504140Z outputs = self.rel_attn( 2025-08-26T20:44:24.6504410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6504529Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6504533Z 2025-08-26T20:44:24.6504662Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6504904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6504977Z return mod(**inputs) 2025-08-26T20:44:24.6505252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6505349Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6505621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6505700Z outputs = layer_module( 2025-08-26T20:44:24.6505976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6506057Z outputs = self.rel_attn( 2025-08-26T20:44:24.6506329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6506410Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6506712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6506875Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6506879Z 2025-08-26T20:44:24.6506995Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6507203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6507276Z return mod(**inputs) 2025-08-26T20:44:24.6507577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6507664Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6507944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6508017Z outputs = layer_module( 2025-08-26T20:44:24.6508298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6508372Z outputs = self.rel_attn( 2025-08-26T20:44:24.6508646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6508798Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6508802Z 2025-08-26T20:44:24.6508909Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6509131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6509200Z return mod(**inputs) 2025-08-26T20:44:24.6509473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6509568Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6509843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6509925Z outputs = layer_module( 2025-08-26T20:44:24.6510196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6510268Z outputs = self.rel_attn( 2025-08-26T20:44:24.6510546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6510625Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6510926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6511067Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6511070Z 2025-08-26T20:44:24.6511184Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6511442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6511514Z return mod(**inputs) 2025-08-26T20:44:24.6511797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6511885Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6512161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6512234Z outputs = layer_module( 2025-08-26T20:44:24.6512508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6512587Z outputs = self.rel_attn( 2025-08-26T20:44:24.6512858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6512979Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6512993Z 2025-08-26T20:44:24.6513097Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6513320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6513387Z return mod(**inputs) 2025-08-26T20:44:24.6513640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6513730Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6513983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6514074Z outputs = layer_module( 2025-08-26T20:44:24.6514333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6514402Z outputs = self.rel_attn( 2025-08-26T20:44:24.6514671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6514744Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6515039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6515171Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6515175Z 2025-08-26T20:44:24.6515291Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6515506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6515579Z return mod(**inputs) 2025-08-26T20:44:24.6515864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6515952Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6516238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6516310Z outputs = layer_module( 2025-08-26T20:44:24.6516588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6516669Z outputs = self.rel_attn( 2025-08-26T20:44:24.6516940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6517043Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6517342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6517466Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6517477Z 2025-08-26T20:44:24.6517586Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6517831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6517928Z return mod(**inputs) 2025-08-26T20:44:24.6518205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6518302Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6518572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6518646Z outputs = layer_module( 2025-08-26T20:44:24.6518921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6518995Z outputs = self.rel_attn( 2025-08-26T20:44:24.6519271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6519431Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6519747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6519906Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6519911Z 2025-08-26T20:44:24.6520025Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6520254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6520327Z return mod(**inputs) 2025-08-26T20:44:24.6520617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6520736Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6521013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6521100Z outputs = layer_module( 2025-08-26T20:44:24.6521382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6521634Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6521901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6521981Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6522246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6522322Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6522584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6522659Z output = self.layer_1(output) 2025-08-26T20:44:24.6522663Z 2025-08-26T20:44:24.6522778Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6522980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6523049Z return mod(**inputs) 2025-08-26T20:44:24.6523313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6523398Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6523660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6523730Z outputs = layer_module( 2025-08-26T20:44:24.6523986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6524202Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6524519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6524620Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6524982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6525096Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6525497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6525620Z output = self.activation_function(output) 2025-08-26T20:44:24.6525968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6526064Z return self.act(input) 2025-08-26T20:44:24.6526069Z 2025-08-26T20:44:24.6526225Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6526556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6526661Z return mod(**inputs) 2025-08-26T20:44:24.6526950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6527067Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6527348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6527424Z outputs = layer_module( 2025-08-26T20:44:24.6527699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6527958Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6528219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6528305Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6528561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6528642Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6528895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6528968Z output = self.layer_2(output) 2025-08-26T20:44:24.6528972Z 2025-08-26T20:44:24.6529084Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6529289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6529367Z return mod(**inputs) 2025-08-26T20:44:24.6529640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6529726Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6530003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6530077Z outputs = layer_module( 2025-08-26T20:44:24.6530353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6530428Z outputs = self.rel_attn( 2025-08-26T20:44:24.6530706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6530814Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6530818Z 2025-08-26T20:44:24.6530927Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6531143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6531213Z return mod(**inputs) 2025-08-26T20:44:24.6531515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6531621Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6531892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6531974Z outputs = layer_module( 2025-08-26T20:44:24.6532241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6532321Z outputs = self.rel_attn( 2025-08-26T20:44:24.6532589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6532706Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6532710Z 2025-08-26T20:44:24.6532817Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6533026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6533105Z return mod(**inputs) 2025-08-26T20:44:24.6533382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6533495Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6533765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6533836Z outputs = layer_module( 2025-08-26T20:44:24.6534113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6534215Z outputs = self.rel_attn( 2025-08-26T20:44:24.6534493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6534572Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6534863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6535018Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6535023Z 2025-08-26T20:44:24.6535131Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6535347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6535417Z return mod(**inputs) 2025-08-26T20:44:24.6535698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6535785Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6536058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6536138Z outputs = layer_module( 2025-08-26T20:44:24.6536405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6536487Z outputs = self.rel_attn( 2025-08-26T20:44:24.6536761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6536905Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6536917Z 2025-08-26T20:44:24.6537026Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6537238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6537315Z return mod(**inputs) 2025-08-26T20:44:24.6537590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6537686Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6537954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6538046Z outputs = layer_module( 2025-08-26T20:44:24.6538338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6538414Z outputs = self.rel_attn( 2025-08-26T20:44:24.6538690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6538767Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6539055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6539204Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6539208Z 2025-08-26T20:44:24.6539324Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6539526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6539592Z return mod(**inputs) 2025-08-26T20:44:24.6539856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6539960Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6540212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6540286Z outputs = layer_module( 2025-08-26T20:44:24.6540542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6540617Z outputs = self.rel_attn( 2025-08-26T20:44:24.6540916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6541020Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6541024Z 2025-08-26T20:44:24.6541134Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6541361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6541457Z return mod(**inputs) 2025-08-26T20:44:24.6541891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6542031Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6542333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6542405Z outputs = layer_module( 2025-08-26T20:44:24.6542694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6542789Z outputs = self.rel_attn( 2025-08-26T20:44:24.6543058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6543133Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6543406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6543543Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6543549Z 2025-08-26T20:44:24.6543652Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6543861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6543934Z return mod(**inputs) 2025-08-26T20:44:24.6544210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6544310Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6544596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6544673Z outputs = layer_module( 2025-08-26T20:44:24.6544980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6545062Z outputs = self.rel_attn( 2025-08-26T20:44:24.6545333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6545431Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6545737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6545863Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6545869Z 2025-08-26T20:44:24.6545987Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6546201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6546274Z return mod(**inputs) 2025-08-26T20:44:24.6546563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6546652Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6546958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6547031Z outputs = layer_module( 2025-08-26T20:44:24.6547313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6547395Z outputs = self.rel_attn( 2025-08-26T20:44:24.6547685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6547788Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6548080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6548212Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6548218Z 2025-08-26T20:44:24.6548325Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6548537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6548613Z return mod(**inputs) 2025-08-26T20:44:24.6548884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6548978Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6549250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6549326Z outputs = layer_module( 2025-08-26T20:44:24.6549601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6549825Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6550115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6550199Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6550477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6550558Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6550835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6550926Z output = self.layer_1(output) 2025-08-26T20:44:24.6550930Z 2025-08-26T20:44:24.6551041Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6551274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6551343Z return mod(**inputs) 2025-08-26T20:44:24.6551646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6551747Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6552018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6552099Z outputs = layer_module( 2025-08-26T20:44:24.6552375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6552611Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6552894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6552979Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6553263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6553345Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6553638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6553734Z output = self.activation_function(output) 2025-08-26T20:44:24.6553960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6554057Z return self.act(input) 2025-08-26T20:44:24.6554060Z 2025-08-26T20:44:24.6554188Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6554410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6554480Z return mod(**inputs) 2025-08-26T20:44:24.6554757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6554847Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6555117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6555199Z outputs = layer_module( 2025-08-26T20:44:24.6555466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6555700Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6555989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6556072Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6556385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6556465Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6556749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6556831Z output = self.layer_2(output) 2025-08-26T20:44:24.6556835Z 2025-08-26T20:44:24.6556953Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6557172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6557243Z return mod(**inputs) 2025-08-26T20:44:24.6557531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6557624Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6557920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6557994Z outputs = layer_module( 2025-08-26T20:44:24.6558302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6558389Z outputs = self.rel_attn( 2025-08-26T20:44:24.6558665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-26T20:44:24.6558783Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-26T20:44:24.6558788Z 2025-08-26T20:44:24.6558901Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6559124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6559198Z return mod(**inputs) 2025-08-26T20:44:24.6559655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6559780Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6560066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6560151Z outputs = layer_module( 2025-08-26T20:44:24.6560430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6560541Z outputs = self.rel_attn( 2025-08-26T20:44:24.6560863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-26T20:44:24.6560972Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-26T20:44:24.6560976Z 2025-08-26T20:44:24.6561091Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6561320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6561391Z return mod(**inputs) 2025-08-26T20:44:24.6561683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6561773Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6562056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6562129Z outputs = layer_module( 2025-08-26T20:44:24.6562432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6562506Z outputs = self.rel_attn( 2025-08-26T20:44:24.6562795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6562883Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6563172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-26T20:44:24.6563322Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-26T20:44:24.6563326Z 2025-08-26T20:44:24.6563434Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6563649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6563727Z return mod(**inputs) 2025-08-26T20:44:24.6564010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6564104Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6564400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6564481Z outputs = layer_module( 2025-08-26T20:44:24.6564764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6564839Z outputs = self.rel_attn( 2025-08-26T20:44:24.6565141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-26T20:44:24.6565316Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-26T20:44:24.6565322Z 2025-08-26T20:44:24.6565439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6565650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6565721Z return mod(**inputs) 2025-08-26T20:44:24.6565999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6566088Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6566396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6566468Z outputs = layer_module( 2025-08-26T20:44:24.6566757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6566834Z outputs = self.rel_attn( 2025-08-26T20:44:24.6567167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6567289Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6567612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-26T20:44:24.6567761Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-26T20:44:24.6567765Z 2025-08-26T20:44:24.6567872Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6568103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6568182Z return mod(**inputs) 2025-08-26T20:44:24.6568459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6568559Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6568839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6568914Z outputs = layer_module( 2025-08-26T20:44:24.6569200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6569275Z outputs = self.rel_attn( 2025-08-26T20:44:24.6569561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-26T20:44:24.6569673Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-26T20:44:24.6569677Z 2025-08-26T20:44:24.6569804Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6570015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6570085Z return mod(**inputs) 2025-08-26T20:44:24.6570366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6570455Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6570734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6570805Z outputs = layer_module( 2025-08-26T20:44:24.6571073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6571152Z outputs = self.rel_attn( 2025-08-26T20:44:24.6571419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-26T20:44:24.6571502Z attn_vec = self.rel_attn_core( 2025-08-26T20:44:24.6571793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-26T20:44:24.6571964Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-26T20:44:24.6571968Z 2025-08-26T20:44:24.6572078Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6572289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6572377Z return mod(**inputs) 2025-08-26T20:44:24.6572629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6572720Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6572972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6573042Z outputs = layer_module( 2025-08-26T20:44:24.6573299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6573369Z outputs = self.rel_attn( 2025-08-26T20:44:24.6573628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6573735Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6574013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6574134Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6574138Z 2025-08-26T20:44:24.6574239Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6574464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6574531Z return mod(**inputs) 2025-08-26T20:44:24.6574791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6574875Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6575130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6575210Z outputs = layer_module( 2025-08-26T20:44:24.6575463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-26T20:44:24.6575539Z outputs = self.rel_attn( 2025-08-26T20:44:24.6575789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-26T20:44:24.6575880Z output_h = self.post_attention(h, attn_vec) 2025-08-26T20:44:24.6576163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-26T20:44:24.6576274Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-26T20:44:24.6576277Z 2025-08-26T20:44:24.6576385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6576589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6576662Z return mod(**inputs) 2025-08-26T20:44:24.6576919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6577003Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6577261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6577328Z outputs = layer_module( 2025-08-26T20:44:24.6577587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6577798Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6578080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6578184Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6578442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6578526Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6578797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-26T20:44:24.6578881Z output = self.layer_1(output) 2025-08-26T20:44:24.6578884Z 2025-08-26T20:44:24.6578992Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6579206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6579284Z return mod(**inputs) 2025-08-26T20:44:24.6579554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6579651Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6579923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6580014Z outputs = layer_module( 2025-08-26T20:44:24.6580292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6580509Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6580793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6580897Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6581175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6581248Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6581511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-26T20:44:24.6581610Z output = self.activation_function(output) 2025-08-26T20:44:24.6581830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:44:24.6581908Z return self.act(input) 2025-08-26T20:44:24.6581911Z 2025-08-26T20:44:24.6582013Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6582219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6582295Z return mod(**inputs) 2025-08-26T20:44:24.6582560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-26T20:44:24.6582653Z transformer_outputs = self.transformer( 2025-08-26T20:44:24.6582915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-26T20:44:24.6582993Z outputs = layer_module( 2025-08-26T20:44:24.6583254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-26T20:44:24.6583463Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-26T20:44:24.6583739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:44:24.6583818Z return forward_fn(*input_tensors) 2025-08-26T20:44:24.6584094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-26T20:44:24.6584171Z output_x = self.ff(output_x) 2025-08-26T20:44:24.6584448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-26T20:44:24.6584555Z output = self.layer_2(output) 2025-08-26T20:44:24.6584617Z 2025-08-26T20:44:24.6584741Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6584950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6585018Z return mod(**inputs) 2025-08-26T20:44:24.6585273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1624, in forward 2025-08-26T20:44:24.6585378Z logits = self.lm_loss(transformer_outputs[0]) 2025-08-26T20:44:24.6585382Z 2025-08-26T20:44:24.6585487Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:44:24.6585692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:44:24.6585759Z return mod(**inputs) 2025-08-26T20:44:24.6586021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1630, in forward 2025-08-26T20:44:24.6586159Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-08-26T20:44:24.6586164Z 2025-08-26T20:44:37.8313927Z Compilation time (from dynamo_timed): 33.750472456 2025-08-26T20:44:37.8347975Z pass 2025-08-26T20:44:37.8352182Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:44:37.8353626Z TIMING: _recursive_pre_grad_passes:0.01347 _recursive_joint_graph_passes:1.42704 _recursive_post_grad_passes:0.24485 async_compile.wait:0.75495 code_gen:11.80668 inductor_compile:17.08705 backend_compile:27.34103 gc:0.001 entire_frame_compile:33.75047 total_wall_time:33.75047 2025-08-26T20:44:37.8355019Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:56659 | FakeTensor.__torch_dispatch__:15989 | ProxyTorchDispatchMode.__torch_dispatch__:18623 2025-08-26T20:44:37.8355604Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-08-26T20:44:44.0404990Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-26T20:44:44.0406012Z from pkg_resources import resource_filename 2025-08-26T20:44:44.6478553Z 2025-08-26T20:44:46.1743908Z loading model: 0it [00:00, ?it/s] 2025-08-26T20:44:46.1746086Z loading model: 0it [00:01, ?it/s] 2025-08-26T20:44:46.1759013Z cpu eval YituTechConvBert 2025-08-26T20:44:47.1536197Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:44:47.4416241Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:44:47.7240838Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:45:00.7528175Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7528638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7533370Z return mod(**inputs) 2025-08-26T20:45:00.7533987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7534453Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7534927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7535427Z hidden_states = self.encoder( 2025-08-26T20:45:00.7535917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7536370Z layer_outputs = layer_module( 2025-08-26T20:45:00.7537098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7537568Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7538014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7538481Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7538922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7539353Z self_outputs = self.self( 2025-08-26T20:45:00.7539769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.7540236Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.7540401Z 2025-08-26T20:45:00.7540543Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7540927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7541259Z return mod(**inputs) 2025-08-26T20:45:00.7541672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7542187Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7542641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7543077Z hidden_states = self.encoder( 2025-08-26T20:45:00.7543494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7543993Z layer_outputs = layer_module( 2025-08-26T20:45:00.7544380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7544776Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7545206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7545656Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7546096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7546545Z self_outputs = self.self( 2025-08-26T20:45:00.7546981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.7547438Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.7547611Z 2025-08-26T20:45:00.7547728Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7548129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7548508Z return mod(**inputs) 2025-08-26T20:45:00.7548912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7549348Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7549788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7550218Z hidden_states = self.encoder( 2025-08-26T20:45:00.7550635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7551060Z layer_outputs = layer_module( 2025-08-26T20:45:00.7551427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7551815Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7552247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7552714Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7553175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7553632Z self_outputs = self.self( 2025-08-26T20:45:00.7554045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.7554511Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.7554678Z 2025-08-26T20:45:00.7554780Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7555016Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7555287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7555692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7556056Z return mod(**inputs) 2025-08-26T20:45:00.7556475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7556935Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7557380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7557847Z hidden_states = self.encoder( 2025-08-26T20:45:00.7558276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7558707Z layer_outputs = layer_module( 2025-08-26T20:45:00.7559096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7559964Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7560431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7560902Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7561343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7561775Z self_outputs = self.self( 2025-08-26T20:45:00.7562193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.7562663Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.7562835Z 2025-08-26T20:45:00.7562928Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7563182Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7563591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7563940Z return mod(**inputs) 2025-08-26T20:45:00.7564352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7564793Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7565240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7565681Z hidden_states = self.encoder( 2025-08-26T20:45:00.7566110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7566546Z layer_outputs = layer_module( 2025-08-26T20:45:00.7566915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7567312Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7567755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7568221Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7568700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7569148Z self_outputs = self.self( 2025-08-26T20:45:00.7569573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7570110Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7570634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.7571061Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.7571223Z 2025-08-26T20:45:00.7571336Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7571730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7572085Z return mod(**inputs) 2025-08-26T20:45:00.7572501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7572934Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7573399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7573827Z hidden_states = self.encoder( 2025-08-26T20:45:00.7574246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7574671Z layer_outputs = layer_module( 2025-08-26T20:45:00.7575041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7575488Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7575926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7576367Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7576795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7577502Z self_outputs = self.self( 2025-08-26T20:45:00.7577916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7578457Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7578981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.7579411Z x = self.pointwise(x) 2025-08-26T20:45:00.7579534Z 2025-08-26T20:45:00.7579651Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7580064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7580412Z return mod(**inputs) 2025-08-26T20:45:00.7580820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7581256Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7581695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7582120Z hidden_states = self.encoder( 2025-08-26T20:45:00.7582555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7582999Z layer_outputs = layer_module( 2025-08-26T20:45:00.7583379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7583778Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7584244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7584726Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7585181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7585637Z self_outputs = self.self( 2025-08-26T20:45:00.7586070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.7586626Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.7586866Z 2025-08-26T20:45:00.7586994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7587405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7587772Z return mod(**inputs) 2025-08-26T20:45:00.7588206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7588670Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7589129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7589602Z hidden_states = self.encoder( 2025-08-26T20:45:00.7590032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7590480Z layer_outputs = layer_module( 2025-08-26T20:45:00.7590860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7591288Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7591743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7592204Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7592662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7593132Z self_outputs = self.self( 2025-08-26T20:45:00.7593559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.7594062Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.7594266Z 2025-08-26T20:45:00.7594385Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7594792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7595180Z return mod(**inputs) 2025-08-26T20:45:00.7595599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7596065Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7596764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7597213Z hidden_states = self.encoder( 2025-08-26T20:45:00.7597652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7598083Z layer_outputs = layer_module( 2025-08-26T20:45:00.7598474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7598880Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7599351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7599910Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7600364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7600876Z self_outputs = self.self( 2025-08-26T20:45:00.7601330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.7601831Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.7602030Z 2025-08-26T20:45:00.7602129Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7602361Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7602624Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7603017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7603365Z return mod(**inputs) 2025-08-26T20:45:00.7603760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7604201Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7604641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7605069Z hidden_states = self.encoder( 2025-08-26T20:45:00.7605519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7605938Z layer_outputs = layer_module( 2025-08-26T20:45:00.7606313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7606707Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7607167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7607599Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7608047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7608483Z self_outputs = self.self( 2025-08-26T20:45:00.7608900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.7609371Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.7609551Z 2025-08-26T20:45:00.7609664Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7610055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7610414Z return mod(**inputs) 2025-08-26T20:45:00.7610825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7611274Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7611719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7612166Z hidden_states = self.encoder( 2025-08-26T20:45:00.7612587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7613023Z layer_outputs = layer_module( 2025-08-26T20:45:00.7613386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7613778Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7614228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7614684Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7615146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.7615628Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.7616170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.7616631Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7616789Z 2025-08-26T20:45:00.7616918Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7617325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7617677Z return mod(**inputs) 2025-08-26T20:45:00.7618098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7618559Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7619009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7619440Z hidden_states = self.encoder( 2025-08-26T20:45:00.7619881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7620319Z layer_outputs = layer_module( 2025-08-26T20:45:00.7620702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7621127Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7621568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7622031Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7622481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7622941Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7623419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7623947Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7624447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.7624902Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7625053Z 2025-08-26T20:45:00.7625176Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7625573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7625925Z return mod(**inputs) 2025-08-26T20:45:00.7626342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7626782Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7627215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7627633Z hidden_states = self.encoder( 2025-08-26T20:45:00.7628056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7628479Z layer_outputs = layer_module( 2025-08-26T20:45:00.7628831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7629194Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7629604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7630037Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7630451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7630864Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7631332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7631833Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7632292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.7632754Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.7633183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.7633566Z return self.act(input) 2025-08-26T20:45:00.7633695Z 2025-08-26T20:45:00.7633808Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7634222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7634582Z return mod(**inputs) 2025-08-26T20:45:00.7634991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7635429Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7635870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7636330Z hidden_states = self.encoder( 2025-08-26T20:45:00.7636750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7637180Z layer_outputs = layer_module( 2025-08-26T20:45:00.7637546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7637957Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7638391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7638856Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7639294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7639813Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7640299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.7640846Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.7641338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.7641776Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7641933Z 2025-08-26T20:45:00.7642045Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7642424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7642760Z return mod(**inputs) 2025-08-26T20:45:00.7643163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7643591Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7644027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7644452Z hidden_states = self.encoder( 2025-08-26T20:45:00.7644867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7645296Z layer_outputs = layer_module( 2025-08-26T20:45:00.7645658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7646047Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7646501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7646974Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7647404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7647835Z self_outputs = self.self( 2025-08-26T20:45:00.7648251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.7648704Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.7648863Z 2025-08-26T20:45:00.7648984Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7649373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7649750Z return mod(**inputs) 2025-08-26T20:45:00.7650158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7650605Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7651043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7651484Z hidden_states = self.encoder( 2025-08-26T20:45:00.7651905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7652329Z layer_outputs = layer_module( 2025-08-26T20:45:00.7652708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7653122Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7653537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7653959Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7654377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7654811Z self_outputs = self.self( 2025-08-26T20:45:00.7655224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.7655674Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.7655829Z 2025-08-26T20:45:00.7655942Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7656334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7656680Z return mod(**inputs) 2025-08-26T20:45:00.7657055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7657467Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7657889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7658312Z hidden_states = self.encoder( 2025-08-26T20:45:00.7658725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7659161Z layer_outputs = layer_module( 2025-08-26T20:45:00.7659534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7659926Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7660360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7660796Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7661241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7661679Z self_outputs = self.self( 2025-08-26T20:45:00.7662146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.7662598Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.7662755Z 2025-08-26T20:45:00.7662842Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7663076Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7663331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7663720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7664064Z return mod(**inputs) 2025-08-26T20:45:00.7664470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7664908Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7665345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7665771Z hidden_states = self.encoder( 2025-08-26T20:45:00.7666175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7666598Z layer_outputs = layer_module( 2025-08-26T20:45:00.7666952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7667323Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7667727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7668160Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7668598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7669053Z self_outputs = self.self( 2025-08-26T20:45:00.7669467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.7669918Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.7670095Z 2025-08-26T20:45:00.7670183Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7670442Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7670829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7671179Z return mod(**inputs) 2025-08-26T20:45:00.7671575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7672016Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7672452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7672875Z hidden_states = self.encoder( 2025-08-26T20:45:00.7673289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7673747Z layer_outputs = layer_module( 2025-08-26T20:45:00.7674121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7674525Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7674978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7675407Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7675864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7676303Z self_outputs = self.self( 2025-08-26T20:45:00.7676755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7677328Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7677875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.7678324Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.7678472Z 2025-08-26T20:45:00.7678584Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7679001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7679355Z return mod(**inputs) 2025-08-26T20:45:00.7679831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7680305Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7680780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7681230Z hidden_states = self.encoder( 2025-08-26T20:45:00.7681661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7682189Z layer_outputs = layer_module( 2025-08-26T20:45:00.7682563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7682948Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7683393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7683840Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7684293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7684710Z self_outputs = self.self( 2025-08-26T20:45:00.7685134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7685656Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7686187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.7686609Z x = self.pointwise(x) 2025-08-26T20:45:00.7686732Z 2025-08-26T20:45:00.7686844Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7687231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7687570Z return mod(**inputs) 2025-08-26T20:45:00.7687970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7688405Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7688844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7689273Z hidden_states = self.encoder( 2025-08-26T20:45:00.7689703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7690127Z layer_outputs = layer_module( 2025-08-26T20:45:00.7690499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7690896Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7691333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7691764Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7692219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7692644Z self_outputs = self.self( 2025-08-26T20:45:00.7693087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.7693602Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.7693900Z 2025-08-26T20:45:00.7694012Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7694401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7694747Z return mod(**inputs) 2025-08-26T20:45:00.7695150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7695589Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7696027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7696636Z hidden_states = self.encoder( 2025-08-26T20:45:00.7697077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7697567Z layer_outputs = layer_module( 2025-08-26T20:45:00.7697933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7698327Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7698748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7699192Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7699588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7699978Z self_outputs = self.self( 2025-08-26T20:45:00.7700362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.7700798Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.7700972Z 2025-08-26T20:45:00.7701082Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7701432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7701751Z return mod(**inputs) 2025-08-26T20:45:00.7702131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7702544Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7702955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7703347Z hidden_states = self.encoder( 2025-08-26T20:45:00.7703745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7704145Z layer_outputs = layer_module( 2025-08-26T20:45:00.7704499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7704855Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7705268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7705686Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7706103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7706506Z self_outputs = self.self( 2025-08-26T20:45:00.7706890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.7707399Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.7707616Z 2025-08-26T20:45:00.7707702Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7707926Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7708171Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7708541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7708875Z return mod(**inputs) 2025-08-26T20:45:00.7709264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7709686Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7710104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7710502Z hidden_states = self.encoder( 2025-08-26T20:45:00.7710895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7711295Z layer_outputs = layer_module( 2025-08-26T20:45:00.7711670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7712028Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7712443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7712865Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7713297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7713700Z self_outputs = self.self( 2025-08-26T20:45:00.7714088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.7714561Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.7714745Z 2025-08-26T20:45:00.7714859Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7715252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7715597Z return mod(**inputs) 2025-08-26T20:45:00.7716018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7716458Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7716896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7717337Z hidden_states = self.encoder( 2025-08-26T20:45:00.7717874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7718312Z layer_outputs = layer_module( 2025-08-26T20:45:00.7718691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7719094Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7719586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7720038Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7720475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.7720973Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.7721461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.7721885Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7722028Z 2025-08-26T20:45:00.7722158Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7722547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7722888Z return mod(**inputs) 2025-08-26T20:45:00.7723291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7723739Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7724190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7724636Z hidden_states = self.encoder( 2025-08-26T20:45:00.7725061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7725504Z layer_outputs = layer_module( 2025-08-26T20:45:00.7725871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7726245Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7726660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7727106Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7727508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7727913Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7728356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7728864Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7729321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.7729731Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7729877Z 2025-08-26T20:45:00.7729986Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7730357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7730691Z return mod(**inputs) 2025-08-26T20:45:00.7731070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7731479Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7731895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7732298Z hidden_states = self.encoder( 2025-08-26T20:45:00.7732697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7733097Z layer_outputs = layer_module( 2025-08-26T20:45:00.7733442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7733813Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7734214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7734632Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7735034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7735431Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7735869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7736351Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7736833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.7737293Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.7737673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.7738011Z return self.act(input) 2025-08-26T20:45:00.7738124Z 2025-08-26T20:45:00.7738236Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7738602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7738921Z return mod(**inputs) 2025-08-26T20:45:00.7739307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7739724Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7740137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7740532Z hidden_states = self.encoder( 2025-08-26T20:45:00.7740915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7741326Z layer_outputs = layer_module( 2025-08-26T20:45:00.7741669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7742024Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7742416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7742853Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7743263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7743670Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7744112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.7744606Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.7745107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.7745550Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7745700Z 2025-08-26T20:45:00.7745820Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7746213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7746540Z return mod(**inputs) 2025-08-26T20:45:00.7746925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7747340Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7747759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7748158Z hidden_states = self.encoder( 2025-08-26T20:45:00.7748562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7748970Z layer_outputs = layer_module( 2025-08-26T20:45:00.7749350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7749741Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7750160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7750589Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7751001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7751427Z self_outputs = self.self( 2025-08-26T20:45:00.7751836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.7752257Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.7752413Z 2025-08-26T20:45:00.7752518Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7752889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7753239Z return mod(**inputs) 2025-08-26T20:45:00.7753636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7754076Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7754514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7754941Z hidden_states = self.encoder( 2025-08-26T20:45:00.7755358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7755801Z layer_outputs = layer_module( 2025-08-26T20:45:00.7756173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7756566Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7757010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7757509Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7757953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7758411Z self_outputs = self.self( 2025-08-26T20:45:00.7758842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.7759305Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.7759530Z 2025-08-26T20:45:00.7759662Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7760064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7760425Z return mod(**inputs) 2025-08-26T20:45:00.7760846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7761290Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7761699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7762106Z hidden_states = self.encoder( 2025-08-26T20:45:00.7762509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7762923Z layer_outputs = layer_module( 2025-08-26T20:45:00.7763278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7763643Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7764061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7764484Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7764898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7765335Z self_outputs = self.self( 2025-08-26T20:45:00.7765750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.7766201Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.7766360Z 2025-08-26T20:45:00.7766489Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7766774Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7767027Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7767418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7767765Z return mod(**inputs) 2025-08-26T20:45:00.7768167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7768596Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7769037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7769468Z hidden_states = self.encoder( 2025-08-26T20:45:00.7769891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7770326Z layer_outputs = layer_module( 2025-08-26T20:45:00.7770694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7771117Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7771547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7771991Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7772429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7772864Z self_outputs = self.self( 2025-08-26T20:45:00.7773274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.7773730Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.7773896Z 2025-08-26T20:45:00.7773992Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7774243Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7774634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7774982Z return mod(**inputs) 2025-08-26T20:45:00.7775394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7775810Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7776212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7776613Z hidden_states = self.encoder( 2025-08-26T20:45:00.7777006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7777410Z layer_outputs = layer_module( 2025-08-26T20:45:00.7777766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7778126Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7778536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7778949Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7779360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7779756Z self_outputs = self.self( 2025-08-26T20:45:00.7780148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7780685Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7781210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.7781635Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.7781769Z 2025-08-26T20:45:00.7781878Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7782245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7782578Z return mod(**inputs) 2025-08-26T20:45:00.7782960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7783376Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7783784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7784191Z hidden_states = self.encoder( 2025-08-26T20:45:00.7784591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7784996Z layer_outputs = layer_module( 2025-08-26T20:45:00.7785348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7785734Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7786148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7786568Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7786984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7787396Z self_outputs = self.self( 2025-08-26T20:45:00.7787785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7788287Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7788808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.7789229Z x = self.pointwise(x) 2025-08-26T20:45:00.7789339Z 2025-08-26T20:45:00.7789445Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7789810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7790140Z return mod(**inputs) 2025-08-26T20:45:00.7790540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7790977Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7791383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7791784Z hidden_states = self.encoder( 2025-08-26T20:45:00.7792181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7792578Z layer_outputs = layer_module( 2025-08-26T20:45:00.7792920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7793286Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7793696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7794131Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7794561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7794972Z self_outputs = self.self( 2025-08-26T20:45:00.7795377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.7795923Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.7796150Z 2025-08-26T20:45:00.7796418Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7796814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7797162Z return mod(**inputs) 2025-08-26T20:45:00.7797568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7798012Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7798452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7798881Z hidden_states = self.encoder( 2025-08-26T20:45:00.7799295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7799791Z layer_outputs = layer_module( 2025-08-26T20:45:00.7800178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7800641Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7801053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7801474Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7801890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7802331Z self_outputs = self.self( 2025-08-26T20:45:00.7802715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.7803151Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.7803333Z 2025-08-26T20:45:00.7803441Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7803804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7804132Z return mod(**inputs) 2025-08-26T20:45:00.7804497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7804901Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7805309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7805712Z hidden_states = self.encoder( 2025-08-26T20:45:00.7806112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7806515Z layer_outputs = layer_module( 2025-08-26T20:45:00.7806866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7807233Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7807642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7808056Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7808465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7808866Z self_outputs = self.self( 2025-08-26T20:45:00.7809255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.7809720Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.7809902Z 2025-08-26T20:45:00.7809991Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7810204Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7810474Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7810869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7811211Z return mod(**inputs) 2025-08-26T20:45:00.7811607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7812066Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7812516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7812973Z hidden_states = self.encoder( 2025-08-26T20:45:00.7813370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7813768Z layer_outputs = layer_module( 2025-08-26T20:45:00.7814123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7814497Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7814907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7815384Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7815799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7816204Z self_outputs = self.self( 2025-08-26T20:45:00.7816594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.7817059Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.7817227Z 2025-08-26T20:45:00.7817334Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7817744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7818078Z return mod(**inputs) 2025-08-26T20:45:00.7818464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7818888Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7819297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7819709Z hidden_states = self.encoder( 2025-08-26T20:45:00.7820107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7820520Z layer_outputs = layer_module( 2025-08-26T20:45:00.7820865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7821238Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7821653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7822071Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7822492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.7822953Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.7823408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.7823831Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7823975Z 2025-08-26T20:45:00.7824089Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7824453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7824778Z return mod(**inputs) 2025-08-26T20:45:00.7825206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7825630Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7826047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7826451Z hidden_states = self.encoder( 2025-08-26T20:45:00.7826842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7827260Z layer_outputs = layer_module( 2025-08-26T20:45:00.7827604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7827961Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7828356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7828776Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7829187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7829610Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7830047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7830529Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7830970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.7831404Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7831541Z 2025-08-26T20:45:00.7831653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7832010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7832324Z return mod(**inputs) 2025-08-26T20:45:00.7832696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7833104Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7833504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7833889Z hidden_states = self.encoder( 2025-08-26T20:45:00.7834286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7834695Z layer_outputs = layer_module( 2025-08-26T20:45:00.7835047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7835421Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7835851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7836290Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7836724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7837148Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7837610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7838115Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7838596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.7839065Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.7839567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.7839968Z return self.act(input) 2025-08-26T20:45:00.7840097Z 2025-08-26T20:45:00.7840211Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7840608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7840969Z return mod(**inputs) 2025-08-26T20:45:00.7841379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7841787Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7842207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7842616Z hidden_states = self.encoder( 2025-08-26T20:45:00.7843037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7843467Z layer_outputs = layer_module( 2025-08-26T20:45:00.7843834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7844259Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7844691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7845149Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7845583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7846030Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7846495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.7847022Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.7847524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.7847959Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7848114Z 2025-08-26T20:45:00.7848228Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7848636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7848999Z return mod(**inputs) 2025-08-26T20:45:00.7849403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7849837Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7850276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7850705Z hidden_states = self.encoder( 2025-08-26T20:45:00.7851131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7851576Z layer_outputs = layer_module( 2025-08-26T20:45:00.7851947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7852337Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7852770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7853209Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7853648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7854057Z self_outputs = self.self( 2025-08-26T20:45:00.7854449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.7854914Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.7855062Z 2025-08-26T20:45:00.7855192Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7855546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7855865Z return mod(**inputs) 2025-08-26T20:45:00.7856249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7856686Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7857116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7857513Z hidden_states = self.encoder( 2025-08-26T20:45:00.7857910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7858328Z layer_outputs = layer_module( 2025-08-26T20:45:00.7858705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7859116Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7859549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7859989Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7860428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7860875Z self_outputs = self.self( 2025-08-26T20:45:00.7861289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.7861708Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.7861856Z 2025-08-26T20:45:00.7861962Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7862329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7862661Z return mod(**inputs) 2025-08-26T20:45:00.7863041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7863456Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7863867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7864271Z hidden_states = self.encoder( 2025-08-26T20:45:00.7864660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7865069Z layer_outputs = layer_module( 2025-08-26T20:45:00.7865419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7865805Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7866245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7866675Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7867105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7867543Z self_outputs = self.self( 2025-08-26T20:45:00.7867931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.7868392Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.7868552Z 2025-08-26T20:45:00.7868639Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7868872Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7869130Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7869574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7869930Z return mod(**inputs) 2025-08-26T20:45:00.7870339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7870789Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7871228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7871656Z hidden_states = self.encoder( 2025-08-26T20:45:00.7872072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7872498Z layer_outputs = layer_module( 2025-08-26T20:45:00.7872868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7873258Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7873687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7874152Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7874592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7875034Z self_outputs = self.self( 2025-08-26T20:45:00.7875480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.7875971Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.7876153Z 2025-08-26T20:45:00.7876241Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7876509Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7876911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7877293Z return mod(**inputs) 2025-08-26T20:45:00.7877708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7878163Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7878619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7879065Z hidden_states = self.encoder( 2025-08-26T20:45:00.7879576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7880029Z layer_outputs = layer_module( 2025-08-26T20:45:00.7880416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7880817Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7881256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7881690Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7882131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7882556Z self_outputs = self.self( 2025-08-26T20:45:00.7882970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7883496Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7884013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.7884452Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.7884600Z 2025-08-26T20:45:00.7884712Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7885149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7885502Z return mod(**inputs) 2025-08-26T20:45:00.7885899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7886334Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7886771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7887198Z hidden_states = self.encoder( 2025-08-26T20:45:00.7887608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7888027Z layer_outputs = layer_module( 2025-08-26T20:45:00.7888394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7888780Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7889210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7889662Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7890101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7890528Z self_outputs = self.self( 2025-08-26T20:45:00.7890941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7891483Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7892008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.7892409Z x = self.pointwise(x) 2025-08-26T20:45:00.7892528Z 2025-08-26T20:45:00.7892635Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7893004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7893328Z return mod(**inputs) 2025-08-26T20:45:00.7893710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7894142Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7894578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7895006Z hidden_states = self.encoder( 2025-08-26T20:45:00.7895417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7895846Z layer_outputs = layer_module( 2025-08-26T20:45:00.7896342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7896725Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7897141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7897552Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7897991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7898415Z self_outputs = self.self( 2025-08-26T20:45:00.7898827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.7899335Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.7899568Z 2025-08-26T20:45:00.7899681Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7900118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7900496Z return mod(**inputs) 2025-08-26T20:45:00.7900904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7901339Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7901771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7902171Z hidden_states = self.encoder( 2025-08-26T20:45:00.7902571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7902991Z layer_outputs = layer_module( 2025-08-26T20:45:00.7903357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7903743Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7904212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7904698Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7905203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7905664Z self_outputs = self.self( 2025-08-26T20:45:00.7906098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.7906680Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.7906868Z 2025-08-26T20:45:00.7906987Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7907402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7907773Z return mod(**inputs) 2025-08-26T20:45:00.7908256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7908734Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7909198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7909640Z hidden_states = self.encoder( 2025-08-26T20:45:00.7910087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7910515Z layer_outputs = layer_module( 2025-08-26T20:45:00.7910889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7911306Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7911762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7912203Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7912646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7913049Z self_outputs = self.self( 2025-08-26T20:45:00.7913461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.7913957Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.7914164Z 2025-08-26T20:45:00.7914254Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7914491Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7914758Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7915169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7915526Z return mod(**inputs) 2025-08-26T20:45:00.7915980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7916427Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7916873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7917320Z hidden_states = self.encoder( 2025-08-26T20:45:00.7917748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7918190Z layer_outputs = layer_module( 2025-08-26T20:45:00.7918567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7918956Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7919454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7919924Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7920377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7920835Z self_outputs = self.self( 2025-08-26T20:45:00.7921238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.7921677Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.7921850Z 2025-08-26T20:45:00.7921954Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7922332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7922651Z return mod(**inputs) 2025-08-26T20:45:00.7923027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7923444Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7923875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7924309Z hidden_states = self.encoder( 2025-08-26T20:45:00.7924725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7925160Z layer_outputs = layer_module( 2025-08-26T20:45:00.7925535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7925920Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7926326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7926734Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7927170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.7927669Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.7928125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.7928543Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7928685Z 2025-08-26T20:45:00.7928797Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7929186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7929539Z return mod(**inputs) 2025-08-26T20:45:00.7929943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7930373Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7930858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7931289Z hidden_states = self.encoder( 2025-08-26T20:45:00.7931712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7932138Z layer_outputs = layer_module( 2025-08-26T20:45:00.7932502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7932882Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7933290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7933707Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7934118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7934531Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7935003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7935510Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7935960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.7936368Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7936514Z 2025-08-26T20:45:00.7936620Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7937004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7937336Z return mod(**inputs) 2025-08-26T20:45:00.7937716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7938126Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7938540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7938943Z hidden_states = self.encoder( 2025-08-26T20:45:00.7939352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7939775Z layer_outputs = layer_module( 2025-08-26T20:45:00.7940138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7940531Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7940965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7941399Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7941800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7942201Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7942639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7943123Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7943574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.7944011Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.7944397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.7944741Z return self.act(input) 2025-08-26T20:45:00.7944855Z 2025-08-26T20:45:00.7944968Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7945356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7945699Z return mod(**inputs) 2025-08-26T20:45:00.7946081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7946494Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7946905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7947304Z hidden_states = self.encoder( 2025-08-26T20:45:00.7947693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7948094Z layer_outputs = layer_module( 2025-08-26T20:45:00.7948453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7948841Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7949264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7949706Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7950118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7950544Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7951012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.7951520Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.7951985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.7952424Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7952575Z 2025-08-26T20:45:00.7952695Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7953083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7953431Z return mod(**inputs) 2025-08-26T20:45:00.7953847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7954302Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7954589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7954680Z hidden_states = self.encoder( 2025-08-26T20:45:00.7954975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7955060Z layer_outputs = layer_module( 2025-08-26T20:45:00.7955307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7955395Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7955698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7955791Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7956090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7956169Z self_outputs = self.self( 2025-08-26T20:45:00.7956458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.7956569Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.7956573Z 2025-08-26T20:45:00.7956687Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7956936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7957012Z return mod(**inputs) 2025-08-26T20:45:00.7957335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7957428Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7957724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7957812Z hidden_states = self.encoder( 2025-08-26T20:45:00.7958106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7958195Z layer_outputs = layer_module( 2025-08-26T20:45:00.7958439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7958525Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7958841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7958930Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7959246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7959323Z self_outputs = self.self( 2025-08-26T20:45:00.7959707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.7959803Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.7959834Z 2025-08-26T20:45:00.7959951Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7960177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7960251Z return mod(**inputs) 2025-08-26T20:45:00.7960557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7960647Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7960943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7961029Z hidden_states = self.encoder( 2025-08-26T20:45:00.7961319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7961406Z layer_outputs = layer_module( 2025-08-26T20:45:00.7961651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7961740Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7962040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7962129Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7962430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7962512Z self_outputs = self.self( 2025-08-26T20:45:00.7962810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.7962911Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.7962915Z 2025-08-26T20:45:00.7963006Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7963099Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7963216Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7963445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7963516Z return mod(**inputs) 2025-08-26T20:45:00.7963825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7963944Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7964242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7964329Z hidden_states = self.encoder( 2025-08-26T20:45:00.7964625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7964711Z layer_outputs = layer_module( 2025-08-26T20:45:00.7964956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7965041Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7965345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7965434Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7965749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7965847Z self_outputs = self.self( 2025-08-26T20:45:00.7966141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.7966267Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.7966271Z 2025-08-26T20:45:00.7966359Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7966479Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7966726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7966799Z return mod(**inputs) 2025-08-26T20:45:00.7967112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7967202Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7967496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7967572Z hidden_states = self.encoder( 2025-08-26T20:45:00.7967847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7967919Z layer_outputs = layer_module( 2025-08-26T20:45:00.7968145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7968234Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7968503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7968592Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7968863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7968936Z self_outputs = self.self( 2025-08-26T20:45:00.7969216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7969378Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7969658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.7969736Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.7969741Z 2025-08-26T20:45:00.7969854Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7970057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7970124Z return mod(**inputs) 2025-08-26T20:45:00.7970419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7970519Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7970802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7970876Z hidden_states = self.encoder( 2025-08-26T20:45:00.7971148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7971226Z layer_outputs = layer_module( 2025-08-26T20:45:00.7971452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7971539Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7971827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7971914Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7972221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7972312Z self_outputs = self.self( 2025-08-26T20:45:00.7972608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.7972780Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.7973084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.7973178Z x = self.pointwise(x) 2025-08-26T20:45:00.7973182Z 2025-08-26T20:45:00.7973294Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7973525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7973591Z return mod(**inputs) 2025-08-26T20:45:00.7973871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7973953Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7974222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7974300Z hidden_states = self.encoder( 2025-08-26T20:45:00.7974569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7974652Z layer_outputs = layer_module( 2025-08-26T20:45:00.7974884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7974973Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7975273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7975360Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7975665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7975742Z self_outputs = self.self( 2025-08-26T20:45:00.7976046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.7976212Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.7976218Z 2025-08-26T20:45:00.7976331Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7976550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7976621Z return mod(**inputs) 2025-08-26T20:45:00.7976938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7977831Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7978120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7978194Z hidden_states = self.encoder( 2025-08-26T20:45:00.7978462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7978541Z layer_outputs = layer_module( 2025-08-26T20:45:00.7978778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7978871Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7979156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7979241Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7979556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7979652Z self_outputs = self.self( 2025-08-26T20:45:00.7979950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.7980075Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.7980080Z 2025-08-26T20:45:00.7980193Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7980396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7980485Z return mod(**inputs) 2025-08-26T20:45:00.7980766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7980850Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7981132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7981204Z hidden_states = self.encoder( 2025-08-26T20:45:00.7981474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7981554Z layer_outputs = layer_module( 2025-08-26T20:45:00.7981779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7981865Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7982134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7982215Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7982521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7982599Z self_outputs = self.self( 2025-08-26T20:45:00.7982895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.7983032Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.7983036Z 2025-08-26T20:45:00.7983128Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7983213Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.7983324Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7983556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7983625Z return mod(**inputs) 2025-08-26T20:45:00.7983903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7983986Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7984305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7984391Z hidden_states = self.encoder( 2025-08-26T20:45:00.7984679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7984762Z layer_outputs = layer_module( 2025-08-26T20:45:00.7985003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7985094Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7985380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7985466Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7985769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.7985844Z self_outputs = self.self( 2025-08-26T20:45:00.7986135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.7986278Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.7986282Z 2025-08-26T20:45:00.7986394Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7986614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7986685Z return mod(**inputs) 2025-08-26T20:45:00.7986992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7987078Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7987360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7987442Z hidden_states = self.encoder( 2025-08-26T20:45:00.7987725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7987810Z layer_outputs = layer_module( 2025-08-26T20:45:00.7988047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7988136Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7988419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.7988506Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.7988797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.7988934Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.7989227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.7989318Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7989324Z 2025-08-26T20:45:00.7989439Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7989649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7989720Z return mod(**inputs) 2025-08-26T20:45:00.7990009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7990096Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7990387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7990462Z hidden_states = self.encoder( 2025-08-26T20:45:00.7990774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7990879Z layer_outputs = layer_module( 2025-08-26T20:45:00.7991120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7991211Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7991494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7991585Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7991874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7991959Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7992288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7992422Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7992715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.7992836Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.7992839Z 2025-08-26T20:45:00.7992950Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7993167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7993237Z return mod(**inputs) 2025-08-26T20:45:00.7993526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7993629Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7993915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7994004Z hidden_states = self.encoder( 2025-08-26T20:45:00.7994291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7994375Z layer_outputs = layer_module( 2025-08-26T20:45:00.7994615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7994703Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7994994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7995088Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7995388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7995470Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.7995809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.7995946Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.7996378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.7996518Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.7996754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.7996852Z return self.act(input) 2025-08-26T20:45:00.7996860Z 2025-08-26T20:45:00.7996970Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.7997190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.7997262Z return mod(**inputs) 2025-08-26T20:45:00.7997595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.7997748Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.7998037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.7998125Z hidden_states = self.encoder( 2025-08-26T20:45:00.7998417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.7998496Z layer_outputs = layer_module( 2025-08-26T20:45:00.7998748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.7998835Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.7999137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.7999229Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.7999580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.7999732Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8000068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8000226Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8000521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8000654Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8000658Z 2025-08-26T20:45:00.8000788Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8001002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8001082Z return mod(**inputs) 2025-08-26T20:45:00.8001386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8001484Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8001769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8001846Z hidden_states = self.encoder( 2025-08-26T20:45:00.8002142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8002225Z layer_outputs = layer_module( 2025-08-26T20:45:00.8002473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8002561Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8002876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8002967Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8003258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8003346Z self_outputs = self.self( 2025-08-26T20:45:00.8003633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.8003742Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.8003746Z 2025-08-26T20:45:00.8003860Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8004083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8004163Z return mod(**inputs) 2025-08-26T20:45:00.8004469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8004589Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8004906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8004994Z hidden_states = self.encoder( 2025-08-26T20:45:00.8005292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8005371Z layer_outputs = layer_module( 2025-08-26T20:45:00.8005622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8005710Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8006019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8006107Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8006409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8006495Z self_outputs = self.self( 2025-08-26T20:45:00.8006807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.8006904Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.8006908Z 2025-08-26T20:45:00.8007022Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8007246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8007338Z return mod(**inputs) 2025-08-26T20:45:00.8007630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8007717Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8007995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8008075Z hidden_states = self.encoder( 2025-08-26T20:45:00.8008345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8008416Z layer_outputs = layer_module( 2025-08-26T20:45:00.8008647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8008725Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8009001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8009081Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8009351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8009426Z self_outputs = self.self( 2025-08-26T20:45:00.8009700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.8009803Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.8009806Z 2025-08-26T20:45:00.8009886Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8009971Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8010076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8010282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8010357Z return mod(**inputs) 2025-08-26T20:45:00.8010638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8010724Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8011021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8011110Z hidden_states = self.encoder( 2025-08-26T20:45:00.8011390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8011464Z layer_outputs = layer_module( 2025-08-26T20:45:00.8011695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8011774Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8012047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8012137Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8012406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8012486Z self_outputs = self.self( 2025-08-26T20:45:00.8012758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.8012872Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.8012893Z 2025-08-26T20:45:00.8012973Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8013078Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8013286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8013352Z return mod(**inputs) 2025-08-26T20:45:00.8013627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8013725Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8013991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8014069Z hidden_states = self.encoder( 2025-08-26T20:45:00.8014340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8014420Z layer_outputs = layer_module( 2025-08-26T20:45:00.8014643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8014721Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8015054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8015139Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8015432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8015506Z self_outputs = self.self( 2025-08-26T20:45:00.8015805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8015971Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8016244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.8016330Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.8016333Z 2025-08-26T20:45:00.8016437Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8016642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8016711Z return mod(**inputs) 2025-08-26T20:45:00.8016981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8017070Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8017354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8017464Z hidden_states = self.encoder( 2025-08-26T20:45:00.8017741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8017820Z layer_outputs = layer_module( 2025-08-26T20:45:00.8018035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8018111Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8018381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8018462Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8018774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8018844Z self_outputs = self.self( 2025-08-26T20:45:00.8019117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8019300Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8019566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.8019644Z x = self.pointwise(x) 2025-08-26T20:45:00.8019648Z 2025-08-26T20:45:00.8019752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8019957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8020041Z return mod(**inputs) 2025-08-26T20:45:00.8020308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8020397Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8020677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8020757Z hidden_states = self.encoder( 2025-08-26T20:45:00.8021017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8021088Z layer_outputs = layer_module( 2025-08-26T20:45:00.8021318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8021397Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8021674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8021755Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8022032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8022104Z self_outputs = self.self( 2025-08-26T20:45:00.8022376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.8022540Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.8022544Z 2025-08-26T20:45:00.8022647Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8022858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8022925Z return mod(**inputs) 2025-08-26T20:45:00.8023197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8023286Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8023569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8023661Z hidden_states = self.encoder( 2025-08-26T20:45:00.8023934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8024015Z layer_outputs = layer_module( 2025-08-26T20:45:00.8024233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8024309Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8024580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8024661Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8024931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8025002Z self_outputs = self.self( 2025-08-26T20:45:00.8025281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.8025412Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.8025431Z 2025-08-26T20:45:00.8025536Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8025743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8025809Z return mod(**inputs) 2025-08-26T20:45:00.8026077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8026182Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8026455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8026534Z hidden_states = self.encoder( 2025-08-26T20:45:00.8026809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8026887Z layer_outputs = layer_module( 2025-08-26T20:45:00.8027116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8027194Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8027471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8027553Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8027831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8027901Z self_outputs = self.self( 2025-08-26T20:45:00.8028170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.8028306Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.8028310Z 2025-08-26T20:45:00.8028391Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8028479Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8028582Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8028789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8028855Z return mod(**inputs) 2025-08-26T20:45:00.8029130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8029221Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8029494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8029572Z hidden_states = self.encoder( 2025-08-26T20:45:00.8029876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8029951Z layer_outputs = layer_module( 2025-08-26T20:45:00.8030184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8030262Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8030536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8030618Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8030887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8030967Z self_outputs = self.self( 2025-08-26T20:45:00.8031234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.8031360Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.8031364Z 2025-08-26T20:45:00.8031468Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8031695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8031762Z return mod(**inputs) 2025-08-26T20:45:00.8032032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8032122Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8032387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8032486Z hidden_states = self.encoder( 2025-08-26T20:45:00.8032754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8032826Z layer_outputs = layer_module( 2025-08-26T20:45:00.8033061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8033142Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8033413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8033494Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8033771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.8033903Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.8034172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.8034262Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8034266Z 2025-08-26T20:45:00.8034379Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8034606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8034678Z return mod(**inputs) 2025-08-26T20:45:00.8034963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8035057Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8035343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8035430Z hidden_states = self.encoder( 2025-08-26T20:45:00.8035711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8035793Z layer_outputs = layer_module( 2025-08-26T20:45:00.8036052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8036152Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8036444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8036538Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8036825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8036907Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8037228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8037370Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8037655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.8037754Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8037758Z 2025-08-26T20:45:00.8037870Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8038107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8038180Z return mod(**inputs) 2025-08-26T20:45:00.8038465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8038561Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8038842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8038946Z hidden_states = self.encoder( 2025-08-26T20:45:00.8039233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8039309Z layer_outputs = layer_module( 2025-08-26T20:45:00.8039637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8039727Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8040027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8040120Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8040411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8040505Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8040839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8040978Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8041269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.8041392Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.8041607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.8041679Z return self.act(input) 2025-08-26T20:45:00.8041683Z 2025-08-26T20:45:00.8041794Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8041994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8042068Z return mod(**inputs) 2025-08-26T20:45:00.8042340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8042421Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8042695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8042794Z hidden_states = self.encoder( 2025-08-26T20:45:00.8043087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8043162Z layer_outputs = layer_module( 2025-08-26T20:45:00.8043392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8043470Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8043741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8043836Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8044099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8044184Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8044488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8044624Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8044914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8044998Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8045001Z 2025-08-26T20:45:00.8045113Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8045313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8045404Z return mod(**inputs) 2025-08-26T20:45:00.8045674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8045755Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8046033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8046105Z hidden_states = self.encoder( 2025-08-26T20:45:00.8046381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8046452Z layer_outputs = layer_module( 2025-08-26T20:45:00.8046677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8046763Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8047030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8047118Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8047389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8047469Z self_outputs = self.self( 2025-08-26T20:45:00.8047742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.8047836Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.8047840Z 2025-08-26T20:45:00.8047954Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8048152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8048228Z return mod(**inputs) 2025-08-26T20:45:00.8048496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8048579Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8048854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8048942Z hidden_states = self.encoder( 2025-08-26T20:45:00.8049233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8049308Z layer_outputs = layer_module( 2025-08-26T20:45:00.8049537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8049623Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8049898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8049990Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8050260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8050339Z self_outputs = self.self( 2025-08-26T20:45:00.8050615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.8050699Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.8050720Z 2025-08-26T20:45:00.8050833Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8051034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8051110Z return mod(**inputs) 2025-08-26T20:45:00.8051407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8051491Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8051815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8051891Z hidden_states = self.encoder( 2025-08-26T20:45:00.8052192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8052270Z layer_outputs = layer_module( 2025-08-26T20:45:00.8052514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8052595Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8052901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8052991Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8053258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8053338Z self_outputs = self.self( 2025-08-26T20:45:00.8053606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.8053700Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.8053704Z 2025-08-26T20:45:00.8053794Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8070202Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8070522Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8070774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8070849Z return mod(**inputs) 2025-08-26T20:45:00.8071172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8071267Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8071559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8071640Z hidden_states = self.encoder( 2025-08-26T20:45:00.8071910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8072077Z layer_outputs = layer_module( 2025-08-26T20:45:00.8072343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8072440Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8072721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8072809Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8073094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8073174Z self_outputs = self.self( 2025-08-26T20:45:00.8073451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.8073558Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.8073564Z 2025-08-26T20:45:00.8073659Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8073773Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8074014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8074093Z return mod(**inputs) 2025-08-26T20:45:00.8074385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8074485Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8074777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8074887Z hidden_states = self.encoder( 2025-08-26T20:45:00.8075177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8075254Z layer_outputs = layer_module( 2025-08-26T20:45:00.8075511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8075598Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8075899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8075988Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8076283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8076369Z self_outputs = self.self( 2025-08-26T20:45:00.8076671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8076863Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8077168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.8077257Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.8077262Z 2025-08-26T20:45:00.8077391Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8077619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8077702Z return mod(**inputs) 2025-08-26T20:45:00.8078013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8078107Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8078401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8078479Z hidden_states = self.encoder( 2025-08-26T20:45:00.8078799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8078917Z layer_outputs = layer_module( 2025-08-26T20:45:00.8079166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8079251Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8079635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8079738Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8080029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8080117Z self_outputs = self.self( 2025-08-26T20:45:00.8080415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8080589Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8080886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.8080983Z x = self.pointwise(x) 2025-08-26T20:45:00.8080988Z 2025-08-26T20:45:00.8081119Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8081321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8081397Z return mod(**inputs) 2025-08-26T20:45:00.8081669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8081770Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8082047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8082119Z hidden_states = self.encoder( 2025-08-26T20:45:00.8082397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8082470Z layer_outputs = layer_module( 2025-08-26T20:45:00.8082697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8082786Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8083070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8083163Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8083458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8083536Z self_outputs = self.self( 2025-08-26T20:45:00.8083802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.8083960Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.8083964Z 2025-08-26T20:45:00.8084075Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8084279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8084352Z return mod(**inputs) 2025-08-26T20:45:00.8084617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8084696Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8084970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8085041Z hidden_states = self.encoder( 2025-08-26T20:45:00.8085322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8085406Z layer_outputs = layer_module( 2025-08-26T20:45:00.8085649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8085729Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8085990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8086076Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8086337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8086412Z self_outputs = self.self( 2025-08-26T20:45:00.8086675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.8086794Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.8086798Z 2025-08-26T20:45:00.8086909Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8087108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8087195Z return mod(**inputs) 2025-08-26T20:45:00.8087472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8087559Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8087845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8087932Z hidden_states = self.encoder( 2025-08-26T20:45:00.8088198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8088267Z layer_outputs = layer_module( 2025-08-26T20:45:00.8088493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8088571Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8088832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8088919Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8089179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8089254Z self_outputs = self.self( 2025-08-26T20:45:00.8089515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.8089643Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.8089654Z 2025-08-26T20:45:00.8089740Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8089820Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8089934Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8090130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8090207Z return mod(**inputs) 2025-08-26T20:45:00.8090476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8090557Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8090825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8090896Z hidden_states = self.encoder( 2025-08-26T20:45:00.8091163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8091231Z layer_outputs = layer_module( 2025-08-26T20:45:00.8091462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8091564Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8091826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8091915Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8092174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8092244Z self_outputs = self.self( 2025-08-26T20:45:00.8092511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.8092627Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.8092631Z 2025-08-26T20:45:00.8092746Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8092947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8093023Z return mod(**inputs) 2025-08-26T20:45:00.8093295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8093394Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8093664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8093733Z hidden_states = self.encoder( 2025-08-26T20:45:00.8094000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8094090Z layer_outputs = layer_module( 2025-08-26T20:45:00.8094310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8094393Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8094659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8094746Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8095009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.8095145Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.8095411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.8095496Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8095500Z 2025-08-26T20:45:00.8095875Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8096071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8096144Z return mod(**inputs) 2025-08-26T20:45:00.8096569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8096669Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8096966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8097037Z hidden_states = self.encoder( 2025-08-26T20:45:00.8097315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8097388Z layer_outputs = layer_module( 2025-08-26T20:45:00.8097623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8097702Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8097969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8098123Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8098438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8098528Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8098837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8098974Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8099246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.8099335Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8099339Z 2025-08-26T20:45:00.8099462Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8099662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8099736Z return mod(**inputs) 2025-08-26T20:45:00.8100002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8100108Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8100377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8100447Z hidden_states = self.encoder( 2025-08-26T20:45:00.8100714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8100811Z layer_outputs = layer_module( 2025-08-26T20:45:00.8101038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8101123Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8101393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8101487Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8101753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8101837Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8102141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8102264Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8102540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.8102654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.8102880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.8102953Z return self.act(input) 2025-08-26T20:45:00.8102957Z 2025-08-26T20:45:00.8103061Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8103269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8103334Z return mod(**inputs) 2025-08-26T20:45:00.8103607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8103687Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8103966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8104039Z hidden_states = self.encoder( 2025-08-26T20:45:00.8104307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8104414Z layer_outputs = layer_module( 2025-08-26T20:45:00.8104657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8104747Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8105014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8105099Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8105368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8105447Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8105757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8105895Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8106172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8106259Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8106285Z 2025-08-26T20:45:00.8106396Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8106618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8106688Z return mod(**inputs) 2025-08-26T20:45:00.8106980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8107085Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8107368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8107452Z hidden_states = self.encoder( 2025-08-26T20:45:00.8107739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8107822Z layer_outputs = layer_module( 2025-08-26T20:45:00.8108060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8108150Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8108435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8108521Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8108813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8108889Z self_outputs = self.self( 2025-08-26T20:45:00.8109177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.8109277Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.8109281Z 2025-08-26T20:45:00.8109393Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8109616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8109686Z return mod(**inputs) 2025-08-26T20:45:00.8109974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8110060Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8110341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8110426Z hidden_states = self.encoder( 2025-08-26T20:45:00.8110710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8110793Z layer_outputs = layer_module( 2025-08-26T20:45:00.8111060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8111154Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8111435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8111520Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8111808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8111892Z self_outputs = self.self( 2025-08-26T20:45:00.8112165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.8112248Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.8112251Z 2025-08-26T20:45:00.8112356Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8112566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8112633Z return mod(**inputs) 2025-08-26T20:45:00.8112965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8113046Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8113320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8113393Z hidden_states = self.encoder( 2025-08-26T20:45:00.8113677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8113756Z layer_outputs = layer_module( 2025-08-26T20:45:00.8113975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8114060Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8114331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8114416Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8114715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8114788Z self_outputs = self.self( 2025-08-26T20:45:00.8115083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.8115183Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.8115187Z 2025-08-26T20:45:00.8115279Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8115365Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8115476Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8115712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8115780Z return mod(**inputs) 2025-08-26T20:45:00.8116081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8116166Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8116460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8116541Z hidden_states = self.encoder( 2025-08-26T20:45:00.8116838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8116917Z layer_outputs = layer_module( 2025-08-26T20:45:00.8117150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8117250Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8117574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8117666Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8117964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8118041Z self_outputs = self.self( 2025-08-26T20:45:00.8118340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.8118454Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.8118458Z 2025-08-26T20:45:00.8118543Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8118670Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8118884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8118956Z return mod(**inputs) 2025-08-26T20:45:00.8119261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8119366Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8119931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8120014Z hidden_states = self.encoder( 2025-08-26T20:45:00.8120317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8120430Z layer_outputs = layer_module( 2025-08-26T20:45:00.8120673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8120769Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8121073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8121171Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8121479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8121554Z self_outputs = self.self( 2025-08-26T20:45:00.8121844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8122020Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8122326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.8122409Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.8122413Z 2025-08-26T20:45:00.8122527Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8122768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8122839Z return mod(**inputs) 2025-08-26T20:45:00.8123138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8123224Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8123526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8123602Z hidden_states = self.encoder( 2025-08-26T20:45:00.8123890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8123971Z layer_outputs = layer_module( 2025-08-26T20:45:00.8124209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8124315Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8124617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8124708Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8124999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8125072Z self_outputs = self.self( 2025-08-26T20:45:00.8125361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8125532Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8125824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.8125898Z x = self.pointwise(x) 2025-08-26T20:45:00.8125902Z 2025-08-26T20:45:00.8126015Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8126233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8126322Z return mod(**inputs) 2025-08-26T20:45:00.8126617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8126703Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8126987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8127096Z hidden_states = self.encoder( 2025-08-26T20:45:00.8127380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8127462Z layer_outputs = layer_module( 2025-08-26T20:45:00.8127698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8127789Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8128081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8128168Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8128463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8128536Z self_outputs = self.self( 2025-08-26T20:45:00.8128828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.8128996Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.8129000Z 2025-08-26T20:45:00.8129111Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8129334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8129405Z return mod(**inputs) 2025-08-26T20:45:00.8129704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8129789Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8130079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8130154Z hidden_states = self.encoder( 2025-08-26T20:45:00.8130439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8130525Z layer_outputs = layer_module( 2025-08-26T20:45:00.8130760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8130868Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8131172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8131263Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8131557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8131632Z self_outputs = self.self( 2025-08-26T20:45:00.8131935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.8132067Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.8132071Z 2025-08-26T20:45:00.8132183Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8132405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8132475Z return mod(**inputs) 2025-08-26T20:45:00.8132766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8132872Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8133164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8133240Z hidden_states = self.encoder( 2025-08-26T20:45:00.8133525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8133627Z layer_outputs = layer_module( 2025-08-26T20:45:00.8133861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8133952Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8134237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8134326Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8134638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8134717Z self_outputs = self.self( 2025-08-26T20:45:00.8135573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.8135893Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.8135900Z 2025-08-26T20:45:00.8136057Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8136180Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8136360Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8136985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8137095Z return mod(**inputs) 2025-08-26T20:45:00.8137479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8137587Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8137903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8137995Z hidden_states = self.encoder( 2025-08-26T20:45:00.8138291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8138379Z layer_outputs = layer_module( 2025-08-26T20:45:00.8138633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8138722Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8139221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8139345Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8139647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8139731Z self_outputs = self.self( 2025-08-26T20:45:00.8140033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.8140157Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.8140162Z 2025-08-26T20:45:00.8140281Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8140516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8140589Z return mod(**inputs) 2025-08-26T20:45:00.8140890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8140981Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8141270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8141384Z hidden_states = self.encoder( 2025-08-26T20:45:00.8141682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8141765Z layer_outputs = layer_module( 2025-08-26T20:45:00.8142012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8142134Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8142426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8142515Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8142815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.8142956Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.8143258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.8143350Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8143355Z 2025-08-26T20:45:00.8143471Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8143708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8143781Z return mod(**inputs) 2025-08-26T20:45:00.8144079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8144166Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8144472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8144548Z hidden_states = self.encoder( 2025-08-26T20:45:00.8145384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8145497Z layer_outputs = layer_module( 2025-08-26T20:45:00.8147097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8147251Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8147700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8147813Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8148140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8148269Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8148634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8148773Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8149065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.8149163Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8149169Z 2025-08-26T20:45:00.8149287Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8149519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8149591Z return mod(**inputs) 2025-08-26T20:45:00.8149885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8149975Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8150262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8150370Z hidden_states = self.encoder( 2025-08-26T20:45:00.8150652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8150734Z layer_outputs = layer_module( 2025-08-26T20:45:00.8150977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8151084Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8151377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8151469Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8151757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8151841Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8152182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8152426Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8152825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.8152959Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.8153196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.8153283Z return self.act(input) 2025-08-26T20:45:00.8153288Z 2025-08-26T20:45:00.8153408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8153641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8153724Z return mod(**inputs) 2025-08-26T20:45:00.8154017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8154118Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8154412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8154500Z hidden_states = self.encoder( 2025-08-26T20:45:00.8154797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8154876Z layer_outputs = layer_module( 2025-08-26T20:45:00.8155132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8155218Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8155555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8155651Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8155939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8156032Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8156374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8156550Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8156869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8156967Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8156971Z 2025-08-26T20:45:00.8157125Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8157355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8157475Z return mod(**inputs) 2025-08-26T20:45:00.8157774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8157871Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8158167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8158267Z hidden_states = self.encoder( 2025-08-26T20:45:00.8158568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8158645Z layer_outputs = layer_module( 2025-08-26T20:45:00.8158897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8158985Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8159280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8159380Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8159903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8159994Z self_outputs = self.self( 2025-08-26T20:45:00.8160289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.8160406Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.8160410Z 2025-08-26T20:45:00.8160528Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8160751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8160832Z return mod(**inputs) 2025-08-26T20:45:00.8161122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8161220Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8161512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8161591Z hidden_states = self.encoder( 2025-08-26T20:45:00.8161896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8161976Z layer_outputs = layer_module( 2025-08-26T20:45:00.8162230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8162315Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8162655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8162748Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8163045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8163133Z self_outputs = self.self( 2025-08-26T20:45:00.8163425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.8163523Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.8163528Z 2025-08-26T20:45:00.8163643Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8163862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8163941Z return mod(**inputs) 2025-08-26T20:45:00.8164234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8164330Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8164641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8164725Z hidden_states = self.encoder( 2025-08-26T20:45:00.8165021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8165098Z layer_outputs = layer_module( 2025-08-26T20:45:00.8165372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8165455Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8165751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8165839Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8166133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8166219Z self_outputs = self.self( 2025-08-26T20:45:00.8166518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.8166615Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.8166619Z 2025-08-26T20:45:00.8166704Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8166784Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8166899Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8167099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8167172Z return mod(**inputs) 2025-08-26T20:45:00.8167447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8167537Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8167803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8167877Z hidden_states = self.encoder( 2025-08-26T20:45:00.8168154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8168225Z layer_outputs = layer_module( 2025-08-26T20:45:00.8168458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8168539Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8168809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8168897Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8169214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8169297Z self_outputs = self.self( 2025-08-26T20:45:00.8169598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.8169703Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.8169715Z 2025-08-26T20:45:00.8169796Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8169903Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8170115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8170186Z return mod(**inputs) 2025-08-26T20:45:00.8170481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8170570Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8170864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8170959Z hidden_states = self.encoder( 2025-08-26T20:45:00.8171228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8171306Z layer_outputs = layer_module( 2025-08-26T20:45:00.8171529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8171625Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8171903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8171985Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8172264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8172337Z self_outputs = self.self( 2025-08-26T20:45:00.8172616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8172786Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8173060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.8173145Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.8173152Z 2025-08-26T20:45:00.8173262Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8173483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8173553Z return mod(**inputs) 2025-08-26T20:45:00.8173840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8173933Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8174218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8174301Z hidden_states = self.encoder( 2025-08-26T20:45:00.8174587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8174668Z layer_outputs = layer_module( 2025-08-26T20:45:00.8174905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8174992Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8175280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8175380Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8175689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8175767Z self_outputs = self.self( 2025-08-26T20:45:00.8176055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8176236Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8176525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.8176611Z x = self.pointwise(x) 2025-08-26T20:45:00.8176614Z 2025-08-26T20:45:00.8176726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8176945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8177017Z return mod(**inputs) 2025-08-26T20:45:00.8177314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8177429Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8177715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8177798Z hidden_states = self.encoder( 2025-08-26T20:45:00.8178082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8178177Z layer_outputs = layer_module( 2025-08-26T20:45:00.8178429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8178513Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8178810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8178898Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8179185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8179267Z self_outputs = self.self( 2025-08-26T20:45:00.8179555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.8179721Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.8179726Z 2025-08-26T20:45:00.8179832Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8180041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8180107Z return mod(**inputs) 2025-08-26T20:45:00.8180382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8180470Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8180744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8180822Z hidden_states = self.encoder( 2025-08-26T20:45:00.8181095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8181165Z layer_outputs = layer_module( 2025-08-26T20:45:00.8181398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8181477Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8181754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8181832Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8182149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8182224Z self_outputs = self.self( 2025-08-26T20:45:00.8182499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.8182636Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.8182641Z 2025-08-26T20:45:00.8182752Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8182973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8183046Z return mod(**inputs) 2025-08-26T20:45:00.8183333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8183425Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8183715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8183797Z hidden_states = self.encoder( 2025-08-26T20:45:00.8184097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8184178Z layer_outputs = layer_module( 2025-08-26T20:45:00.8184416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8184498Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8184809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8184894Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8185189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8185266Z self_outputs = self.self( 2025-08-26T20:45:00.8185550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.8185700Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.8185704Z 2025-08-26T20:45:00.8185790Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8185882Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8185994Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8186207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8186286Z return mod(**inputs) 2025-08-26T20:45:00.8186575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8186668Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8186957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8187042Z hidden_states = self.encoder( 2025-08-26T20:45:00.8187326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8187401Z layer_outputs = layer_module( 2025-08-26T20:45:00.8187648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8187730Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8188024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8188108Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8188394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8188494Z self_outputs = self.self( 2025-08-26T20:45:00.8188807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.8188944Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.8188948Z 2025-08-26T20:45:00.8189058Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8189277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8189349Z return mod(**inputs) 2025-08-26T20:45:00.8189635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8189728Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8190017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8190102Z hidden_states = self.encoder( 2025-08-26T20:45:00.8190395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8190491Z layer_outputs = layer_module( 2025-08-26T20:45:00.8190743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8190827Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8191124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8191230Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8191515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.8191661Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.8191951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.8192051Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8192057Z 2025-08-26T20:45:00.8192167Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8192387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8192458Z return mod(**inputs) 2025-08-26T20:45:00.8192744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8192843Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8193125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8193205Z hidden_states = self.encoder( 2025-08-26T20:45:00.8193488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8193564Z layer_outputs = layer_module( 2025-08-26T20:45:00.8193813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8193895Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8194185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8194280Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8194569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8194654Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8194981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8195138Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8195439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.8195539Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8195543Z 2025-08-26T20:45:00.8195653Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8195867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8195944Z return mod(**inputs) 2025-08-26T20:45:00.8196505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8196665Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8196972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8197064Z hidden_states = self.encoder( 2025-08-26T20:45:00.8197350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8197510Z layer_outputs = layer_module( 2025-08-26T20:45:00.8197757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8197838Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8198139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8198267Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8198556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8198650Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8198980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8199123Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8199498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.8199678Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.8199926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.8200007Z return self.act(input) 2025-08-26T20:45:00.8200015Z 2025-08-26T20:45:00.8200142Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8200361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8200440Z return mod(**inputs) 2025-08-26T20:45:00.8200742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8200831Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8201126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8201205Z hidden_states = self.encoder( 2025-08-26T20:45:00.8201499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8201575Z layer_outputs = layer_module( 2025-08-26T20:45:00.8201815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8201910Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8202208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8202300Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8202619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8202709Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8203017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8203154Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8203432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8203518Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8203522Z 2025-08-26T20:45:00.8203633Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8203835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8203910Z return mod(**inputs) 2025-08-26T20:45:00.8204183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8204283Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8204563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8204635Z hidden_states = self.encoder( 2025-08-26T20:45:00.8204914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8205012Z layer_outputs = layer_module( 2025-08-26T20:45:00.8205236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8205324Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8205597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8205689Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8205958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8206031Z self_outputs = self.self( 2025-08-26T20:45:00.8206305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.8206399Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.8206403Z 2025-08-26T20:45:00.8206515Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8206719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8206792Z return mod(**inputs) 2025-08-26T20:45:00.8207063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8207147Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8207424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8207499Z hidden_states = self.encoder( 2025-08-26T20:45:00.8207772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8207842Z layer_outputs = layer_module( 2025-08-26T20:45:00.8208064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8208153Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8208421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8208509Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8208861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8208941Z self_outputs = self.self( 2025-08-26T20:45:00.8209218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.8209303Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.8209306Z 2025-08-26T20:45:00.8209421Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8209623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8209700Z return mod(**inputs) 2025-08-26T20:45:00.8209974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8210054Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8210335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8210409Z hidden_states = self.encoder( 2025-08-26T20:45:00.8210688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8210779Z layer_outputs = layer_module( 2025-08-26T20:45:00.8211008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8211093Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8211364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8211474Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8211746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8211826Z self_outputs = self.self( 2025-08-26T20:45:00.8212097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.8212190Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.8212196Z 2025-08-26T20:45:00.8212286Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8212367Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8212480Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8212678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8212745Z return mod(**inputs) 2025-08-26T20:45:00.8213019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8213098Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8213376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8213449Z hidden_states = self.encoder( 2025-08-26T20:45:00.8213715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8213795Z layer_outputs = layer_module( 2025-08-26T20:45:00.8214018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8214104Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8214371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8214463Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8214731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8214802Z self_outputs = self.self( 2025-08-26T20:45:00.8215106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.8215212Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.8215218Z 2025-08-26T20:45:00.8215305Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8215412Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8215624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8215696Z return mod(**inputs) 2025-08-26T20:45:00.8215958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8216047Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8216307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8216387Z hidden_states = self.encoder( 2025-08-26T20:45:00.8216650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8216737Z layer_outputs = layer_module( 2025-08-26T20:45:00.8216963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8217042Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8217310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8217408Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8217666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8217744Z self_outputs = self.self( 2025-08-26T20:45:00.8218014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8218181Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8218455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.8218539Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.8218542Z 2025-08-26T20:45:00.8218645Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8218844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8218921Z return mod(**inputs) 2025-08-26T20:45:00.8219194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8219281Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8219560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8219632Z hidden_states = self.encoder( 2025-08-26T20:45:00.8219914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8219988Z layer_outputs = layer_module( 2025-08-26T20:45:00.8220231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8220308Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8220575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8220665Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8220932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8221008Z self_outputs = self.self( 2025-08-26T20:45:00.8221337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8221508Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8221800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.8221873Z x = self.pointwise(x) 2025-08-26T20:45:00.8221877Z 2025-08-26T20:45:00.8221989Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8222192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8222267Z return mod(**inputs) 2025-08-26T20:45:00.8222537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8222618Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8222897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8222989Z hidden_states = self.encoder( 2025-08-26T20:45:00.8223275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8223344Z layer_outputs = layer_module( 2025-08-26T20:45:00.8223570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8223648Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8223930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8224018Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8224280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8224360Z self_outputs = self.self( 2025-08-26T20:45:00.8224623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.8224779Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.8224791Z 2025-08-26T20:45:00.8224893Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8225092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8225165Z return mod(**inputs) 2025-08-26T20:45:00.8225437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8225523Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8225799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8225869Z hidden_states = self.encoder( 2025-08-26T20:45:00.8226134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8226208Z layer_outputs = layer_module( 2025-08-26T20:45:00.8226433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8226509Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8226770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8226857Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8227120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8227199Z self_outputs = self.self( 2025-08-26T20:45:00.8227490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.8227617Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.8227623Z 2025-08-26T20:45:00.8227726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8227921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8227993Z return mod(**inputs) 2025-08-26T20:45:00.8228262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8228350Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8228617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8228689Z hidden_states = self.encoder( 2025-08-26T20:45:00.8228971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8229041Z layer_outputs = layer_module( 2025-08-26T20:45:00.8229285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8229362Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8229623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8229709Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8230000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8230077Z self_outputs = self.self( 2025-08-26T20:45:00.8230343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.8230478Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.8230483Z 2025-08-26T20:45:00.8230565Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8230645Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8230756Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8230953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8231026Z return mod(**inputs) 2025-08-26T20:45:00.8231297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8231378Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8231650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8231721Z hidden_states = self.encoder( 2025-08-26T20:45:00.8231994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8232064Z layer_outputs = layer_module( 2025-08-26T20:45:00.8232284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8232366Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8232632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8232720Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8232991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8233069Z self_outputs = self.self( 2025-08-26T20:45:00.8233336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.8233467Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.8233486Z 2025-08-26T20:45:00.8233605Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8233809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8233884Z return mod(**inputs) 2025-08-26T20:45:00.8234152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8234235Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8234534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8234611Z hidden_states = self.encoder( 2025-08-26T20:45:00.8234910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8234985Z layer_outputs = layer_module( 2025-08-26T20:45:00.8235231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8235333Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8235627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8235720Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8236014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.8236180Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.8236475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.8236564Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8236576Z 2025-08-26T20:45:00.8236690Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8236903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8236982Z return mod(**inputs) 2025-08-26T20:45:00.8237263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8237354Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8237636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8237715Z hidden_states = self.encoder( 2025-08-26T20:45:00.8238013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8238087Z layer_outputs = layer_module( 2025-08-26T20:45:00.8238334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8238417Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8238705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8238809Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8239089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8239179Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8239613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8239769Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8240065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.8240178Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8240183Z 2025-08-26T20:45:00.8240327Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8240548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8240630Z return mod(**inputs) 2025-08-26T20:45:00.8240925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8241014Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8241323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8241401Z hidden_states = self.encoder( 2025-08-26T20:45:00.8241693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8241769Z layer_outputs = layer_module( 2025-08-26T20:45:00.8242021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8242105Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8242409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8242505Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8242784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8242895Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8243216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8243346Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8243645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.8243776Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.8244016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.8244091Z return self.act(input) 2025-08-26T20:45:00.8244095Z 2025-08-26T20:45:00.8244212Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8244435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8244505Z return mod(**inputs) 2025-08-26T20:45:00.8244798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8244887Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8245180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8245256Z hidden_states = self.encoder( 2025-08-26T20:45:00.8245543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8245629Z layer_outputs = layer_module( 2025-08-26T20:45:00.8245866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8245958Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8246246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8246331Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8246595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8246671Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8247008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8247148Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8247422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8247507Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8247510Z 2025-08-26T20:45:00.8247616Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8247828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8247895Z return mod(**inputs) 2025-08-26T20:45:00.8248171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8248253Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8248525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8248607Z hidden_states = self.encoder( 2025-08-26T20:45:00.8248894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8248974Z layer_outputs = layer_module( 2025-08-26T20:45:00.8249199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8249287Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8249571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8249653Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8249930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8250006Z self_outputs = self.self( 2025-08-26T20:45:00.8250284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.8250380Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.8250383Z 2025-08-26T20:45:00.8250490Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8250707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8250771Z return mod(**inputs) 2025-08-26T20:45:00.8251041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8251122Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8251392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8251464Z hidden_states = self.encoder( 2025-08-26T20:45:00.8251728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8251807Z layer_outputs = layer_module( 2025-08-26T20:45:00.8252029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8252123Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8252382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8252464Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8252731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8252801Z self_outputs = self.self( 2025-08-26T20:45:00.8253828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.8253936Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.8253941Z 2025-08-26T20:45:00.8254053Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8254252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8254318Z return mod(**inputs) 2025-08-26T20:45:00.8254603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8254685Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8254965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8255040Z hidden_states = self.encoder( 2025-08-26T20:45:00.8255316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8255401Z layer_outputs = layer_module( 2025-08-26T20:45:00.8255638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8255758Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8256041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8256131Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8256424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8256522Z self_outputs = self.self( 2025-08-26T20:45:00.8256817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.8256916Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.8256920Z 2025-08-26T20:45:00.8257018Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8257107Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8257223Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8257442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8257511Z return mod(**inputs) 2025-08-26T20:45:00.8257805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8257890Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8258177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8258260Z hidden_states = self.encoder( 2025-08-26T20:45:00.8258547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8258630Z layer_outputs = layer_module( 2025-08-26T20:45:00.8258875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8258961Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8259263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8259350Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8259649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8259728Z self_outputs = self.self( 2025-08-26T20:45:00.8260025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.8260140Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.8260144Z 2025-08-26T20:45:00.8260253Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8260398Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8260616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8260696Z return mod(**inputs) 2025-08-26T20:45:00.8260990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8261086Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8261376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8261454Z hidden_states = self.encoder( 2025-08-26T20:45:00.8261748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8261820Z layer_outputs = layer_module( 2025-08-26T20:45:00.8262049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8262135Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8262422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8262509Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8262775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8262852Z self_outputs = self.self( 2025-08-26T20:45:00.8263137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8263302Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8263591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.8263677Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.8263682Z 2025-08-26T20:45:00.8263803Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8264015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8264086Z return mod(**inputs) 2025-08-26T20:45:00.8264378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8264465Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8264759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8264835Z hidden_states = self.encoder( 2025-08-26T20:45:00.8265124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8265203Z layer_outputs = layer_module( 2025-08-26T20:45:00.8265441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8265532Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8265818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8265914Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8266194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8266271Z self_outputs = self.self( 2025-08-26T20:45:00.8266561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8266749Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8267061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.8267145Z x = self.pointwise(x) 2025-08-26T20:45:00.8267149Z 2025-08-26T20:45:00.8267268Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8267478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8267548Z return mod(**inputs) 2025-08-26T20:45:00.8267842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8267929Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8268222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8268298Z hidden_states = self.encoder( 2025-08-26T20:45:00.8268586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8268670Z layer_outputs = layer_module( 2025-08-26T20:45:00.8268928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8269018Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8269308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8269400Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8269703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8269778Z self_outputs = self.self( 2025-08-26T20:45:00.8270071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.8270239Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.8270243Z 2025-08-26T20:45:00.8270362Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8270577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8270646Z return mod(**inputs) 2025-08-26T20:45:00.8270942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8271028Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8271320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8271396Z hidden_states = self.encoder( 2025-08-26T20:45:00.8271688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8271766Z layer_outputs = layer_module( 2025-08-26T20:45:00.8272009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8272103Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8272391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8272485Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8272769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8272845Z self_outputs = self.self( 2025-08-26T20:45:00.8273136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.8273267Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.8273271Z 2025-08-26T20:45:00.8273407Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8273641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8273712Z return mod(**inputs) 2025-08-26T20:45:00.8273985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8274067Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8274351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8274430Z hidden_states = self.encoder( 2025-08-26T20:45:00.8274725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8274800Z layer_outputs = layer_module( 2025-08-26T20:45:00.8275041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8275133Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8275413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8275526Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8275816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8275894Z self_outputs = self.self( 2025-08-26T20:45:00.8276191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.8276350Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.8276355Z 2025-08-26T20:45:00.8276451Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8276543Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8276677Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8276896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8276972Z return mod(**inputs) 2025-08-26T20:45:00.8277266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8277358Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8277658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8277741Z hidden_states = self.encoder( 2025-08-26T20:45:00.8278034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8278121Z layer_outputs = layer_module( 2025-08-26T20:45:00.8278371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8278470Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8278765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8278859Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8279159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8279240Z self_outputs = self.self( 2025-08-26T20:45:00.8279718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.8279863Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.8279868Z 2025-08-26T20:45:00.8279990Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8280253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8280360Z return mod(**inputs) 2025-08-26T20:45:00.8280664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8280756Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8281057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8281145Z hidden_states = self.encoder( 2025-08-26T20:45:00.8281424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8281505Z layer_outputs = layer_module( 2025-08-26T20:45:00.8281732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8281819Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8282098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8282187Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8282476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.8282607Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.8282885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.8282993Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8282996Z 2025-08-26T20:45:00.8283108Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8283309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8283375Z return mod(**inputs) 2025-08-26T20:45:00.8283653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8283734Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8284015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8284091Z hidden_states = self.encoder( 2025-08-26T20:45:00.8284385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8284460Z layer_outputs = layer_module( 2025-08-26T20:45:00.8284697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8284787Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8285071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8285169Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8285449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8285533Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8285864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8285996Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8286289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.8286378Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8286382Z 2025-08-26T20:45:00.8286497Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8286736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8286808Z return mod(**inputs) 2025-08-26T20:45:00.8287115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8287203Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8287493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8287565Z hidden_states = self.encoder( 2025-08-26T20:45:00.8287833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8287915Z layer_outputs = layer_module( 2025-08-26T20:45:00.8288136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8288221Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8288489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8288574Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8288865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8288944Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8289265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8289395Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8289711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.8289832Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.8290066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.8290151Z return self.act(input) 2025-08-26T20:45:00.8290154Z 2025-08-26T20:45:00.8290266Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8290488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8290558Z return mod(**inputs) 2025-08-26T20:45:00.8290842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8290936Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8291221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8291305Z hidden_states = self.encoder( 2025-08-26T20:45:00.8291583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8291666Z layer_outputs = layer_module( 2025-08-26T20:45:00.8291902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8291986Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8292279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8292367Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8292648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8292731Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8293049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8293198Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8293515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8293610Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8293615Z 2025-08-26T20:45:00.8293726Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8293942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8294013Z return mod(**inputs) 2025-08-26T20:45:00.8294296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8294391Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8294671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8294755Z hidden_states = self.encoder( 2025-08-26T20:45:00.8295040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8295114Z layer_outputs = layer_module( 2025-08-26T20:45:00.8295381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8295465Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8295754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8295840Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8296303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8296430Z self_outputs = self.self( 2025-08-26T20:45:00.8296843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-26T20:45:00.8296961Z mixed_query_layer = self.query(hidden_states) 2025-08-26T20:45:00.8296966Z 2025-08-26T20:45:00.8297081Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8297304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8297375Z return mod(**inputs) 2025-08-26T20:45:00.8297661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8297757Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8298042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8298129Z hidden_states = self.encoder( 2025-08-26T20:45:00.8298419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8298495Z layer_outputs = layer_module( 2025-08-26T20:45:00.8298743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8298829Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8299120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8299206Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8299502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8299580Z self_outputs = self.self( 2025-08-26T20:45:00.8299865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-26T20:45:00.8299961Z mixed_key_layer = self.key(hidden_states) 2025-08-26T20:45:00.8299965Z 2025-08-26T20:45:00.8300076Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8300381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8300455Z return mod(**inputs) 2025-08-26T20:45:00.8300744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8300838Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8301129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8301211Z hidden_states = self.encoder( 2025-08-26T20:45:00.8301500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8301589Z layer_outputs = layer_module( 2025-08-26T20:45:00.8301832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8301927Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8302214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8302325Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8302604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8302678Z self_outputs = self.self( 2025-08-26T20:45:00.8302950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-26T20:45:00.8303076Z mixed_value_layer = self.value(hidden_states) 2025-08-26T20:45:00.8303080Z 2025-08-26T20:45:00.8303165Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8303252Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8303358Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8303559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8303634Z return mod(**inputs) 2025-08-26T20:45:00.8303904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8303991Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8304259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8304338Z hidden_states = self.encoder( 2025-08-26T20:45:00.8304606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8304677Z layer_outputs = layer_module( 2025-08-26T20:45:00.8304907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8304986Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8305260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8305344Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8305610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8305688Z self_outputs = self.self( 2025-08-26T20:45:00.8305951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-26T20:45:00.8306065Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-26T20:45:00.8306068Z 2025-08-26T20:45:00.8306148Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8306258Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8306481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8306551Z return mod(**inputs) 2025-08-26T20:45:00.8306841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8306926Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8307204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8307275Z hidden_states = self.encoder( 2025-08-26T20:45:00.8307555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8307639Z layer_outputs = layer_module( 2025-08-26T20:45:00.8307878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8307968Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8308256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8308340Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8308650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8308733Z self_outputs = self.self( 2025-08-26T20:45:00.8309005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8309168Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8309463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-26T20:45:00.8309542Z x = self.depthwise(hidden_states) 2025-08-26T20:45:00.8309545Z 2025-08-26T20:45:00.8309650Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8309860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8309926Z return mod(**inputs) 2025-08-26T20:45:00.8310203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8310282Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8310548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8310632Z hidden_states = self.encoder( 2025-08-26T20:45:00.8310916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8310997Z layer_outputs = layer_module( 2025-08-26T20:45:00.8311234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8311326Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8311616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8311703Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8311997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8312071Z self_outputs = self.self( 2025-08-26T20:45:00.8312363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-26T20:45:00.8312533Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-26T20:45:00.8312820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-26T20:45:00.8312903Z x = self.pointwise(x) 2025-08-26T20:45:00.8312907Z 2025-08-26T20:45:00.8313059Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8313285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8313356Z return mod(**inputs) 2025-08-26T20:45:00.8313632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8313714Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8313983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8314066Z hidden_states = self.encoder( 2025-08-26T20:45:00.8314332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8314411Z layer_outputs = layer_module( 2025-08-26T20:45:00.8314636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8314715Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8315012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8315093Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8315370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8315440Z self_outputs = self.self( 2025-08-26T20:45:00.8315709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-26T20:45:00.8315890Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-26T20:45:00.8315894Z 2025-08-26T20:45:00.8315999Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8316209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8316276Z return mod(**inputs) 2025-08-26T20:45:00.8316555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8316634Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8316899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8316978Z hidden_states = self.encoder( 2025-08-26T20:45:00.8317256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8317333Z layer_outputs = layer_module( 2025-08-26T20:45:00.8317564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8317645Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8317938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8318026Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8318316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8318390Z self_outputs = self.self( 2025-08-26T20:45:00.8318678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-26T20:45:00.8318808Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-26T20:45:00.8318812Z 2025-08-26T20:45:00.8318921Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8319139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8319208Z return mod(**inputs) 2025-08-26T20:45:00.8319604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8319698Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8319984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8320069Z hidden_states = self.encoder( 2025-08-26T20:45:00.8320354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8320437Z layer_outputs = layer_module( 2025-08-26T20:45:00.8320674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8320766Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8321053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8321141Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8321437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8321535Z self_outputs = self.self( 2025-08-26T20:45:00.8321830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-26T20:45:00.8321970Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-26T20:45:00.8321974Z 2025-08-26T20:45:00.8322083Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8322177Z cudagraph partition due to non gpu ops 2025-08-26T20:45:00.8322290Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8322524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8322595Z return mod(**inputs) 2025-08-26T20:45:00.8322884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8322981Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8323268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8323351Z hidden_states = self.encoder( 2025-08-26T20:45:00.8323634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8323719Z layer_outputs = layer_module( 2025-08-26T20:45:00.8323956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8324038Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8324330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8324418Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8324708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-26T20:45:00.8324783Z self_outputs = self.self( 2025-08-26T20:45:00.8325066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-26T20:45:00.8325193Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-26T20:45:00.8325197Z 2025-08-26T20:45:00.8325311Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8325534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8325604Z return mod(**inputs) 2025-08-26T20:45:00.8325896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8326029Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8326331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8326418Z hidden_states = self.encoder( 2025-08-26T20:45:00.8326703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8326786Z layer_outputs = layer_module( 2025-08-26T20:45:00.8327027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8327114Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8327409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-26T20:45:00.8327495Z self_attention_outputs = self.attention( 2025-08-26T20:45:00.8327786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-26T20:45:00.8327924Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-26T20:45:00.8328220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-26T20:45:00.8328303Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8328307Z 2025-08-26T20:45:00.8328408Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8328610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8328702Z return mod(**inputs) 2025-08-26T20:45:00.8328984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8329065Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8329342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8329423Z hidden_states = self.encoder( 2025-08-26T20:45:00.8329695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8329774Z layer_outputs = layer_module( 2025-08-26T20:45:00.8330000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8330078Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8330363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8330450Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8330731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8330811Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8331130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8331258Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8331533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-26T20:45:00.8331625Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8331628Z 2025-08-26T20:45:00.8331733Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8331956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8332026Z return mod(**inputs) 2025-08-26T20:45:00.8332317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8332440Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8332724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8332810Z hidden_states = self.encoder( 2025-08-26T20:45:00.8333078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8333158Z layer_outputs = layer_module( 2025-08-26T20:45:00.8333383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8333464Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8333741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8333826Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8334100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8334179Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8334503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-26T20:45:00.8334631Z intermediate_output = self.intermediate(attention_output) 2025-08-26T20:45:00.8334890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-26T20:45:00.8335009Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-26T20:45:00.8335248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-26T20:45:00.8335326Z return self.act(input) 2025-08-26T20:45:00.8335330Z 2025-08-26T20:45:00.8335434Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8335638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8335713Z return mod(**inputs) 2025-08-26T20:45:00.8335988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-26T20:45:00.8336080Z generator_hidden_states = self.convbert( 2025-08-26T20:45:00.8336357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-26T20:45:00.8336430Z hidden_states = self.encoder( 2025-08-26T20:45:00.8336708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-26T20:45:00.8336781Z layer_outputs = layer_module( 2025-08-26T20:45:00.8337024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-26T20:45:00.8337100Z return super().__call__(*args, **kwargs) 2025-08-26T20:45:00.8337377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-26T20:45:00.8337461Z layer_output = apply_chunking_to_forward( 2025-08-26T20:45:00.8337722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-26T20:45:00.8337805Z return forward_fn(*input_tensors) 2025-08-26T20:45:00.8338107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-26T20:45:00.8338247Z layer_output = self.output(intermediate_output, attention_output) 2025-08-26T20:45:00.8338516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-26T20:45:00.8338597Z hidden_states = self.dense(hidden_states) 2025-08-26T20:45:00.8338600Z 2025-08-26T20:45:00.8338731Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8338950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8339027Z return mod(**inputs) 2025-08-26T20:45:00.8339297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-08-26T20:45:00.8339459Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-08-26T20:45:00.8339727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 876, in forward 2025-08-26T20:45:00.8339832Z hidden_states = self.dense(generator_hidden_states) 2025-08-26T20:45:00.8339836Z 2025-08-26T20:45:00.8339948Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8340148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8340224Z return mod(**inputs) 2025-08-26T20:45:00.8340493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 939, in forward 2025-08-26T20:45:00.8340644Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-08-26T20:45:00.8340658Z 2025-08-26T20:45:00.8340767Z cudagraph partition due to non gpu ops. Found from : 2025-08-26T20:45:00.8340976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-08-26T20:45:00.8341053Z return mod(**inputs) 2025-08-26T20:45:00.8341346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 945, in forward 2025-08-26T20:45:00.8341539Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-26T20:45:00.8341543Z 2025-08-26T20:45:10.6262069Z Compilation time (from dynamo_timed): 21.516617397 2025-08-26T20:45:10.6297714Z pass 2025-08-26T20:45:10.6302985Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-26T20:45:10.6307316Z TIMING: _recursive_pre_grad_passes:0.01096 _recursive_joint_graph_passes:0.64178 _recursive_post_grad_passes:0.18334 async_compile.wait:0.59707 code_gen:9.25099 inductor_compile:11.80234 backend_compile:17.00103 gc:0.0007 entire_frame_compile:21.51662 total_wall_time:21.51662 2025-08-26T20:45:10.6308393Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:23079 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2025-08-26T20:45:10.6308982Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-08-26T20:45:12.8313981Z accuracy pass_rate=95.35% 2025-08-26T20:45:12.8316147Z calls_captured gmean=0.00x mean=609.233x 2025-08-26T20:45:12.8317242Z unique_graphs gmean=0.00x mean=1.093x 2025-08-26T20:45:12.8317563Z graph_breaks gmean=0.00x mean=0.140x 2025-08-26T20:45:12.8317848Z unique_graph_breaks gmean=0.00x mean=0.047x 2025-08-26T20:45:12.8319226Z autograd_captures gmean=0.00x mean=0.000x 2025-08-26T20:45:12.8324262Z autograd_compiles gmean=0.00x mean=0.000x 2025-08-26T20:45:12.8329333Z cudagraph_skips gmean=0.00x mean=1.093x 2025-08-26T20:45:12.8329642Z compilation_latency mean=20.803 seconds 2025-08-26T20:45:13.8605162Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-08-26T20:45:14.1353475Z AlbertForMaskedLM PASS 2025-08-26T20:45:14.1353843Z AlbertForQuestionAnswering PASS 2025-08-26T20:45:14.1358174Z AllenaiLongformerBase PASS 2025-08-26T20:45:14.1358528Z BartForCausalLM PASS 2025-08-26T20:45:14.1359669Z BartForConditionalGeneration PASS 2025-08-26T20:45:14.1367337Z BertForMaskedLM PASS 2025-08-26T20:45:14.1369103Z BertForQuestionAnswering PASS 2025-08-26T20:45:14.1373680Z BlenderbotForCausalLM XFAIL 2025-08-26T20:45:14.1377308Z BlenderbotSmallForCausalLM PASS 2025-08-26T20:45:14.1381434Z BlenderbotSmallForConditionalGeneration PASS 2025-08-26T20:45:14.1389821Z CamemBert PASS 2025-08-26T20:45:14.1390103Z DebertaV2ForMaskedLM XFAIL 2025-08-26T20:45:14.1390361Z DebertaV2ForQuestionAnswering PASS 2025-08-26T20:45:14.1390611Z DistilBertForMaskedLM PASS 2025-08-26T20:45:14.1395089Z DistilBertForQuestionAnswering PASS 2025-08-26T20:45:14.1398327Z DistillGPT2 PASS 2025-08-26T20:45:14.1403315Z ElectraForCausalLM PASS 2025-08-26T20:45:14.1410627Z ElectraForQuestionAnswering PASS 2025-08-26T20:45:14.1410979Z GPT2ForSequenceClassification PASS 2025-08-26T20:45:14.1416468Z GoogleFnet PASS 2025-08-26T20:45:14.1421256Z LayoutLMForMaskedLM PASS 2025-08-26T20:45:14.1424677Z LayoutLMForSequenceClassification PASS 2025-08-26T20:45:14.1424941Z M2M100ForConditionalGeneration PASS 2025-08-26T20:45:14.1431437Z MBartForCausalLM PASS 2025-08-26T20:45:14.1431736Z MBartForConditionalGeneration PASS 2025-08-26T20:45:14.1436642Z MT5ForConditionalGeneration PASS 2025-08-26T20:45:14.1436891Z MegatronBertForCausalLM PASS 2025-08-26T20:45:14.1438563Z MegatronBertForQuestionAnswering PASS 2025-08-26T20:45:14.1450539Z MobileBertForMaskedLM PASS 2025-08-26T20:45:14.1451007Z MobileBertForQuestionAnswering PASS 2025-08-26T20:45:14.1455135Z OPTForCausalLM PASS 2025-08-26T20:45:14.1455589Z PLBartForCausalLM PASS 2025-08-26T20:45:14.1455970Z PLBartForConditionalGeneration PASS 2025-08-26T20:45:14.1460219Z PegasusForCausalLM PASS 2025-08-26T20:45:14.1464216Z PegasusForConditionalGeneration PASS 2025-08-26T20:45:14.1467788Z RobertaForCausalLM PASS 2025-08-26T20:45:14.1468050Z RobertaForQuestionAnswering PASS 2025-08-26T20:45:14.1473947Z T5ForConditionalGeneration PASS 2025-08-26T20:45:14.1478201Z T5Small PASS 2025-08-26T20:45:14.1478490Z TrOCRForCausalLM PASS 2025-08-26T20:45:14.1490368Z XGLMForCausalLM PASS 2025-08-26T20:45:14.1490659Z XLNetLMHeadModel PASS 2025-08-26T20:45:14.1490887Z YituTechConvBert PASS 2025-08-26T20:45:14.2028968Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-08-26T20:45:14.4797493Z AlbertForMaskedLM PASS 2025-08-26T20:45:14.4797828Z AlbertForQuestionAnswering PASS 2025-08-26T20:45:14.4798321Z AllenaiLongformerBase PASS 2025-08-26T20:45:14.4809925Z BartForCausalLM PASS 2025-08-26T20:45:14.4815728Z BartForConditionalGeneration PASS 2025-08-26T20:45:14.4820785Z BertForMaskedLM PASS 2025-08-26T20:45:14.4825797Z BertForQuestionAnswering PASS 2025-08-26T20:45:14.4830718Z BlenderbotForCausalLM PASS 2025-08-26T20:45:14.4832886Z BlenderbotSmallForCausalLM PASS 2025-08-26T20:45:14.4833202Z BlenderbotSmallForConditionalGeneration PASS 2025-08-26T20:45:14.4833473Z CamemBert PASS 2025-08-26T20:45:14.4833696Z DebertaV2ForMaskedLM PASS 2025-08-26T20:45:14.4837048Z DebertaV2ForQuestionAnswering PASS 2025-08-26T20:45:14.4837406Z DistilBertForMaskedLM PASS 2025-08-26T20:45:14.4838900Z DistilBertForQuestionAnswering PASS 2025-08-26T20:45:14.4847995Z DistillGPT2 PASS 2025-08-26T20:45:14.4848436Z ElectraForCausalLM PASS 2025-08-26T20:45:14.4854101Z ElectraForQuestionAnswering PASS 2025-08-26T20:45:14.4854739Z GPT2ForSequenceClassification PASS 2025-08-26T20:45:14.4857368Z GoogleFnet PASS 2025-08-26T20:45:14.4858900Z LayoutLMForMaskedLM PASS 2025-08-26T20:45:14.4863073Z LayoutLMForSequenceClassification PASS 2025-08-26T20:45:14.4865120Z M2M100ForConditionalGeneration PASS 2025-08-26T20:45:14.4868098Z MBartForCausalLM PASS 2025-08-26T20:45:14.4873474Z MBartForConditionalGeneration PASS 2025-08-26T20:45:14.4876425Z MT5ForConditionalGeneration PASS 2025-08-26T20:45:14.4878767Z MegatronBertForCausalLM PASS 2025-08-26T20:45:14.4888446Z MegatronBertForQuestionAnswering PASS 2025-08-26T20:45:14.4888744Z MobileBertForMaskedLM PASS 2025-08-26T20:45:14.4899003Z MobileBertForQuestionAnswering PASS 2025-08-26T20:45:14.4904623Z OPTForCausalLM PASS 2025-08-26T20:45:14.4910493Z PLBartForCausalLM PASS 2025-08-26T20:45:14.4915978Z PLBartForConditionalGeneration PASS 2025-08-26T20:45:14.4916295Z PegasusForCausalLM PASS 2025-08-26T20:45:14.4916561Z PegasusForConditionalGeneration PASS 2025-08-26T20:45:14.4917011Z RobertaForCausalLM PASS 2025-08-26T20:45:14.4917256Z RobertaForQuestionAnswering PASS 2025-08-26T20:45:14.4917497Z T5ForConditionalGeneration PASS 2025-08-26T20:45:14.4920326Z T5Small PASS 2025-08-26T20:45:14.4934621Z TrOCRForCausalLM PASS 2025-08-26T20:45:14.4936493Z XGLMForCausalLM PASS_BUT_FLAKY 2025-08-26T20:45:14.4939871Z XLNetLMHeadModel PASS 2025-08-26T20:45:14.4940359Z YituTechConvBert PASS 2025-08-26T20:45:14.5449629Z + sccache_epilogue 2025-08-26T20:45:14.5452459Z + echo '::group::Sccache Compilation Log' 2025-08-26T20:45:14.5453102Z ##[group]Sccache Compilation Log 2025-08-26T20:45:14.5457456Z + echo '=================== sccache compilation log ===================' 2025-08-26T20:45:14.5457838Z =================== sccache compilation log =================== 2025-08-26T20:45:14.5458289Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-08-26T20:45:14.5686463Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-08-26T20:45:14.5687003Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-08-26T20:45:14.5687643Z + sccache --show-stats 2025-08-26T20:45:14.5721804Z Compile requests 381 2025-08-26T20:45:14.5722257Z Compile requests executed 0 2025-08-26T20:45:14.5722584Z Cache hits 0 2025-08-26T20:45:14.5722908Z Cache misses 0 2025-08-26T20:45:14.5723300Z Cache hits rate - 2025-08-26T20:45:14.5723543Z Cache timeouts 0 2025-08-26T20:45:14.5723824Z Cache read errors 0 2025-08-26T20:45:14.5724083Z Forced recaches 0 2025-08-26T20:45:14.5724400Z Cache write errors 0 2025-08-26T20:45:14.5724716Z Cache errors 0 2025-08-26T20:45:14.5725017Z Compilations 0 2025-08-26T20:45:14.5725851Z Compilation failures 0 2025-08-26T20:45:14.5726200Z Non-cacheable compilations 0 2025-08-26T20:45:14.5726448Z Non-cacheable calls 41 2025-08-26T20:45:14.5726687Z Non-compilation calls 340 2025-08-26T20:45:14.5726915Z Unsupported compiler calls 0 2025-08-26T20:45:14.5727197Z Average cache write 0.000 s 2025-08-26T20:45:14.5727432Z Average compiler 0.000 s 2025-08-26T20:45:14.5727665Z Average cache read hit 0.000 s 2025-08-26T20:45:14.5727967Z Failed distributed compilations 0 2025-08-26T20:45:14.5728136Z 2025-08-26T20:45:14.5728215Z Non-cacheable reasons: 2025-08-26T20:45:14.5728714Z -E 41 2025-08-26T20:45:14.5728894Z 2025-08-26T20:45:14.5729158Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-08-26T20:45:14.5729505Z Version (client) 0.10.0 2025-08-26T20:45:14.5729734Z + sccache --stop-server 2025-08-26T20:45:14.5742850Z Stopping sccache server... 2025-08-26T20:45:14.5749574Z Compile requests 381 2025-08-26T20:45:14.5749826Z Compile requests executed 0 2025-08-26T20:45:14.5750071Z Cache hits 0 2025-08-26T20:45:14.5750269Z Cache misses 0 2025-08-26T20:45:14.5750505Z Cache hits rate - 2025-08-26T20:45:14.5750771Z Cache timeouts 0 2025-08-26T20:45:14.5750980Z Cache read errors 0 2025-08-26T20:45:14.5751201Z Forced recaches 0 2025-08-26T20:45:14.5751410Z Cache write errors 0 2025-08-26T20:45:14.5751622Z Cache errors 0 2025-08-26T20:45:14.5751827Z Compilations 0 2025-08-26T20:45:14.5752034Z Compilation failures 0 2025-08-26T20:45:14.5752443Z Non-cacheable compilations 0 2025-08-26T20:45:14.5752661Z Non-cacheable calls 41 2025-08-26T20:45:14.5752877Z Non-compilation calls 340 2025-08-26T20:45:14.5753090Z Unsupported compiler calls 0 2025-08-26T20:45:14.5753315Z Average cache write 0.000 s 2025-08-26T20:45:14.5753546Z Average compiler 0.000 s 2025-08-26T20:45:14.5753768Z Average cache read hit 0.000 s 2025-08-26T20:45:14.5754041Z Failed distributed compilations 0 2025-08-26T20:45:14.5754192Z 2025-08-26T20:45:14.5754269Z Non-cacheable reasons: 2025-08-26T20:45:14.5754464Z -E 41 2025-08-26T20:45:14.5754597Z 2025-08-26T20:45:14.5754776Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-08-26T20:45:14.5755096Z Version (client) 0.10.0 2025-08-26T20:45:14.5755360Z + echo ::endgroup:: 2025-08-26T20:45:14.5755949Z ##[endgroup] 2025-08-26T20:45:14.5756132Z + cleanup_workspace 2025-08-26T20:45:14.5756475Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-08-26T20:45:14.5756975Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-08-26T20:45:14.5757386Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-08-26T20:45:14.5757699Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-08-26T20:45:14.5758077Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-08-26T20:45:14.5758469Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-08-26T20:45:14.5758790Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-08-26T20:45:15.0044263Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-08-26T20:45:15.0044648Z with: 2025-08-26T20:45:15.0044879Z benchmark-results-dir: test/test-reports 2025-08-26T20:45:15.0045150Z dry-run: false 2025-08-26T20:45:15.0045365Z schema-version: v3 2025-08-26T20:45:15.0045815Z github-token: *** 2025-08-26T20:45:15.0046010Z env: 2025-08-26T20:45:15.0046198Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:15.0046563Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:15.0046947Z ##[endgroup] 2025-08-26T20:45:15.0075097Z ##[group]Run set -eux 2025-08-26T20:45:15.0075324Z set -eux 2025-08-26T20:45:15.0075607Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-26T20:45:15.0075914Z  2025-08-26T20:45:15.0076092Z DEVICE_NAME="" 2025-08-26T20:45:15.0076287Z DEVICE_TYPE="" 2025-08-26T20:45:15.0076478Z  2025-08-26T20:45:15.0076699Z if command -v nvidia-smi; then 2025-08-26T20:45:15.0077021Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-08-26T20:45:15.0077416Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-08-26T20:45:15.0077794Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-08-26T20:45:15.0078101Z  python3 -mpip install torch==2.7.1 2025-08-26T20:45:15.0078354Z elif command -v rocminfo; then 2025-08-26T20:45:15.0078660Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-08-26T20:45:15.0079041Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-08-26T20:45:15.0079815Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-08-26T20:45:15.0080174Z  # GPU device name coming from rocminfo instead 2025-08-26T20:45:15.0080439Z  DEVICE_NAME=rocm 2025-08-26T20:45:15.0080785Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-08-26T20:45:15.0081130Z fi 2025-08-26T20:45:15.0081298Z  2025-08-26T20:45:15.0081507Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-08-26T20:45:15.0081884Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-08-26T20:45:15.0090868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:15.0091144Z env: 2025-08-26T20:45:15.0091323Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:15.0091675Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:15.0092027Z ##[endgroup] 2025-08-26T20:45:15.0123012Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-26T20:45:15.2046364Z Defaulting to user installation because normal site-packages is not writeable 2025-08-26T20:45:15.9654247Z Collecting boto3==1.35.33 2025-08-26T20:45:15.9795842Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-08-26T20:45:16.2145751Z Collecting psutil==7.0.0 2025-08-26T20:45:16.2179594Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-08-26T20:45:16.2437017Z Collecting pynvml==12.0.0 2025-08-26T20:45:16.2465211Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-08-26T20:45:16.2549605Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-08-26T20:45:17.0975089Z Collecting botocore<1.36.0,>=1.35.33 2025-08-26T20:45:17.1002341Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-08-26T20:45:17.2347766Z Collecting s3transfer<0.11.0,>=0.10.0 2025-08-26T20:45:17.2379190Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-08-26T20:45:17.2752980Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-08-26T20:45:17.2786657Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-08-26T20:45:17.2880770Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-08-26T20:45:17.2889918Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-08-26T20:45:17.4437512Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-08-26T20:45:17.5516544Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-08-26T20:45:17.9125397Z Attempting uninstall: nvidia-ml-py 2025-08-26T20:45:17.9130698Z Found existing installation: nvidia-ml-py 11.525.84 2025-08-26T20:45:17.9138158Z Uninstalling nvidia-ml-py-11.525.84: 2025-08-26T20:45:17.9270392Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-08-26T20:45:17.9788033Z Attempting uninstall: psutil 2025-08-26T20:45:17.9788453Z Found existing installation: psutil 5.9.8 2025-08-26T20:45:17.9843510Z Uninstalling psutil-5.9.8: 2025-08-26T20:45:17.9848669Z Successfully uninstalled psutil-5.9.8 2025-08-26T20:45:18.1252460Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-08-26T20:45:18.2444677Z + DEVICE_NAME= 2025-08-26T20:45:18.2448266Z + DEVICE_TYPE= 2025-08-26T20:45:18.2448685Z + command -v nvidia-smi 2025-08-26T20:45:18.2449102Z + command -v rocminfo 2025-08-26T20:45:18.2449400Z + echo DEVICE_NAME= 2025-08-26T20:45:18.2449590Z + echo DEVICE_TYPE= 2025-08-26T20:45:18.2474615Z ##[group]Run set -eux 2025-08-26T20:45:18.2474836Z set -eux 2025-08-26T20:45:18.2475002Z  2025-08-26T20:45:18.2475179Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-08-26T20:45:18.2475431Z  echo "Missing github-token input" 2025-08-26T20:45:18.2475649Z  exit 1 2025-08-26T20:45:18.2475819Z fi 2025-08-26T20:45:18.2480881Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:18.2481132Z env: 2025-08-26T20:45:18.2481294Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:18.2481601Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:18.2482004Z DEVICE_NAME: 2025-08-26T20:45:18.2482163Z DEVICE_TYPE: 2025-08-26T20:45:18.2482542Z GITHUB_TOKEN: *** 2025-08-26T20:45:18.2482711Z ##[endgroup] 2025-08-26T20:45:18.2505811Z + [[ -z *** ]] 2025-08-26T20:45:18.2551560Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-08-26T20:45:18.2551856Z with: 2025-08-26T20:45:18.2552194Z github-token: *** 2025-08-26T20:45:18.2552478Z env: 2025-08-26T20:45:18.2552664Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:18.2553028Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:18.2553409Z DEVICE_NAME: 2025-08-26T20:45:18.2553604Z DEVICE_TYPE: 2025-08-26T20:45:18.2553807Z ##[endgroup] 2025-08-26T20:45:18.2575881Z ##[group]Run set -eux 2025-08-26T20:45:18.2576088Z set -eux 2025-08-26T20:45:18.2576257Z  2025-08-26T20:45:18.2576584Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-26T20:45:18.2581243Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:18.2581488Z env: 2025-08-26T20:45:18.2581660Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:18.2581985Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:18.2582340Z DEVICE_NAME: 2025-08-26T20:45:18.2582516Z DEVICE_TYPE: 2025-08-26T20:45:18.2582843Z GITHUB_TOKEN: *** 2025-08-26T20:45:18.2583019Z ##[endgroup] 2025-08-26T20:45:18.2608437Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 17248463670 i-04c468ba96b53884f 2025-08-26T20:45:19.3215393Z setting job-id=48946862580 2025-08-26T20:45:19.3215949Z setting job-name=linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:45:19.3316042Z ##[group]Run set -eux 2025-08-26T20:45:19.3316291Z set -eux 2025-08-26T20:45:19.3316465Z  2025-08-26T20:45:19.3316738Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-08-26T20:45:19.3317081Z  --schema-version "${SCHEMA_VERSION}" \ 2025-08-26T20:45:19.3317330Z  --repo "${REPO}" \ 2025-08-26T20:45:19.3317557Z  --head-branch "${HEAD_BRANCH}" \ 2025-08-26T20:45:19.3317795Z  --head-sha "${HEAD_SHA}" \ 2025-08-26T20:45:19.3318039Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-08-26T20:45:19.3318294Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-08-26T20:45:19.3318529Z  --job-id "${JOB_ID}" \ 2025-08-26T20:45:19.3318755Z  --job-name "${JOB_NAME}" 2025-08-26T20:45:19.3323972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:19.3324217Z env: 2025-08-26T20:45:19.3324381Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:19.3324695Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:19.3325026Z DEVICE_NAME: 2025-08-26T20:45:19.3325179Z DEVICE_TYPE: 2025-08-26T20:45:19.3325339Z SCHEMA_VERSION: v3 2025-08-26T20:45:19.3325515Z REPO: pytorch/pytorch 2025-08-26T20:45:19.3325699Z HEAD_BRANCH: refs/heads/main 2025-08-26T20:45:19.3325908Z HEAD_SHA: 262640fd220236042fbf4443cc163c8838c84c3d 2025-08-26T20:45:19.3326132Z WORKFLOW_RUN_ID: 17248463670 2025-08-26T20:45:19.3326307Z RUN_ATTEMPT: 1 2025-08-26T20:45:19.3326467Z JOB_ID: 48946862580 2025-08-26T20:45:19.3326814Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:45:19.3327180Z ##[endgroup] 2025-08-26T20:45:19.3353196Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 262640fd220236042fbf4443cc163c8838c84c3d --workflow-id 17248463670 --run-attempt 1 --job-id 48946862580 --job-name 'linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-08-26T20:45:19.3615490Z ##[group]Run set -eux 2025-08-26T20:45:19.3615710Z set -eux 2025-08-26T20:45:19.3615881Z  2025-08-26T20:45:19.3616151Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-08-26T20:45:19.3620892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:19.3621149Z env: 2025-08-26T20:45:19.3621314Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:19.3621812Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:19.3622175Z DEVICE_NAME: 2025-08-26T20:45:19.3622348Z DEVICE_TYPE: 2025-08-26T20:45:19.3622519Z ##[endgroup] 2025-08-26T20:45:19.3647200Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-08-26T20:45:19.3980302Z INFO:root:Fail to import torch to get the device name 2025-08-26T20:45:19.4076149Z ##[group]Run set -eux 2025-08-26T20:45:19.4076356Z set -eux 2025-08-26T20:45:19.4076526Z  2025-08-26T20:45:19.4076711Z # TODO (huydhn): Implement this part 2025-08-26T20:45:19.4076983Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-08-26T20:45:19.4082100Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:19.4082386Z env: 2025-08-26T20:45:19.4082556Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:19.4082893Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:19.4083382Z DEVICE_NAME: 2025-08-26T20:45:19.4083564Z DEVICE_TYPE: 2025-08-26T20:45:19.4083739Z ##[endgroup] 2025-08-26T20:45:19.4106338Z + echo 'dependencies={}' 2025-08-26T20:45:19.4135566Z ##[group]Run set -eux 2025-08-26T20:45:19.4135798Z set -eux 2025-08-26T20:45:19.4135988Z  2025-08-26T20:45:19.4136182Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-08-26T20:45:19.4136528Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-08-26T20:45:19.4136850Z  # We don't want the job to fail if the directory doesn't exist 2025-08-26T20:45:19.4137110Z  exit 0 2025-08-26T20:45:19.4137268Z fi 2025-08-26T20:45:19.4137418Z  2025-08-26T20:45:19.4137585Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-08-26T20:45:19.4137893Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-26T20:45:19.4138269Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-26T20:45:19.4138565Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-26T20:45:19.4138795Z  --runners "${RUNNER_INFO}" \ 2025-08-26T20:45:19.4139018Z  --dependencies "${DEPENDENCIES}" \ 2025-08-26T20:45:19.4139235Z  --dry-run 2025-08-26T20:45:19.4139409Z else 2025-08-26T20:45:19.4139661Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-26T20:45:19.4139978Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-26T20:45:19.4140236Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-26T20:45:19.4140503Z  --runners "${RUNNER_INFO}" \ 2025-08-26T20:45:19.4140727Z  --dependencies "${DEPENDENCIES}" 2025-08-26T20:45:19.4140929Z fi 2025-08-26T20:45:19.4144620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:19.4144857Z env: 2025-08-26T20:45:19.4145014Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:19.4145320Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:19.4145623Z DEVICE_NAME: 2025-08-26T20:45:19.4145779Z DEVICE_TYPE: 2025-08-26T20:45:19.4145955Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-08-26T20:45:19.4146239Z DRY_RUN: false 2025-08-26T20:45:19.4147079Z BENCHMARK_METADATA: {"timestamp": 1756241119, "schema_version": "v3", "name": "linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "262640fd220236042fbf4443cc163c8838c84c3d", "workflow_id": 17248463670, "run_attempt": 1, "job_id": 48946862580} 2025-08-26T20:45:19.4148123Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-58-230.ec2.internal"}, "name": "", "type": ""}] 2025-08-26T20:45:19.4148536Z DEPENDENCIES: {} 2025-08-26T20:45:19.4148696Z ##[endgroup] 2025-08-26T20:45:19.4171962Z + [[ ! -d test/test-reports ]] 2025-08-26T20:45:19.4172981Z + [[ false == \t\r\u\e ]] 2025-08-26T20:45:19.4175046Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1756241119, "schema_version": "v3", "name": "linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "262640fd220236042fbf4443cc163c8838c84c3d", "workflow_id": 17248463670, "run_attempt": 1, "job_id": 48946862580}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-58-230.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-08-26T20:45:19.5404175Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17248463670/48946862580/inference_huggingface.json 2025-08-26T20:45:19.5709217Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-08-26T20:45:19.8001687Z ##[group]Run cat test/**/*_toprint.log || true 2025-08-26T20:45:19.8002035Z cat test/**/*_toprint.log || true 2025-08-26T20:45:19.8006699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:19.8006934Z env: 2025-08-26T20:45:19.8007096Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:19.8007394Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:19.8007704Z DEVICE_NAME: 2025-08-26T20:45:19.8007856Z DEVICE_TYPE: 2025-08-26T20:45:19.8008014Z ##[endgroup] 2025-08-26T20:45:19.8085382Z cat: 'test/**/*_toprint.log': No such file or directory 2025-08-26T20:45:19.8119905Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-08-26T20:45:19.8120196Z kill "$MONITOR_SCRIPT_PID" 2025-08-26T20:45:19.8125084Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:19.8125353Z env: 2025-08-26T20:45:19.8125523Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:19.8125858Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:19.8126203Z DEVICE_NAME: 2025-08-26T20:45:19.8126395Z DEVICE_TYPE: 2025-08-26T20:45:19.8126576Z MONITOR_SCRIPT_PID: 48598 2025-08-26T20:45:19.8126779Z ##[endgroup] 2025-08-26T20:45:19.8223981Z Prepare all required actions 2025-08-26T20:45:19.8224369Z Getting action download info 2025-08-26T20:45:19.9892796Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-08-26T20:45:20.2030256Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-08-26T20:45:20.5657565Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-08-26T20:45:20.5657819Z with: 2025-08-26T20:45:20.5658104Z file-suffix: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:45:20.5658425Z s3-bucket: gha-artifacts 2025-08-26T20:45:20.5658598Z env: 2025-08-26T20:45:20.5658751Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:20.5659068Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:20.5659487Z DEVICE_NAME: 2025-08-26T20:45:20.5659643Z DEVICE_TYPE: 2025-08-26T20:45:20.5659803Z ##[endgroup] 2025-08-26T20:45:20.5682175Z ##[group]Run # Remove any previous test jsons if they exist 2025-08-26T20:45:20.5682518Z # Remove any previous test jsons if they exist 2025-08-26T20:45:20.5682773Z rm -f test-jsons-*.zip 2025-08-26T20:45:20.5683075Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-08-26T20:45:20.5687846Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:20.5688106Z env: 2025-08-26T20:45:20.5688269Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:20.5688597Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:20.5688933Z DEVICE_NAME: 2025-08-26T20:45:20.5689160Z DEVICE_TYPE: 2025-08-26T20:45:20.5689450Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:45:20.5689768Z ##[endgroup] 2025-08-26T20:45:20.5855490Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-08-26T20:45:20.5881177Z ##[group]Run # Remove any previous test reports if they exist 2025-08-26T20:45:20.5881545Z # Remove any previous test reports if they exist 2025-08-26T20:45:20.5881813Z rm -f test-reports-*.zip 2025-08-26T20:45:20.5882152Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-08-26T20:45:20.5886871Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:20.5887138Z env: 2025-08-26T20:45:20.5887311Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:20.5887641Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:20.5887973Z DEVICE_NAME: 2025-08-26T20:45:20.5888145Z DEVICE_TYPE: 2025-08-26T20:45:20.5888437Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:45:20.5888761Z ##[endgroup] 2025-08-26T20:45:20.5944038Z adding: test/test-reports/inference_huggingface.csv (deflated 69%) 2025-08-26T20:45:20.5944557Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-08-26T20:45:20.5945153Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 63%) 2025-08-26T20:45:20.5965797Z ##[group]Run # Remove any previous usage logs if they exist 2025-08-26T20:45:20.5966139Z # Remove any previous usage logs if they exist 2025-08-26T20:45:20.5966415Z rm -f logs-*.zip 2025-08-26T20:45:20.5966675Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-08-26T20:45:20.5967021Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-08-26T20:45:20.5971571Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:20.5971842Z env: 2025-08-26T20:45:20.5972010Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:20.5972338Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:20.5972673Z DEVICE_NAME: 2025-08-26T20:45:20.5972844Z DEVICE_TYPE: 2025-08-26T20:45:20.5973251Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:45:20.5973583Z ##[endgroup] 2025-08-26T20:45:20.6038956Z adding: usage_log.txt (deflated 96%) 2025-08-26T20:45:20.6050994Z 2025-08-26T20:45:20.6051542Z zip error: Nothing to do! (logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip) 2025-08-26T20:45:20.6077525Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-08-26T20:45:20.6077915Z # Remove any previous debugging artifacts if they exist 2025-08-26T20:45:20.6078191Z rm -f debug-*.zip 2025-08-26T20:45:20.6078406Z if [ -d 'test/debug' ]; then 2025-08-26T20:45:20.6078663Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-08-26T20:45:20.6078898Z fi 2025-08-26T20:45:20.6083647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:20.6084000Z env: 2025-08-26T20:45:20.6084169Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:20.6084509Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:20.6084857Z DEVICE_NAME: 2025-08-26T20:45:20.6085024Z DEVICE_TYPE: 2025-08-26T20:45:20.6085319Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580 2025-08-26T20:45:20.6085643Z ##[endgroup] 2025-08-26T20:45:20.6152732Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-26T20:45:20.6152963Z with: 2025-08-26T20:45:20.6153147Z s3-bucket: gha-artifacts 2025-08-26T20:45:20.6153386Z s3-prefix: pytorch/pytorch/17248463670/1/artifact 2025-08-26T20:45:20.6153629Z retention-days: 14 2025-08-26T20:45:20.6153810Z if-no-files-found: warn 2025-08-26T20:45:20.6154007Z path: test-jsons-*.zip 2025-08-26T20:45:20.6154194Z name: artifact 2025-08-26T20:45:20.6154369Z region: us-east-1 2025-08-26T20:45:20.6154535Z env: 2025-08-26T20:45:20.6154699Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:20.6155054Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:20.6155402Z DEVICE_NAME: 2025-08-26T20:45:20.6155570Z DEVICE_TYPE: 2025-08-26T20:45:20.6155740Z ##[endgroup] 2025-08-26T20:45:20.8786098Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-26T20:45:20.8786475Z With the provided path, there will be 1 file uploaded 2025-08-26T20:45:20.8786807Z Uploading to s3 prefix: pytorch/pytorch/17248463670/1/artifact 2025-08-26T20:45:20.8819680Z Starting upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:20.9906089Z Finished upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:21.0059411Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-26T20:45:21.0059659Z with: 2025-08-26T20:45:21.0059842Z s3-bucket: gha-artifacts 2025-08-26T20:45:21.0060084Z s3-prefix: pytorch/pytorch/17248463670/1/artifact 2025-08-26T20:45:21.0060358Z retention-days: 14 2025-08-26T20:45:21.0060554Z if-no-files-found: error 2025-08-26T20:45:21.0060756Z path: test-reports-*.zip 2025-08-26T20:45:21.0060944Z name: artifact 2025-08-26T20:45:21.0061116Z region: us-east-1 2025-08-26T20:45:21.0061278Z env: 2025-08-26T20:45:21.0061437Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:21.0061760Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:21.0062103Z DEVICE_NAME: 2025-08-26T20:45:21.0062264Z DEVICE_TYPE: 2025-08-26T20:45:21.0062428Z ##[endgroup] 2025-08-26T20:45:21.2602506Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-26T20:45:21.2602935Z With the provided path, there will be 1 file uploaded 2025-08-26T20:45:21.2603255Z Uploading to s3 prefix: pytorch/pytorch/17248463670/1/artifact 2025-08-26T20:45:21.2633752Z Starting upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:21.3679795Z Finished upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:21.3848530Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-26T20:45:21.3848786Z with: 2025-08-26T20:45:21.3848962Z s3-bucket: gha-artifacts 2025-08-26T20:45:21.3849202Z s3-prefix: pytorch/pytorch/17248463670/1/artifact 2025-08-26T20:45:21.3849449Z retention-days: 14 2025-08-26T20:45:21.3849631Z if-no-files-found: ignore 2025-08-26T20:45:21.3849843Z path: logs-*.zip 2025-08-26T20:45:21.3850017Z name: artifact 2025-08-26T20:45:21.3850187Z region: us-east-1 2025-08-26T20:45:21.3850349Z env: 2025-08-26T20:45:21.3850508Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:21.3850833Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:21.3851169Z DEVICE_NAME: 2025-08-26T20:45:21.3851331Z DEVICE_TYPE: 2025-08-26T20:45:21.3851497Z ##[endgroup] 2025-08-26T20:45:21.6497108Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-26T20:45:21.6497858Z With the provided path, there will be 1 file uploaded 2025-08-26T20:45:21.6498216Z Uploading to s3 prefix: pytorch/pytorch/17248463670/1/artifact 2025-08-26T20:45:21.6533187Z Starting upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:21.7677292Z Finished upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:21.7825609Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-26T20:45:21.7825857Z with: 2025-08-26T20:45:21.7826024Z s3-bucket: gha-artifacts 2025-08-26T20:45:21.7826251Z s3-prefix: pytorch/pytorch/17248463670/1/artifact 2025-08-26T20:45:21.7826483Z retention-days: 14 2025-08-26T20:45:21.7826652Z if-no-files-found: ignore 2025-08-26T20:45:21.7826839Z path: debug-*.zip 2025-08-26T20:45:21.7827003Z name: artifact 2025-08-26T20:45:21.7827163Z region: us-east-1 2025-08-26T20:45:21.7827344Z env: 2025-08-26T20:45:21.7827553Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:21.7827988Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:21.7828305Z DEVICE_NAME: 2025-08-26T20:45:21.7828455Z DEVICE_TYPE: 2025-08-26T20:45:21.7828610Z ##[endgroup] 2025-08-26T20:45:22.0376341Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-08-26T20:45:22.0552313Z ##[group]Run # shellcheck disable=SC2156 2025-08-26T20:45:22.0552604Z # shellcheck disable=SC2156 2025-08-26T20:45:22.0552986Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-08-26T20:45:22.0557940Z shell: /usr/bin/bash -e {0} 2025-08-26T20:45:22.0558142Z env: 2025-08-26T20:45:22.0558311Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:22.0558645Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:22.0558985Z DEVICE_NAME: 2025-08-26T20:45:22.0559151Z DEVICE_TYPE: 2025-08-26T20:45:22.0559332Z ##[endgroup] 2025-08-26T20:45:22.2374997Z Prepare all required actions 2025-08-26T20:45:22.2375365Z Getting action download info 2025-08-26T20:45:22.3679705Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-08-26T20:45:22.3679989Z with: 2025-08-26T20:45:22.3680163Z job_id: 48946862580 2025-08-26T20:45:22.3680551Z job_name: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:45:22.3680987Z workflow_name: inductor 2025-08-26T20:45:22.3681181Z workflow_run_id: 17248463670 2025-08-26T20:45:22.3681386Z workflow_attempt: 1 2025-08-26T20:45:22.3681549Z env: 2025-08-26T20:45:22.3681700Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:22.3681996Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:22.3682305Z DEVICE_NAME: 2025-08-26T20:45:22.3682465Z DEVICE_TYPE: 2025-08-26T20:45:22.3682619Z ##[endgroup] 2025-08-26T20:45:22.3698983Z ##[group]Run echo "workflow_id: 17248463670" 2025-08-26T20:45:22.3699298Z echo "workflow_id: 17248463670" 2025-08-26T20:45:22.3699534Z echo "workflow_attempt: 1" 2025-08-26T20:45:22.3699755Z echo "workflow_Name: inductor" 2025-08-26T20:45:22.3699980Z echo "job_id: 48946862580" 2025-08-26T20:45:22.3700407Z echo "job_name: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" 2025-08-26T20:45:22.3700836Z echo "artifact_prefix: " 2025-08-26T20:45:22.3701057Z python3 --version 2025-08-26T20:45:22.3705761Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:22.3706011Z env: 2025-08-26T20:45:22.3706171Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:22.3706481Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:22.3706796Z DEVICE_NAME: 2025-08-26T20:45:22.3706959Z DEVICE_TYPE: 2025-08-26T20:45:22.3707117Z ##[endgroup] 2025-08-26T20:45:22.3729597Z workflow_id: 17248463670 2025-08-26T20:45:22.3729874Z workflow_attempt: 1 2025-08-26T20:45:22.3730064Z workflow_Name: inductor 2025-08-26T20:45:22.3730252Z job_id: 48946862580 2025-08-26T20:45:22.3730663Z job_name: linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-26T20:45:22.3731073Z artifact_prefix: 2025-08-26T20:45:22.3742320Z Python 3.9.23 2025-08-26T20:45:22.3779006Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-26T20:45:22.3779347Z with: 2025-08-26T20:45:22.3779514Z shell: bash 2025-08-26T20:45:22.3779700Z timeout_minutes: 5 2025-08-26T20:45:22.3779922Z max_attempts: 5 2025-08-26T20:45:22.3780110Z retry_wait_seconds: 30 2025-08-26T20:45:22.3780489Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7 2025-08-26T20:45:22.3780899Z polling_interval_seconds: 1 2025-08-26T20:45:22.3781111Z warning_on_retry: true 2025-08-26T20:45:22.3781308Z continue_on_error: false 2025-08-26T20:45:22.3781520Z env: 2025-08-26T20:45:22.3781681Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:22.3782020Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:22.3782368Z DEVICE_NAME: 2025-08-26T20:45:22.3782545Z DEVICE_TYPE: 2025-08-26T20:45:22.3782710Z ##[endgroup] 2025-08-26T20:45:22.6475735Z Defaulting to user installation because normal site-packages is not writeable 2025-08-26T20:45:22.7084852Z Collecting python-dateutil==2.8.2 2025-08-26T20:45:22.7229697Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2025-08-26T20:45:23.4359821Z Collecting boto3==1.35.42 2025-08-26T20:45:23.4393007Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-08-26T20:45:23.8203458Z Collecting pandas==2.1.3 2025-08-26T20:45:23.8236887Z Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) 2025-08-26T20:45:23.9717858Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-08-26T20:45:23.9734157Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-08-26T20:45:23.9771544Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-08-26T20:45:23.9772177Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-08-26T20:45:23.9772792Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-08-26T20:45:24.0245537Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-08-26T20:45:24.0494259Z Collecting tzdata>=2022.1 2025-08-26T20:45:24.0531766Z Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB) 2025-08-26T20:45:24.6720254Z Collecting numpy<2,>=1.22.4 2025-08-26T20:45:24.6775849Z Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) 2025-08-26T20:45:24.8213524Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-08-26T20:45:24.8214383Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-08-26T20:45:24.8259946Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-08-26T20:45:24.8348080Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-08-26T20:45:24.8429496Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0) 2025-08-26T20:45:24.8430738Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.15.0) 2025-08-26T20:45:24.9983554Z Installing collected packages: python-dateutil, tzdata, numpy, pandas, boto3 2025-08-26T20:45:29.0609786Z Attempting uninstall: boto3 2025-08-26T20:45:29.0610128Z Found existing installation: boto3 1.35.33 2025-08-26T20:45:29.0683455Z Uninstalling boto3-1.35.33: 2025-08-26T20:45:29.0693278Z Successfully uninstalled boto3-1.35.33 2025-08-26T20:45:29.1154530Z Successfully installed boto3-1.35.42 numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2 2025-08-26T20:45:29.4477386Z Command completed after 1 attempt(s). 2025-08-26T20:45:29.4541908Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-08-26T20:45:29.4542380Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-08-26T20:45:29.4542746Z  --workflow-run-id "17248463670" \ 2025-08-26T20:45:29.4542994Z  --workflow-name "inductor" \ 2025-08-26T20:45:29.4543233Z  --workflow-run-attempt "1" \ 2025-08-26T20:45:29.4543456Z  --job-id "48946862580" \ 2025-08-26T20:45:29.4543881Z  --job-name "linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" \ 2025-08-26T20:45:29.4544312Z  --local-path "" \ 2025-08-26T20:45:29.4544526Z  --artifact-prefix "" 2025-08-26T20:45:29.4550560Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:29.4550821Z env: 2025-08-26T20:45:29.4550987Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:29.4551320Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:29.4551659Z DEVICE_NAME: 2025-08-26T20:45:29.4551833Z DEVICE_TYPE: 2025-08-26T20:45:29.4551998Z ##[endgroup] 2025-08-26T20:45:30.4267820Z repo: pytorch/pytorch 2025-08-26T20:45:30.4268830Z Search for test log in s3 bucket: ossci-utilization 2025-08-26T20:45:30.4269324Z Downloading logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:30.4269865Z extracting usage_log.txt from zip file logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48946862580.zip 2025-08-26T20:45:30.4270270Z Converted Log Model: UtilizationMetadata: 2025-08-26T20:45:30.4271190Z UtilizationMetadata(level='metadata', workflow_id='17248463670', job_id='48946862580', workflow_name='inductor', job_name='linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)', usage_collect_interval=1.0, data_model_version=1.5, start_at=1756239592, gpu_count=0, cpu_count=32, gpu_type=None, error=None) 2025-08-26T20:45:30.4272146Z [Db Segments] detected pytest cmd: 9, generated segments: 9 2025-08-26T20:45:30.4272402Z [db model] Peek db timeseries 2025-08-26T20:45:30.4272593Z :{ 2025-08-26T20:45:30.4272741Z "created_at": 1756241130, 2025-08-26T20:45:30.4272929Z "type": "utilization", 2025-08-26T20:45:30.4273100Z "tags": [ 2025-08-26T20:45:30.4273244Z "record" 2025-08-26T20:45:30.4273386Z ], 2025-08-26T20:45:30.4273532Z "time_stamp": 1756239592, 2025-08-26T20:45:30.4273717Z "repo": "pytorch/pytorch", 2025-08-26T20:45:30.4273908Z "workflow_id": 17248463670, 2025-08-26T20:45:30.4274087Z "run_attempt": 1, 2025-08-26T20:45:30.4274262Z "job_id": 48946862580, 2025-08-26T20:45:30.4274446Z "workflow_name": "inductor", 2025-08-26T20:45:30.4274830Z "job_name": "linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", 2025-08-26T20:45:30.4275193Z "json_data": "{}" 2025-08-26T20:45:30.4275357Z } 2025-08-26T20:45:30.4275690Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/17248463670/1/48946862580/metadata 2025-08-26T20:45:30.4276267Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/17248463670/1/48946862580/metadata 2025-08-26T20:45:30.4277073Z Writing 304 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/17248463670/1/48946862580/time_series 2025-08-26T20:45:30.4277663Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/17248463670/1/48946862580/time_series 2025-08-26T20:45:30.5379461Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-08-26T20:45:30.5379923Z with: 2025-08-26T20:45:30.5380103Z env: 2025-08-26T20:45:30.5380282Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:30.5380662Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:30.5381049Z DEVICE_NAME: 2025-08-26T20:45:30.5381247Z DEVICE_TYPE: 2025-08-26T20:45:30.5381433Z ##[endgroup] 2025-08-26T20:45:30.5399374Z ##[group]Run set -eou pipefail 2025-08-26T20:45:30.5399877Z set -eou pipefail 2025-08-26T20:45:30.5400124Z  2025-08-26T20:45:30.5400443Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-08-26T20:45:30.5400823Z for _ in $(seq 1440); do 2025-08-26T20:45:30.5401108Z  # Break if no ssh session exists anymore 2025-08-26T20:45:30.5401397Z  if [ "$(who)" = "" ]; then 2025-08-26T20:45:30.5401647Z  break 2025-08-26T20:45:30.5401864Z  fi 2025-08-26T20:45:30.5402061Z  echo "." 2025-08-26T20:45:30.5402253Z  sleep 5 2025-08-26T20:45:30.5402445Z done 2025-08-26T20:45:30.5407363Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:30.5407622Z env: 2025-08-26T20:45:30.5407781Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:30.5408108Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:30.5408440Z DEVICE_NAME: 2025-08-26T20:45:30.5408607Z DEVICE_TYPE: 2025-08-26T20:45:30.5408771Z ##[endgroup] 2025-08-26T20:45:30.5432194Z Holding runner for 2 hours until all ssh sessions have logged out 2025-08-26T20:45:30.5509904Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-26T20:45:30.5510579Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-26T20:45:30.5510982Z # shellcheck disable=SC2046 2025-08-26T20:45:30.5511304Z docker stop $(docker ps -q) || true 2025-08-26T20:45:30.5511638Z # Prune all of the docker images 2025-08-26T20:45:30.5511960Z docker system prune -af 2025-08-26T20:45:30.5517136Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:30.5517403Z env: 2025-08-26T20:45:30.5517570Z GIT_DEFAULT_BRANCH: main 2025-08-26T20:45:30.5517935Z DOCKER_CONTAINER_ID: 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:30.5518284Z DEVICE_NAME: 2025-08-26T20:45:30.5518460Z DEVICE_TYPE: 2025-08-26T20:45:30.5518627Z ##[endgroup] 2025-08-26T20:45:41.4850085Z 0dca33bcc852 2025-08-26T20:45:41.7847559Z Deleted Containers: 2025-08-26T20:45:41.7847957Z 0dca33bcc85228d4f7babbeaa3b05b6a0983ad0c115212d2f1433227323840ce 2025-08-26T20:45:41.7848213Z 2025-08-26T20:45:49.0969108Z Deleted Images: 2025-08-26T20:45:49.0969927Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-16b1c8d10f4f7ec1a604612d52021e8c98b48fe6 2025-08-26T20:45:49.0970828Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:acbbd4ce4ca5911beba428e48e3c25069f341e6f142804bf943d333ccc654c8c 2025-08-26T20:45:49.0971424Z deleted: sha256:932da535a977ab0b0008738bb22f52295068c34f7c3dedec486c588f4545c297 2025-08-26T20:45:49.0971855Z deleted: sha256:3156ea3f68016dfccfe2a536aa882b62a00395a6ca49b2ba301d13e44abb83fb 2025-08-26T20:45:49.0972298Z deleted: sha256:9a45d555dff580d21be11d679a1e837b745b873cedadc9e566a602c328aed3f9 2025-08-26T20:45:49.0972737Z deleted: sha256:daacbcb681fd57619606aab266f3ef2ef8e95a94bc72b5bc883eb754da198dd0 2025-08-26T20:45:49.0973535Z deleted: sha256:6f3fbaa56c6a2e579d2ba504e6dc8988a53f21edd9362b80c637dd0f986af706 2025-08-26T20:45:49.0973970Z deleted: sha256:0f59cd25c9612e8273fe496db436f3e7b77bc98272ba190ce9d5fc52bfc6ae1e 2025-08-26T20:45:49.0974405Z deleted: sha256:bb7d42cfe899a5cc7890b1cdeacf96ab3078904e798814ae6553db5629b011b0 2025-08-26T20:45:49.0975011Z deleted: sha256:0368680d4759f526a0230f7ab947f5ff788c0051024f5a7a5d89370f8cfdf6c4 2025-08-26T20:45:49.0975712Z deleted: sha256:4fd2473fa0515f71b6eb686fcb67c9b81d1d8dc587090242be7c3bb856ec5279 2025-08-26T20:45:49.0976166Z deleted: sha256:3f3d8fdcc126ce6c4d64636da45b1974621083f0311cd39e83c44a95236d4b2d 2025-08-26T20:45:49.0976598Z deleted: sha256:d0efbbe1f1404811bd328b19ab44fb13919f9f167b541819b4318e6756b5dda5 2025-08-26T20:45:49.0977015Z deleted: sha256:e6b349b1e58ba586e3deb3614dd8f89d1827cf032075c5e6bda88d43427154a0 2025-08-26T20:45:49.0977431Z deleted: sha256:20d80dc7cd6833183b01c3047299ec4a4c0dff02c348453530dcd112f8aafadf 2025-08-26T20:45:49.0977860Z deleted: sha256:59ffae9fb0f9d6800e2af3eb4a9ba2e33dbe8542fa52adfe5f4a44d1110333ec 2025-08-26T20:45:49.0978300Z deleted: sha256:3c8c8c9d49865e4baf47d614cd14d1bc61d707d3b568805a4e85a7bcf74f79f5 2025-08-26T20:45:49.0978763Z deleted: sha256:da52ea4cd5269f152631fb9aebe1a3f2afbb25b6f905627f4d9e0dd66a2d76ad 2025-08-26T20:45:49.0979186Z deleted: sha256:21feaf7133ed123b02de1af3d0554bf8b5ec0d0658f3735d5afe31073cd48aaa 2025-08-26T20:45:49.0979614Z deleted: sha256:75d674077e9288a3c7e3d95753931a8dfbacf0903eab4115e51ca11e517cf487 2025-08-26T20:45:49.0980026Z deleted: sha256:408f7aac39a3a9d96ef050499909b911708700828d89eb6afa629f71735f1b7b 2025-08-26T20:45:49.0980439Z deleted: sha256:2fd2fb552e32e423f74651a7422e6026cb64ccefc0af2919003dcfdb896a6fca 2025-08-26T20:45:49.0980856Z deleted: sha256:124e8f08b0ae3f1512279119a6e0eace8c6033a79f54e8853e6ca505a06dbf55 2025-08-26T20:45:49.0981264Z deleted: sha256:886281f226562d96f829be9ad27d3cc2d60358e0b4a1f15e960772d0910effaa 2025-08-26T20:45:49.0981678Z deleted: sha256:da6d03400da08da89796db7be759be8ad8940d683e7f276123a5553ec9d24a57 2025-08-26T20:45:49.0982098Z deleted: sha256:70d05a51e316f5a19f8b0631b96ccfb897a650deb917ec5c086ce3fe65f1fd0c 2025-08-26T20:45:49.0982515Z deleted: sha256:9784a26692bf6c95593aaefafffa4d56455f6eb6467404cf4f76b79372f30e4f 2025-08-26T20:45:49.0982928Z deleted: sha256:2bcffdbb59956a4717723d51f4323e665b98deaf142776a33fd610c646db860f 2025-08-26T20:45:49.0983370Z deleted: sha256:8bbddb5220d45dafe0030c7757727c3c069d86030a553f41b1f26f2e0d84b6e4 2025-08-26T20:45:49.0983805Z deleted: sha256:962d7d894f9919c1971f86b96bd19bf53721b38c227cea9e529b1db7fc80fb2e 2025-08-26T20:45:49.0984224Z deleted: sha256:9c9d93e4d4dd16b3a3d281a9a900cfb97dcce951b24cbcdb2338bdc74a6c063f 2025-08-26T20:45:49.0984648Z deleted: sha256:898c4ae00d01adeac99b034ca9ae9d71f98043a619ed3416f6afeb33622037c7 2025-08-26T20:45:49.0985087Z deleted: sha256:e6f1aa0296a85597e241ca21553f75914bfd6f9c138dafa58a1da34b6ea6c3c0 2025-08-26T20:45:49.0985503Z deleted: sha256:f53cdb93298a9c02d5a507a0f8834ba20a1dab383657f2261301cae17c8c83d2 2025-08-26T20:45:49.0985925Z deleted: sha256:9cf9854b49330c66e9fd844ccc3176494b18db80b8befa296e5b4528173cf8d1 2025-08-26T20:45:49.0986360Z deleted: sha256:00c249d37f86b7ebc84025ef002cd2f7e980b907d0ec033ff7d0ea4d1264ab1a 2025-08-26T20:45:49.0986794Z deleted: sha256:79be195b392524620bc3c3037e351999a68113dcad267d070ec43525e589a60c 2025-08-26T20:45:49.0987199Z deleted: sha256:c3008072664f92ccb9ed58aab39622f94bb36bfd41a19e2f1d420507054e1827 2025-08-26T20:45:49.0987644Z deleted: sha256:f5a3048af287e09824ae55074388f59b1fa65f22fe6dccb6faa06f6278d43dce 2025-08-26T20:45:49.0988053Z deleted: sha256:0d870fd182b585783501e1d946e3acb1b558290e650bb698ae4b2cb9cf6880a8 2025-08-26T20:45:49.0988474Z deleted: sha256:93da7fe2be0b82e03b2b3ac15ab51c8faa1b2238e56cd9ca5ce1e7476e2305cb 2025-08-26T20:45:49.0988887Z deleted: sha256:483856587b063fee6068ef50607a29eb2b214aa6f30bc9ef6eedfe90e3cda082 2025-08-26T20:45:49.0989290Z deleted: sha256:8d51b4b73a66559bf8b432663d2d608fb10900e12e33d4c5d7865998f28eeb32 2025-08-26T20:45:49.0989778Z deleted: sha256:05c1eb9d6bf1ac198c607c93e2a3d516eafd0c5d0832a870662cbfa6e9672cf5 2025-08-26T20:45:49.0990198Z deleted: sha256:d6555842191f7d93047f5b8e764ee51e09297160989fcbd64012e35a25f31404 2025-08-26T20:45:49.0990626Z deleted: sha256:9c37a35dc9a8f4c40caf440422180306711115ba043c5faa502b8c96a5eeb543 2025-08-26T20:45:49.0991120Z deleted: sha256:f47a0e3db7dca4d8e4217670ded860c7245af2c360c65ade3e78ee74618e5abb 2025-08-26T20:45:49.0991559Z deleted: sha256:45bcab22be63d761cb4e663edded34b5fcf1bf35d81fdbde11acd976d8f056c7 2025-08-26T20:45:49.0991975Z deleted: sha256:df8174fd63de0f53060c60b4e8fd435d60b39bed840065d8d681b853201c4fc7 2025-08-26T20:45:49.0992370Z deleted: sha256:5434c1485755948e233661454a808122ee876fb38c8cf827ad068cfd16aa9911 2025-08-26T20:45:49.0992792Z deleted: sha256:fefabde8ff27ac23bcaf7b61b8eb5b29e3dc9695fab4d5dad6a144db987a1c61 2025-08-26T20:45:49.0993226Z deleted: sha256:b88942f7cd22bc5670adb08b936ea4be3c34e2dc744fd9ac21a12a02408286ff 2025-08-26T20:45:49.0993661Z deleted: sha256:5db012aa0d4b57a2eafe03bb22b876c52b0c37923895813ebe0e8b88db5d3d06 2025-08-26T20:45:49.0994081Z deleted: sha256:02480388e3201a70920862a36cde623dba5d8f0aef14cbc8b8d6e86227d21418 2025-08-26T20:45:49.0994488Z deleted: sha256:a713c70ee7bdc9693825a35cef98b0f3a271a0015c904f725a456e47280752d6 2025-08-26T20:45:49.0994913Z deleted: sha256:d8f3fd0e0ae0d3b955cda7a4e28615b2ab0f71dfe6234a1d13b2bb9fc190b8de 2025-08-26T20:45:49.0995387Z deleted: sha256:967a51e1cb7095e3ed117ca1aa460b40c01c247abd428e70098df9fc79b46dad 2025-08-26T20:45:49.0995808Z deleted: sha256:bb23a0f7a6c02fcdf389ef412f9ecd22aa476ff3328f0227a325605697041fa0 2025-08-26T20:45:49.0996409Z deleted: sha256:72aedc7bd8ad37c56ab3af00deee323a3462b5acd4a148b0bc60f1cf8f107054 2025-08-26T20:45:49.0996856Z deleted: sha256:328a95f2cc869f7472c8e8246d260f4f88415b814ded53dcf65e14e9a6e0ca7a 2025-08-26T20:45:49.0997284Z deleted: sha256:822b44de6305caad1e5dc05dd69cb1d97943643f9ee64920cfcb503e0bf42e97 2025-08-26T20:45:49.0997719Z deleted: sha256:0692ae1d6bb27b1f38130fe3821bfa22518fb62e9df7348ef592533bc6de0643 2025-08-26T20:45:49.0998136Z deleted: sha256:7f148c740013645d9ca7c81672751d4f5c663268c6f41bbb023376d95eab0fe9 2025-08-26T20:45:49.0998553Z deleted: sha256:888d91ae532d8da6c76dd06e66f0b9eba31a3c34db7473162310d3d826531677 2025-08-26T20:45:49.0998978Z deleted: sha256:95f9eb56b666bddc92611dde7a1e528d20727088fcab96029a285b87863e68fe 2025-08-26T20:45:49.0999667Z deleted: sha256:9ce7d3ff2bf5368b1093479ec795208bc9c27acf3f1eaf082ec961fb98ccc34c 2025-08-26T20:45:49.1000167Z deleted: sha256:d784bd35678b07daa5cad60903b2ad799d7a0b718f0369f5f9a025c907ecec88 2025-08-26T20:45:49.1000595Z deleted: sha256:2d79ff0ccf3568d27b32eba526c0685a4026a9aedc9bce178c62cc5300cecda2 2025-08-26T20:45:49.1001024Z deleted: sha256:e6d07d4c0bd50947e4d3c16528fab6c41cc912d7c237a9ea27db2d24228a88f4 2025-08-26T20:45:49.1001455Z deleted: sha256:5da975aef14c12e2b472b92aee3c3bdc962725ba0c16213dc38cb9dba53a0291 2025-08-26T20:45:49.1001871Z deleted: sha256:af0f5da6f1f197a45609d0e68e589e32b3aeb1daa69ab8e9df6f9ab5c80ec17b 2025-08-26T20:45:49.1002300Z deleted: sha256:f6e9282d4bd65dd1f6b7d33a1ea460db63181cc5367693e00c82be5cad8a8bc7 2025-08-26T20:45:49.1002715Z deleted: sha256:d7c4b9415b941bb5d3c7f5c10aed336d45d24315f746c4fcb3dfc23c6b8908d6 2025-08-26T20:45:49.1003130Z deleted: sha256:94fa86b861fd00cea6bff33e0bbc9078b581d6deb3779fcf9906a9619f1cbd06 2025-08-26T20:45:49.1003541Z deleted: sha256:2c44a97ad498ca2eb5b0bcbb68b73dc8d3cbc9181b2b4215d5976fee8c2b5dde 2025-08-26T20:45:49.1003955Z deleted: sha256:2fac967f0d0c18773694f70ef626be174a5adce8737ced4ea4b9941934750025 2025-08-26T20:45:49.1004358Z deleted: sha256:4f98077e501f50be7f43259df16ebd857ddcd29cc93b34fad4e5b874e9f35f05 2025-08-26T20:45:49.1004763Z deleted: sha256:fc2d753ded90860f5030cfcf5f0a3c2e57dfb41484540bd81bfc7a74746eb55f 2025-08-26T20:45:49.1005170Z deleted: sha256:90a2bf02e851326fc70d05470553ed33e578342d6e06bfa0cfaf331c4079b7e4 2025-08-26T20:45:49.1005405Z 2025-08-26T20:45:49.1005506Z Total reclaimed space: 52.72GB 2025-08-26T20:45:49.1075749Z Post job cleanup. 2025-08-26T20:45:49.1125042Z Post job cleanup. 2025-08-26T20:45:49.1956992Z [command]/usr/bin/git version 2025-08-26T20:45:49.1989775Z git version 2.47.1 2025-08-26T20:45:49.2022049Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/587ad427-5e99-4249-bb9f-1d95f745ae92/.gitconfig' 2025-08-26T20:45:49.2037804Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/587ad427-5e99-4249-bb9f-1d95f745ae92' before making global git config changes 2025-08-26T20:45:49.2038739Z Adding repository directory to the temporary git global config as a safe directory 2025-08-26T20:45:49.2043843Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-26T20:45:49.2100054Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-26T20:45:49.2133033Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-26T20:45:49.2463309Z Entering 'android/libs/fbjni' 2025-08-26T20:45:49.2515130Z Entering 'third_party/FP16' 2025-08-26T20:45:49.2571495Z Entering 'third_party/FXdiv' 2025-08-26T20:45:49.2629351Z Entering 'third_party/NNPACK' 2025-08-26T20:45:49.2687977Z Entering 'third_party/NVTX' 2025-08-26T20:45:49.2749687Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-26T20:45:49.2807095Z Entering 'third_party/XNNPACK' 2025-08-26T20:45:49.2869717Z Entering 'third_party/aiter' 2025-08-26T20:45:49.2928663Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:45:49.2992768Z Entering 'third_party/benchmark' 2025-08-26T20:45:49.3046678Z Entering 'third_party/composable_kernel' 2025-08-26T20:45:49.3109289Z Entering 'third_party/cpp-httplib' 2025-08-26T20:45:49.3158592Z Entering 'third_party/cpuinfo' 2025-08-26T20:45:49.3216249Z Entering 'third_party/cudnn_frontend' 2025-08-26T20:45:49.3277214Z Entering 'third_party/cutlass' 2025-08-26T20:45:49.3338151Z Entering 'third_party/fbgemm' 2025-08-26T20:45:49.3393185Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-26T20:45:49.3449061Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:45:49.3507029Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:45:49.3562103Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-26T20:45:49.3621884Z Entering 'third_party/fbgemm/external/googletest' 2025-08-26T20:45:49.3678976Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:45:49.3739536Z Entering 'third_party/fbgemm/external/json' 2025-08-26T20:45:49.3795191Z Entering 'third_party/flash-attention' 2025-08-26T20:45:49.3855128Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:45:49.3910759Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:45:49.3975929Z Entering 'third_party/flatbuffers' 2025-08-26T20:45:49.4032148Z Entering 'third_party/fmt' 2025-08-26T20:45:49.4088474Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:45:49.4145057Z Entering 'third_party/gloo' 2025-08-26T20:45:49.4195486Z Entering 'third_party/googletest' 2025-08-26T20:45:49.4257521Z Entering 'third_party/ideep' 2025-08-26T20:45:49.4308521Z Entering 'third_party/ideep/mkl-dnn' 2025-08-26T20:45:49.4368478Z Entering 'third_party/ittapi' 2025-08-26T20:45:49.4423918Z Entering 'third_party/kineto' 2025-08-26T20:45:49.4474349Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:45:49.4532534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:45:49.4581994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:45:49.4636875Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:45:49.4691670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:45:49.4749454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:45:49.4813296Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:45:49.4870391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:45:49.4929728Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:45:49.4975093Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:45:49.5030617Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:45:49.5086354Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:45:49.5150891Z Entering 'third_party/kleidiai' 2025-08-26T20:45:49.5206443Z Entering 'third_party/mimalloc' 2025-08-26T20:45:49.5262017Z Entering 'third_party/nlohmann' 2025-08-26T20:45:49.5318479Z Entering 'third_party/onnx' 2025-08-26T20:45:49.5391278Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-26T20:45:49.5449026Z Entering 'third_party/opentelemetry-cpp' 2025-08-26T20:45:49.5504257Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:45:49.5555115Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:45:49.5610287Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:45:49.5664876Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:45:49.5725437Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:45:49.5775097Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:45:49.5824121Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:45:49.5883839Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:45:49.5938735Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:45:49.5991164Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:45:49.6059780Z Entering 'third_party/pocketfft' 2025-08-26T20:45:49.6116021Z Entering 'third_party/protobuf' 2025-08-26T20:45:49.6168935Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:45:49.6230232Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-26T20:45:49.6289230Z Entering 'third_party/psimd' 2025-08-26T20:45:49.6340635Z Entering 'third_party/pthreadpool' 2025-08-26T20:45:49.6395146Z Entering 'third_party/pybind11' 2025-08-26T20:45:49.6451599Z Entering 'third_party/python-peachpy' 2025-08-26T20:45:49.6506004Z Entering 'third_party/sleef' 2025-08-26T20:45:49.6568100Z Entering 'third_party/tensorpipe' 2025-08-26T20:45:49.6622161Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:45:49.6676294Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:45:49.6736660Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:45:49.6788493Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:45:49.6848592Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:45:49.6926366Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-26T20:45:49.6952533Z http.https://github.com/.extraheader 2025-08-26T20:45:49.6958165Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-08-26T20:45:49.6992039Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-26T20:45:49.7299449Z Entering 'android/libs/fbjni' 2025-08-26T20:45:49.7338794Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7374053Z Entering 'third_party/FP16' 2025-08-26T20:45:49.7410985Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7448634Z Entering 'third_party/FXdiv' 2025-08-26T20:45:49.7479878Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7511842Z Entering 'third_party/NNPACK' 2025-08-26T20:45:49.7552502Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7587793Z Entering 'third_party/NVTX' 2025-08-26T20:45:49.7626143Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7661628Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-26T20:45:49.7696100Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7730972Z Entering 'third_party/XNNPACK' 2025-08-26T20:45:49.7769541Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7816866Z Entering 'third_party/aiter' 2025-08-26T20:45:49.7853445Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7891700Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-26T20:45:49.7930095Z http.https://github.com/.extraheader 2025-08-26T20:45:49.7968941Z Entering 'third_party/benchmark' 2025-08-26T20:45:49.8008261Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8048652Z Entering 'third_party/composable_kernel' 2025-08-26T20:45:49.8084003Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8128662Z Entering 'third_party/cpp-httplib' 2025-08-26T20:45:49.8165436Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8198608Z Entering 'third_party/cpuinfo' 2025-08-26T20:45:49.8231649Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8277078Z Entering 'third_party/cudnn_frontend' 2025-08-26T20:45:49.8312832Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8352981Z Entering 'third_party/cutlass' 2025-08-26T20:45:49.8384643Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8431099Z Entering 'third_party/fbgemm' 2025-08-26T20:45:49.8466065Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8500841Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-26T20:45:49.8538608Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8573822Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-26T20:45:49.8608619Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8649528Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-26T20:45:49.8685238Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8725415Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-26T20:45:49.8759625Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8800332Z Entering 'third_party/fbgemm/external/googletest' 2025-08-26T20:45:49.8835655Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8873855Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-26T20:45:49.8912369Z http.https://github.com/.extraheader 2025-08-26T20:45:49.8950806Z Entering 'third_party/fbgemm/external/json' 2025-08-26T20:45:49.8983171Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9017023Z Entering 'third_party/flash-attention' 2025-08-26T20:45:49.9056163Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9088153Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-26T20:45:49.9132030Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9166497Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-26T20:45:49.9205948Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9248743Z Entering 'third_party/flatbuffers' 2025-08-26T20:45:49.9283982Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9321703Z Entering 'third_party/fmt' 2025-08-26T20:45:49.9357154Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9394098Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-26T20:45:49.9430386Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9466727Z Entering 'third_party/gloo' 2025-08-26T20:45:49.9506745Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9547045Z Entering 'third_party/googletest' 2025-08-26T20:45:49.9581560Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9617490Z Entering 'third_party/ideep' 2025-08-26T20:45:49.9660102Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9693601Z Entering 'third_party/ideep/mkl-dnn' 2025-08-26T20:45:49.9730156Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9773977Z Entering 'third_party/ittapi' 2025-08-26T20:45:49.9814641Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9853854Z Entering 'third_party/kineto' 2025-08-26T20:45:49.9884521Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9922925Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-26T20:45:49.9958788Z http.https://github.com/.extraheader 2025-08-26T20:45:49.9994540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-26T20:45:50.0032071Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0067154Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-26T20:45:50.0105147Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0143180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-26T20:45:50.0178191Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0217021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-26T20:45:50.0250160Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0287173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-26T20:45:50.0329479Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0364218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-26T20:45:50.0399099Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0438492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-26T20:45:50.0473174Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0507597Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-26T20:45:50.0546164Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0579489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-26T20:45:50.0617100Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0657867Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-26T20:45:50.0688450Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0728903Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-26T20:45:50.0764980Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0812361Z Entering 'third_party/kleidiai' 2025-08-26T20:45:50.0846605Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0881318Z Entering 'third_party/mimalloc' 2025-08-26T20:45:50.0921003Z http.https://github.com/.extraheader 2025-08-26T20:45:50.0957738Z Entering 'third_party/nlohmann' 2025-08-26T20:45:50.0990316Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1030181Z Entering 'third_party/onnx' 2025-08-26T20:45:50.1065072Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1113096Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-26T20:45:50.1147517Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1188368Z Entering 'third_party/opentelemetry-cpp' 2025-08-26T20:45:50.1234597Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1270487Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-26T20:45:50.1304311Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1334972Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-26T20:45:50.1370033Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1410070Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-26T20:45:50.1441110Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1478984Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-26T20:45:50.1515249Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1552028Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-26T20:45:50.1585039Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1625424Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-26T20:45:50.1662123Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1695644Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-26T20:45:50.1732595Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1767872Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-26T20:45:50.1799357Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1838726Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-26T20:45:50.1874239Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1908690Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-26T20:45:50.1943161Z http.https://github.com/.extraheader 2025-08-26T20:45:50.1995033Z Entering 'third_party/pocketfft' 2025-08-26T20:45:50.2031628Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2065570Z Entering 'third_party/protobuf' 2025-08-26T20:45:50.2104762Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2141888Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-26T20:45:50.2177632Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2210402Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-26T20:45:50.2250887Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2279064Z Entering 'third_party/psimd' 2025-08-26T20:45:50.2319153Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2355567Z Entering 'third_party/pthreadpool' 2025-08-26T20:45:50.2391121Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2425656Z Entering 'third_party/pybind11' 2025-08-26T20:45:50.2464937Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2497020Z Entering 'third_party/python-peachpy' 2025-08-26T20:45:50.2537398Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2574914Z Entering 'third_party/sleef' 2025-08-26T20:45:50.2609667Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2648584Z Entering 'third_party/tensorpipe' 2025-08-26T20:45:50.2682501Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2717850Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-26T20:45:50.2751527Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2788351Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-26T20:45:50.2829649Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2862734Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-26T20:45:50.2899152Z http.https://github.com/.extraheader 2025-08-26T20:45:50.2939457Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-26T20:45:50.2976066Z http.https://github.com/.extraheader 2025-08-26T20:45:50.3006289Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-26T20:45:50.3046168Z http.https://github.com/.extraheader 2025-08-26T20:45:50.3189415Z A job completed hook has been configured by the self-hosted runner administrator 2025-08-26T20:45:50.3208684Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-08-26T20:45:50.3211981Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-26T20:45:50.3212243Z ##[endgroup] 2025-08-26T20:45:50.3296420Z [!ALERT!] Swap in detected! [!ALERT!] 2025-08-26T20:45:59.6020216Z [!ALERT!] Swap out detected [!ALERT!] 2025-08-26T20:46:15.2264073Z Cleaning up orphan processes